What Joe said. Neither your friend nor you are in the wrong imo. It was fine for you to say under what condition you could accept the invitation. It was fine for them to say "Sorry, we aren't willing to meet that condition." It was fine for you to then decline the invitation.

Sorry your wife is mad at you. Hope all of you have a great day, however and with whomever you spend it.