I think there are other options. He would (hopefully) not begrudge you and your wife a private trip away as long as he is cared for. You could have a little Christmas dinner or special holiday events before or after. You could find a nice caring individual to take care of his needs at home.

Your marriage vows remember are to your wife....he does not need every moment of your life..some..not all...and not necessarily every holiday. mg