My husband is an intelligent and aware person. If he were at all offended by anyone's remarks, he's more than capable of objecting on his own behalf.
But he's not offended. It's just banter, and he actually enjoys it.
He's insightful enough to know what their true feelings are, and that they like him enormously.

They aren't elderly at all, they're aged about 50 (much younger than us!).

I do wish people would let the so-called 'victims of racism' defend themselves if they want to. It's belittling to assume they're too weak to speak.