"I spend a lot of time apologizing and not knowing why"

Then don't apologise, Dude.
Make those demanding your apology explain why, then explain to them that whatever you said didn't mean what they think it meant - ie that the fault is theirs and lies with their own incomprehension of the English language.

If that doesn't work, tell 'em to Foxtrot Oscar. I do.