But then you get situations where the offensiveness is an act of bullying - people who make racist or sexist remarks as a way of showing who is in charge, and that kind of stuff.

It's not a simple matter of people unreasonably taking offence at things that are really fair enough. Context is everything. A word can be like a blow in the face sometimes.