even when it has clearly not been used to cause offence

Surely you mean it was not intended to cause offence - because it sounds very much as if it did actually offend someone.

What matters isn't just what we mean to communicate by our words, it's also, and more important, what other people understand us to mean when they hear what our words.

There are no taboo words. There are words which have a history which means that they are likely to be understood as indicating an intention to cause hurt and insult. Clearly it is important to avoid sending out that message, which means being careful about the context in which we use those words.

But using other words to send that same message is no great improvement - if Michael Richards had avoided actually using the word "nigger" that wouldn't have meant that his outburst was any less racist and repulsive.