I agree that there is a reason to feel a bit punished by events, but the term 'colored' dates from the 30s if not earlier. The term 'Negro' was used in the era from the 50s to 70s and we now not only have to deal with the long existing NAACP (rarely reading out what it stands for), but the phrase "people of color" has brought us close to full circle.

A rule of thumb I was introduced to when I moved into a college dormitory that was ethnically 'concentrated' was that people heard a term that may or not be a slur, they reacted much more calmly if you said it with a smile. (You get extra credit if you remember the relevant quote from "The Virginian" by Owen Lister).