Here's a real life example of the phrase "of color".

A few months ago, a social worker wrote a column in my newspaper about prostitution in our town. His last sentence was the sad comment that the typical new prostitute is a 13-year-old girl of color.

When you submit such a column, you are allowed a certain number of words, and your writing must be succinct and have an impact.

She might be African-American, Central American, Asian or a mix of those or a mix of those with white. He's got a word limit, so he said she's 'of color'.