My (extremely!) black husband always says he doesn't mind one bit being called 'black', 'coloured' 'African' or even 'nigger' (sorry about that). He can tell immediately if someone is being nasty or objects to his race. He's lucky in that he's perfectly secure within himself.
However, I'm sure many non-white people can be a bit sensitive to how they're referred to, and they have every right to that.
The thing is to try very hard not to offend, and to be respectful and kind. To have no innate prejudice and not to generalise. Every human being is after all an individual, whatever their outward appearance, race or origin.