In both terms one thing is most important.
"A coloured person" - it's "coloured".
"A person of color" - it's "person".

I think it comes down to pigeon-holing a human or describing them.

I also should say that I've been told "Brit" is offensive to some.

When it comes right down to it, it's about whether the people hearing whatever term you use will take whatever else the know, or think they know about you into consideration, and think you're a dick (NOI).