It's a phrase used with specific connotations in specific situations. All the jibber jabber about what things used to be like in My Day or in Days of Yore and in My Country or in That Country I Once Went To, along with all manner of the It's Not Fair and It's Not Logical is unhelpful.

"Person of Color" means "non-White", in a place and in a discussion where "White" is believed to mean something reasonably consistent.

The alternative to "Person of Color" vs. "White" would be "non-White" vs. "White." I think the latter is the one that would truly imply, as Mrrzy said (of the former), white = normal.

The proposed alternatives of either distinguishing people more finely or not distinguishing them at all are not helpful because the distinction between precisely TWO groups is the intention of phrase when and where it is being used.

Perhaps there is another alternative -though I don't see why "we" would be the ones to prescribe it here - but it would have to retain the useful aspects of the current term.