You have never heard any reasoned argument in favour of colour prejudice? There were plenty. That is, I must emphasise, "reasoned" as opposed to "right". So much depends on what your starting axioms are.

Morality is tricky: you cannot rely on what is obvious, or how many agree with you, and whether those affected by discrimination is a small minority or not is of no consequence, unless you want to get onto the unsteady ground of arguing that it is ok to discriminate against a group providing there are not very many of them.