It does seem to me that the term "racism" must imply that the racist is part of the part of the community that has the prevalence of power or approbation.

But the issue is complex. When does the refusal to be prejudiced morph into a foolish refusal to learn from experience?