If you have that reaction yourself, you might notice who it is that tends to set it off. A group of hoodies on the street at night might tend to prompt me to cross the road discreetly. I don't think their colour would make any difference, even if I could see that kind of thing.