Nature tends to use bright colours as a warning signal. Some poisonous insects and animals use bright colours (red and yellow mainly) to advertise the fact. (Some who are not poisonous also use these colours hoping predators will mistake them for the real thing and thus avoid being eaten without having to go to the trouble of making their own poison - but that's another story!!)

Maybe that is why we use the colour red on traffic signals as a warning sign for danger.

Perhaps using bright colours for clothing is seen sub-consciously as using "tainted" colours that in nature are signalling danger??