I've never heard it clearly said that bright colours are evil, but I suspect it's the thing that's been mentioned about good Christians being supposed to modest and sober, and the idea that "loose women" "flaunt themselves" in "loud clothing". That was the feeling I picked up from my grandparents.

My mother told me that my grandfather had an aversion to green and didn't want any women in his family to wear it. She thought he considered it unlucky, but I guessed it was maybe more to do with it being one of the suffragette colours, along with purple. Come to think of it, I never saw them wearing purple, either.