Green does seem to have some witchy suggestion in some ballads. And the term "scarlet women" is still a loaded term. But generally I think colours are just colours.

Though "Beige" these days does tend to be used as a put down. If a song described someone as being dressed in beige there'd be a plain (and most unfair) implication that they were a bit boring.