Well I think in this day and age just about anything goes, but we live in a time where it's much more acceptable to call attention to yourself or stand out in a crowd - it's encouraged, really.   In earlier times, especially with some religious groups, wearing bright colors might have been perceived as an egotistical thing to do. Someone who dressed in bright colors wouldn't have been considered "humble" or "modest" which was what one was supposed to be. That may be where the main objection to bright colors would have come from.