For a while, wearing mismatched socks was a fad among teenage girls. An older gentlemen, observing one doing that, mused (quiped), "I wonder if she has an identical pair in a drawer at home?"
Wear whatever makes you happy. I think the palette for men's clothing should include all the colors and not be limited to black, brown, gray, blue, and white. I like seeing men wearing other colors, especially yellow and pink.