Many manners conventions arise from the need to demonstrate "no offence" -- for example, taking off your hat to greet someone, taking off gloves when shaking hands and, more recently, taking off sunglasses when meeting someone for the first time. Without the hat, gloves and sunglasses, your face is clearly visible and your hands are obviously empty. Thus, your interlocutor can see that, should you strike him or her, you are unlikely to do extensive damage, and she or he can pick you out of the police mug-book later.
My pet peeve is the ever-so-self-consciousness of the ball cap worn backwards or sideways, especially by someone who is consequently squinting into the sun. It looks fake to me and, as such, a sure sign of somebody I would rather not know.