It may seem their appearance doesn't matter, but facial expression and body language is a primal way we communicate. I know someone well who has Asperger's syndrome. Unlike most of us, he cannot read facial expression, tone of voice, body language, to judge how people are reacting to him. There is definitely a snap decision we make by reading expressions and voice tone. Check out the book BLINK, about the first two seconds of decision making, by Malcolm Gladwell. Then there is the book that refutes BLINK, by Michael R. LeGault, called THINK. Blink is described at wikipedia, here:
http://en.wikipedia.org/wiki/Blink_(book)