Neil even made an appearance on the show after Colbert called Midnight. He's too cool to be a nerd, although I'm sure he needed lots of media training.
As for the retarded and retard references, bear in mind there are people here with IQ's of 82 who don't think "you're' too smart either.
The type of person whom you wished to refer to are called savants.