Well gee whiz, was the hatter in Alice really mad? More "annoying inaccuracies."
The tea party was characterized as mad, not the hatter.
Than that libelous Cheshire cat said, "In that direction lives a hatter: and in That direction lives a March Hare. Visit either you like, they're both mad."
Visit that fount of all miscellaneous knowledge, and several possible explanations are given for the origin of the phrase.