According to Wikipedia, haberdasher "is most likely derived from the Anglo-Norman hapertas, meaning small ware."

And, as leeneia said how mongers are looked askance at nowadays, would a used car salesman be a carmonger?