probably because the derivation is "man" = person in Olde English from the sanskit manu;

and "woman" from the specific phrase meaning "female person"

BTW - do you know the original meaning of "husband" is "dweller in the house"?