"I would suspect it would be because only the manual laborour would be seen in his undershirt."

Good point, Mario -- but the problem is that skivvies were almost invariably *female* servants.

Actually, skivvy=vest sticks out like a sore thumb, and also seems to be lateish. There's a run of words, most notably "skive" meaning to shirk, avoid, etc. But nobody seems happy to see these as leading to skivvy=female dogsbody.

A muddle and a problem.

I just wrote, in response to a backchannel from the guy who origianlly raised the "Pharoah's Daughter" test:

"
> Presumably 'slavey' began as some ponderous Upstairs joke, following
> Abolition. But why did 'skivvy' turn up?

Yeah -- that's one of the odd things. As far as I can see, there isn't any semantic difference between "slavey" nd "skivvy". "slavey" is used for coming on for a hundred years, then suddenly, in the early 20thC, "skivvy" appears beside it, and within about ten years, "slavey" has been completely replaced by "skivvy".
"

Robin