I remembered to ask my dad about this one tonight.

Apparently the lady in question was part of the aristocracy and was one of the first to start driving in a car, which at that time were also known as mechanical horses. One night a new chauffeur picked her up. The conversation went something like this:

"You're new! What's your name"
"James My Lady"
"We call our staff by their surnames - What's your surname?"
"Darling My Lady"
"Home James, and don't spare the horses!"

As soon as this (supposedly genuine) story got out the musical hall picked up on it. Hence the song.