All right, Leadfingers, I found the album and forced myself to listen to the song.
The lyrics you remember differ only trivially from the recorded version. The number of old ladies in the chorus is "three."
Old lady number three:
The third lady's name was Elizabeth Spender, Who knew all the scandal but liked to pretend ter Not breathe a word except when the three met And nobody knew they were there.
Bob Cort performs in Sinatra-swing style "with chorus and orchestra directed by Billy Munn" on Decca LK4301, "Eskimo Nell and Other Barrack Room Ballads."