OH!, I just remembered! There was a chorus that followed the above stanza. It went:
CHO: Walk in, walk in, walk in, I say.
Walk in the parlor and hear the banjo play.
Walk in the parlor, and hear the banjo ring
And watch the nigger's fingers as they pick upon the strings.
I remember my father sang it that way at first, which was the way he had learned it. Even at a very young age I was aware of the "n-word" as an insult to negroes (as we called black people then in Wisconsin). So I remember telling my dad to change it to "old man's fingers." After that, we always used that phrase.