One important part of the charm of this song, for me, is the comfort of what one might call "illicitude". One thing after another is mildly against the rules. The memory of racial discrimination is stirred up twice, and allowed to settle. The old men are gambling in a public place, and no-one cares, least of all themselves. They are drinking their own booze and cheating the Illinois Central out of its monopoly profits on the same. The conductor sings a slightly smutty song, and the mothers presumably do not take offense.

There was a lively thread on this song on rec.music.folk a few years ago.