Here's another game rhyme that is found in Talley's collection thath may be a source of "I'm goin' down town to smoke my pipe", {particularly the version of that game that katlaughing found.

Note: I'm using asterisks instead of spelling out the "n word"

In A Mulberry Tree
Jes looky, looky yonder; w'at I see!
Two liddle N**gers in a Mulberry tree,
One cain't read, an' de t'other cain't write.
But dey bofe can smoke deir daddy's pipe.

"One ma two! One ma two!"
Dat Mulberry Witch, he *titterer too!
"Big bait'o Mulberries make 'em bofe sick.
Dem liddle N**gers gwinter roll an' kick.
-snip-

* titterer=laugh
Thomas Talley; Negro Folk Rhymes, Wise And Otherwise {Kennnikat Press Edition, 1968; p. 158, originally published, 1922, The Macmillan Company}.

**
It's possible that these this rhyme was combined with "Chickama Chickama Craney Crow" {also known as "Hawk And Chickens"} to create the "Goin' Downtown To Smoke My Pipe" game. But it's also possible that although these three examples have some text {and possible performance activities} in common, they aren't part of the same family of rhymes.