The DT has one version of this song, from Lomax, "Our Singing Country." The Traditional Ballad Index (cufresno) only traces the song back to 1927, in spite of the fact that it has been reported numerous times by ballad and Negro song collectors. Verses are many and varied, but the chorus with the words "Toting my tobaccer down," or "To carry my tobacco down, " or "To sell my 'bacca down," etc. is fairly constant.
It seems to have originated with the minstrels, as so many songs did, and was taken up by both Negroes and whites and made their own. White, N. I., 1928, "American Negro Folk Songs," found it in "Negro Singers' Own Book," ca. 1846, in "A Going Along Down," p. 56, and "Lynchburg Town," p. 157; also in "Ethiopian Serenaders' Own Book," 1857(?) in "Lynchburg Town," p. 80; and in "Christy's Nigga Songster." The song persisted and it was collected in North Carolina from blacks in 1909 and 1915-1916 (I'm gwine down to Richmond town) and in 1925 (Vicksburg town) by Scarborough.
A still older version does not have the usual chorus, and the lyrics are quite different, but the relationship seems to be evident;
JONNY BOKER OR THE BROKEN YOKE

Chorus:
As I went up to Lynchburg town,
I broke my yoke on de coaling ground;
I drove from dare to bowling spring,
And tried for to mend my yoke and ring.
Etc.
From Sweeney's Virginia Melodies pub. 1840, composer unknown. The words and sheet music may be found at Levy's Sheet Music website. Changes in the lyrics would have to be made in order to perform it today, but the verses are basically interesting and the effort to revise it could be worth the effort.

Sandburg (1927), has a slightly different version of Lynchburg Town in his "American Songbag," p. 145, under the title "Goin' Down To Town." (with music)

Lyr. Add: GOIN' DOWN TO TOWN

I used to have an old grey horse,
He weighed ten thousand pounds,
Every tooth he had in his head,
Was eighteen inches around.

That horse he had a holler tooth,
He could eat ten bushels of corn,
Ev'ry time he opened his mouth,
Ten bushels and a half were gone.

I had a yaller gal,
I brought her from the south,
All the fault I had with her,
She had too big a mouth.

I took her down to the blacksmith shop,
To get her mouth made small,
She opened her mouth to get a breath,
And swallowed blacksmith, shop and all.

I'm a-goin' to get me some sticks and sand,
To make my chimney higher,
To keep that dog-goned old tom cat,
From puttin' out my fire.

Chorus:
I'm a-goin' down to town,
I'm a-goin' down to town,
I'm a-goin' down to Lynchburg town,
To carry my tobacco down.