If you need the same number of syllables as pickaninnies - which I assume you do - you could try:

When the workers go to pick the cotton

It's only a two-syllable word, but the added "go to" echoes against the going-to-town in the next line, pointing up a nice contrast between the destinations.

If you don't need four syllables you can just leave out "go to" and use Workers or Farmers.

Apparently picking cotton is HORRIBLE work! I read a detailed description once of what you have to actually do (while bending over in the boiling-hot sun) to release the bit of the plant you want from the clutches of that which you don't, probably cutting yourself in the process. Sure puts paid to all those city-slicker illusions of just plucking it like a piece of fruit.