Look at the words of the verses -

|: That nigger from Shiloh
Can pick a bale of cotton
That nigger from Shiloh
Can pick a bale a day. :|
Chorus:

Me and my gal can pick . . . .

Me and my wife . . . .

Me and my buddy . . . .

Me and my poppa . . . .

Takes a mighty big man to . . . .

*****

We all help each other - but that nigger can do it by himself.

No. You're right. How on earth could the words or sentiment cause any offence to anyone?