Here is a previous thread about this song with some interesting contributions (and the text of the song) Ole Black Joe, some don't believe it exists.

I can't begin to see how there is anything in the least racist or offensive in the song. Is the idea that there is cause for offense because the song refers to the fact that it can be possible to have happy times even in a life where we are oppressed? Because that is just common sense, and the common experience of humanity. It doesn't imply that oppression isn't really such a bad thing. It just means that human beings can be incredibly tough and brave, even when things are against them.

One suggestion in that other thread which struck me as really strange was that, if it was "Poor Old Joe" it wouldn't be offensive, but with "Old Black Joe", it is somehow. I just can't get my head round that one.