Sure enough there are the associations which mean the song gets read one way rather than another. But that's not all there is about a song. and it would be perfectly possible to read most of those lines in quite the other way:

"I wish I was in the land of cotton" - that could be any exile wishing that they could go home and live freely. "By the water's of Babylon..."

"In Dixie Land I'll take my stand" could mean "I won't back down and be oppressed"

"Old times they are not forgotten" could mean "I won't forget what has been done to my people"

I'm not saying that's what the song really means, but rather, that's what it can mean. Good songs often don't just have one meaning. Abandoning a song to the enemy is like abandoning an occupied country to the enemy.