People might be uncomfortable with the reference to "High browns from down the levy". It's not one of my favorite Berlin lines, because it harkens to a time when skin color was connected to social standing, and legal rights. That wasn't his intention, though. Consider that POTR was introduced in a 1930 film of the same name, and had the distinction of being the first song ever sung in a film by an interracial chorus. Not really something any sort of racist would be likely to do.

Anyway, he re-wrote the lyric when the song was filmed again 1946.