In _A Death in the Family_ by James Agee, a little white boy whose name happens to be Rufus is taunted by the neighborhood bullies for having a "nigger name", and they use that song. That would have been about 1915.

Then there was

"What does you lahk bestes', Rastus?"
"Ah lahks asbestos, Rufus."

So, yes, the names carry some baggage. Maybe they will manage to shed it by & by.