I first heard the song (the "darky" version) in an assembly at my (progressive, private) high school about 1952. The man who sang it was black, a teacher, a former minister, and of British origin. We did not consider "darky" (or "pickaninny", for that matter) offensive -- such words came across as affectionate slang, unlike "nigger", which everybody knew was a nasty word.