Discussion in "The Alabama Folk Lyric: A Study in Origins and Media of Dissemination," Ray B. Browne, 1979, Bowling Green Univ. Popular Press.
No. 38, p. 117, section on 'Unhappy Love Songs'.

"This piece, very much in the Stephen Collins Foster tradition, was published by H. M. Higgins Co., around 1865. It appeared in "Beadle's Dime Song Book, No. 16" (1865, p. 40), with no author givin. Beck (Lumberjack Songs, p. 217) reports it. My informant did not sing it as a Negro song. It is probably a pseudo-Negro creation. Possibly it is a song about a white man's love for a mulatto, in the same spirit as "The Little Mohea." ........ The piece bears no resemblance to Longfellow's "The Quadroon Girl"."

A portion of the book is on line. I have not been able to check the citations.
If correct, this would explain remarks such as "My grandfather sang it years ago," seen in some references.