William W. Trotter in 'Silk Flags and Cold Steel- The Civil War in North Carolina: The Piedmont'cites a letter from young woman discussing how they celebrated Christmas which includes a quote that we sang ". . . When I lived in the state of Virginia, to Carolina in did go, and there I saw a handsome lady, Oh! Her name, I did not know."
That places that fragment of lyric to 1864 or earlier, but I haven't been able to find the song referenced by just that fragment.