The earliest American slang usage for 'fairy' was prostitute, found in print in 1856.
"I met eight men on their way to Kansas... a hard & rough looking set of fellows... they told me they stopped the night before at Bangor in a house of the "Farries" & [persuaded one of them] to accompany him to the Territory...She goes for the especial accommodation of the company, though under the name of the wife of one:" Arkansas Historical Quarterly.
1866- H. J. Harris, "Southern Sketches." You will tell us no more of the fairy. I dare say she has you irrevocably charmed."

I don't doubt the usage by Blacks of fairy for a "fair-skinned brown." (Jon W).