"Yella" /"yeller" [yellow] is a very common referent for light skin in African American secular slave songs.

Most of the references that I have found refer to women and not men. The majority are complimentary.

See this example:

Come down to Tennessee
{Ride er ole grey horse}
Yaller gal's de gal for me
{Ride er ole grey horse}
Kiss her under de mulberry tree
{Ride er ole grey horse}
Oh my, N----g,* don't you see
Better come to Tennessee.

Dorothy Scarborough, "On The Trail Of Negro Folk-Songs",
{FolkLore Associates Edition, 1963, page 183; originally published in 1925}

* I choose not to write this entire word, but that's me and I'm not suggesting everyone do as I do.