It would seem that there were many "brown girls" and also "brown men" in old English ballads. In "English and Scottish Popular Ballads" [edited from Child's ballads] there is one [#295] called "The Brown Girl." There are two versions that are very similar except that version B is twice as long [16 verses]. It begins:
I am as brown as brown can be,
And my eyes as black as sloe;
I am as brisk as brisk can be,
And wild as forest doe.
My love he was so high and proud,
His fortune too so high,
He for another fair pretty maid
Me left and passed me by.
Me did he send a love-letter,
He sent it from the town,
Saying no more he loved me,
For that I was so brown.
In the same book, #97 is called "Brown Robin." It tells of a king's daughter who has her "e'e on Brown Robin." Her father disapproves of Brown Robin, who is a commoner, but she manages to run off with him and "never came back again, her auld father to see." There is also #98 entitled "Brown Adam" who is identified in the ballad as "Brown Adam the Smith."
It would appear in all of these ballads that the appellation of the term "brown" is used to denote a person who is a "commoner" as opposed to persons of the upper class or aristocracy. It's possible that the term could also refer to persons whose complection is browned from outdoor living or an occupation such as that of a blacksmith. I would think it likely that other references in early ballads and songs to "Brown Girls" are also references to "class" rather than to race, nationality or ethnicity, or hair color. Just my "two cents worth."