Brown doesn't necessarily refer to hereditary skin colour. To have white skin was a sign of aristocracy, of not being exposed to sun and wind in manual work. So the brown girl with the houses and land had worked for her wealth, at the cost of her appearance.

Same reason female office workers of the 60s had often unfeasably long fingernails. And why there's a market for artificial suntan potions- being brown now implies (for pink skinned folk) the wealth and leisure to go to sunny climes.

Somewhere in this lot there are several references to the banning or expulsion of Gypsies (with as contorted spellings as can be imagined, which is why I've left you to find them) from the city of Edinburgh in the 16th century. I don't recall them as being accused of seducing noble ladies though.