The brown/poor fair/rich convention makes no sense in this song, because it is the brown girl who had the wealth and the fair one who was poor.
There is no reason to suppose any racial implication; Moors did not own houses and lands in the place and time when Lord Thomas might have been in the Forester. To assume any racial intent, the story would need to be relocated to Spain, or the brown girl cast as the offspring of some Mediterranean follower of the the Conqueror, with fair Aelwyna a dispossessed Saxon, either of which would be over-elaborate and unnecessary back-stories.
The black=night=bad v fair=day=good convention seems the most likely and consistent in this case.