There is another "Brown Girl" song that starts
"I am as brown as brown can be" - it's in the DT as Brown Girl 2
It appears to be the story of Barb'ry Allen told from her point of view. The "true" lover at first scorns her because she is so brown; upon discovering that he is perishing for the love of her he comes back to beg her pardon. She tells him to get lost, saying she will "dance on his grave".
There are some who hypothesize that "Barb'ry" = Barbary, or a dark-complected woman; if Barbara Allen was a Romany, it adds a twist to the story. Martin Carthy talks a bit about this in the liner notes for the latest "Waterson:Carthy" album, Broken Ground.