I'm sure I read fairly recently in some learned tome that in the 17th and 18th centuries these sort of terms simply referred to hair colour, but the outdoor skin coloration sounds very plausible.
We speak of a redhead even today and some quite offensive equivalents.

Child 295 only dates from the late 18thc and is largely a collection of stanzas from other ballads, but it certainly seems to refer to skin colour rather than hair colour. In the broadside A version 'he could not fancy me because I was so brown' could hardly refer to hair colour.