It's not just with women tghat coolour comes into it:

"Some say he's black, I say he's bonny
fairest of them all is my handsome young love Johnny."

I always take that to be hair, though I suppose it might be temperament - after all he is "fairest" in the next line - but I think that's just another way of saying he's bonny. The expression "Black Irish" is primarily hair, but again can be temperament. And now it can mean colour as well, I'm glad to say.

But the Star of the County Down I'm sure should be "brown colleen". "Sweet" and "fair" sound like they might be anyone, "brown" tells you it's a particular girl he's singing about. And I'd take it to be brown hair together with an outdoor complexion, like the Nut Brown Maiden.