French has specific words for hair colour (black-haired is brun). So we wouldn't think of the hair colour. You're right though about noir not being at first about the skin. In France it came to mean your skin is tanned and dirty, and all the usual prejudices apply. Think of Heathcliffe.