There isn't a whole lot to do on a farm at the end of December or the beginning of January, unless you're raising sugar, then it's harvest time. If you're growing tobacco, harvest has just ended, and celebration is in order.
The early church converted the Roman festival of Saturnalia, when no work was done, into the twelve days of Christmas, when peasants partied from 12/25 to 1/6. (Only December wasn't the twelfth month then, and January wasn't the first.)   
It would make sense that slaveowners would allow their slaves that time off.
Booker T. Washington wrote about free Blacks who continued the custom of not working during the Christmas season. Washington harped more or less continuously on the indolence of African Americans.

In central Pennsylvania the Belsnickle (I may not have spelled that right.) turns up to frighten bad children. I don't know what color he is.
The Hopi Kachina have a creature called the oger, who appears to frighten naughty children. I don't know what his color is either.