Even if the original song was "trois hommes en noir" (three men [dressed] in black) it doesn't alter the fact that some people understood it as "trois hommes noirs" (three black men) and it certainly made sense for them. These black men are not "Black" from anywhere in the world but their skin is dark because they come from Hell and they are usually portrayed (or imagined) as if somewhat overbaked in an oven. At least it's how pictures of these men from Hell generally appear in folk tales books.