Black actually refers to the color of clothing they are wearing. Because in most of the versions of that song they say "trois hommes EN noir" wich mean three men IN black. In Québec's legends the devil(or is men) is often described as a men dressed in black who love to dance ( wich was considered a sin after midnight)