Roustabout was more used in the past- it could be a dock, levee or railroad worker, a hand on the oil rigs- in other words, a semi-skilled laborer.
Take off your roustabout means take off your working clothes and put on your navy blue (glad rags).