I'm amazed to see that no one seems to have mentioned the derivation of the word "navvy" that I learned, and which I find completely and unquestionably credible. The low-paid construction workers were called navvies not because some workers had once worked on inland navigations, but because they were immigrants. Because they had come over on a boat, they were mockingly called "navigators", and this got shortened to "navvies".