It would be resonable to accept the canals were the first large scale projects to use large mobile groups of unskilled workers and that is indeed the origin of the term navvie. It's later adoption for unskilled workers in general is perfectly logical, but the word comes from working on the inland navigations.