Navvy was indeed sourced from navigator: the same people who built the canals built the railways (the first railways often following the canals because there was no need of wayleaves, and the telephone lines ditto.
They were neither common nor unskilled.
In America, these working people were often Chinese; I've heard it said that a Chinese man died for every yard of railroad built.
Irish people are still working in construction across Europe, whether as the engineers and architects designing projects or the skilled crane and drill and spade men and women fulfilling them. In fact, as recently as during the building of the Channel Tunnel it was said that there was more Connemara Irish spoken in the depths under the seas than there was English or any other language.