Hi John, a couple of dfinitions for 'tarry' include

1. To loiter in a place, or 'hang about'
2. To leave a place slowly and reluctantly.

This would fit in OK with your interpretation .....

Give us more detail as to the song, mate. Is it Irish or Irish/American? Was the work railway or canal building? If so..

Check, Drill ye Tarriers Drill - I found the - Lyrics here.

The Tarriers were the "Navigators" (Navvies), who drilled and blasted the rock, like a pack of canine terriers, to start the holes for cuttings etc, for the 19thC UK and US canals and railways (also here in NZ). The debris was then shifted out by wheelbarrow etc...

Which could explain your "tarried and shifted", depending on the 'work' context of the song.

Cheers - Sam