Thanks for the responses. The version I recall had the chorus:
With my toor-a-li toor-a-li toor-a-li-ay What have you been doing this long summers day Well, you ain't ploughed an acre I'll swear and I'll vow For you're damned idle fellows that follow the plough
It also had a verse refering to an early morning hour (4 or 5 o'clock) when "we rise from our bed - and its down to the pump house where we douses our head.
Does anyone know what album the Watersons performed this song?