In the 19th century, convicts were sometimes used to provide labour for the construction of Public Works and the like.  The Harbour of Refuge Breakwater was one such project; a parliamentary sub-committee of 1882 decided to build it at Peterhead.  This caused some resentment from other places which had wanted it.  The version given in Buchan and Hall's The Scottish Folksinger (1973) makes the sense a bit more clear:

When the harbour o' refuge was first spoken aboot
Aiberdeen and Stonehaven they were fairly pit oot,
For the Queen kent the convicts wid get their best farin'
Upon Buchan tatties an' Peterheid herrin'.

From the singing of Jake Mitchell of Peterhead.  A well-nourished labourer will do better work.

More details of the Harbour of Refuge may be seen at  PETERHEAD PRISON

Malcolm