"Ewan's glosses can be a bit odd. Verse 2 starts "Sister, we'll gang tae the broom", with "tae the broom" glossed as "make love". Go figure!"

My interpretation (OK - guess!) at what the broom motif means is as follows: In many parts of the UK (especially on neutral or acid soils) the yellow flowered shrub 'Broom' (Cytisus scoparius) is characteristic of commons and waste places. In the days when people lived in cramped and crowded conditions the only places, in which young couples could pursue 'amorous dalliances', in relative privacy, would be those places where the broom grows (and provides good cover). The only people you would be likely to meet in such places (apart from other courting couples!) would be shepherds - and as Bob Copper says (I'm quoting from memory), "shepherd 'e don't say nothing, only swear at 'e."