I first heard it not as a song but as a story, recounted by Gerard Hoffnung, recorded addressing the Oxford Union in December 1958. According to the BBC CD: 'The speech contained the now-famous Bricklayer story, which can be traced back to the 1920s and is probably much older.'
No song could match the timing in that performance.