"And one by one and two by two
They have sailed from the wharf since then"

- and in those lines the singer sums up a long sequence of partings, that ends up leaving him the last to go.

The power of the song lies in the way that it's not actually just about a bunch of mates heading off on their travels. It's about the way life sooner or later takes things and people we value away, and then it's our turn, because that's how the world goes. It's rather like The Mary Ellen Carter in that way - there's a universal story behind the particular events, and that's what moves us.