Somebody comments that the 'let her go' verse seems like an add-on.

Robert Harwood has traced it to a song in a Harvard University song book, which exists in various versions.

It has been interpreted as stating that the singer is an arrogant pimp, on the basis that 'sweet man' was slang for pimp, but this is cited as a Caribbean usage, so I don't know if it was also a USA usage.

Robert Harwood's book is wonderful