The only sort of "Duff" I can think of in the USA is the name of Homer Simpson's favorite brand of beer.

The song doesn't even vaguely suggest that the speaker is "pompous." Calling himself a "poor old buffer" is self-deprecating (unless one thinks of him as a naval petty officer).

In outspoken circles "buffer" may easily have been replaced by something stronger and more generic.