That song was widely song long before Dylan was a squint in his father's eye, mates. And the word bummer in that context means one who bums around, without visible means of support.

Somewhere between the 20's and the 50's the expression developed to give someone a "bum steer", meaning advice that had no merit, perhaps the kind of advice you might get from a bum.

This expanded later on into the expression "a bum deal", meaning an arrangement that did not deliver any value. With the advent of drug exploration, this was expanded to the phrase "a bum trip", a drug experience that was unpleasant, and from there the expression was born "that's a bummer", meaning that experience or relationship was depressing in the way that a bad drug trip is depressing.

So a "bummer", once someone who bummed around not working, was migrated over to mean a bad scene, deal, or experience. In today's argot it has been condensed into an expression of sympathy: "Bummer, dude!".

A