"Bugger is now commonly applied to the old" Well the bloke in the song is getting on a bit isn't he?

But I don't think in my experience there's any age implication. If a kid in a skateboard hared across the road in front of my car I'd certainly be quite likely to describe him as a silly bugger. Maybe there is a difference in usage.