I'll not be as subtle as LH, Martin you're talkin' out you're arse mate.
The fact that you don't like his voice is fare enough but to them conclude that he is a "bad" singer is sheer self-indulgence.
Dylan gives you everything he's got with an intensity and commitment that few singers can manage.
As to the lyrics well it does take a fairly broad familiarity with the diverse cultural resources he uses, to appreciate his stuff, but that's hardly his fault.
The music particularly the electric stuff with the Band is just wonderful, check out the re-mixes of Highway 61, Blond on Blond, and any live stuff from that period.
So you played his songs because they were trendy, enough said.