Tinnitus can be a tremendous problem with some people, and as far as I can judge it does seem to be associated with being subjected to noise (though I'm only being anecdotal on that). Also, you sound as if the noise is making the job a chore rather than a pleasure. Difficult, because they won't sack the drummer just because you ask them too, and you probably dont want to be that confrontational anyway.
   A lot will depend on who sets up the gigs. In general, do the band ask you to cal for their gigs, or do you ring up the band and ask them to play for yours? If the latter, it's easier, you just book another band that you enjoy more. If they hire you, it's trickier.
If you start turning them down, because the noise is becoming intolerable, you should definitely tell them why.The rest of the band may hate it too, actually, but never like to say!