If it was a paying gig I would think twice about pulling out. Also, did the audience pay to come in? If it is no to both then good for you. About two years ago I was playing at a session and the audience was so loud that I stopped half way through a song and crossed my arms. When the last two people stopped talking I addressed the audience by say that the musicians that are here tonight are here for you to hear what they would like to say. They are all on their time and get no monetary compensation except the pleasure in playing for you. I've never went back and as I hear it is still just as noisy.