One problem is that, on the whole, people who are into playing in sessions are the kind who let things go until they can work up enough steam to get angry. So rather than asking politely at the right time, they wait and wait, and then lose their temper.
That's the worst of both worlds. It ends with the objector feeling embarrassed and upset, because in retrospect they overdid it, while the twit with the bongoes is bewildered, because, by that time, he (or conceivably she) had been given the message that the bongo playing was quite all right - and then, out of the blue, it wasn't.