As a concert goer I only join in when invited.
As a singer at the 'Irish session' in our local pub, usually unaccompanied, I think its down to me. Sometimes the joining in puts me off: as the beer goes down around me so does the tempo, and I'm trying to move on to the next verse while the audience is still on the last line of the chorus. Still, everyone seems to have a good time and in that context its fine. I go elsewhere for a more 'respectful' audience.