Mick, it depends what you're doing. If you are a bunch of musicians on a night off, just wanting to let your hair down & play informally, then you are not going necessarily to want an audience. If you have too many non-participants then it can change the feel of the evening into more of a performance thing.
Also - an audience can put pressure on less confident performers & put them off. Ad hoc audiences at singarounds/sessions sometimes don't quite "get it", and may want more of one thing & less of another - sometimes loudly.
I think maybe you took the comment a bit out of context?