I've been pondering this for many years, and the closest I can come is that *IF* it is a private party and almost everyone knows each other and knows what to expect, then a free, open, no rules except politeness basic singing session is possible.
   In the US, as Susan mentions, we used to call those 'hoots'...short for Hootenanny, a name which just grew back in the 50s & 60s. I used to attend a few of those, and it was nice. One song led to another and people would ask each other for songs and those with 'more' experience would usually lead more stuff, while 'learners' would ...ummm... learn and do a bit less until they got more proficient.

But, if the event is widely publicized and 'open', it usually mst be more 'formal', with everyone being invited to take a turn...often by simply going around a circle. Otherwise, it is as Bert says and ends up with "a few of the loudest and brashest hogging the show". I quite realize that many of those loud, brash ones ARE often better, seasoned performers who have a large repertoire and who get weary of what they consider to be weak, boring awkward attempts by 'amateurs'. Some, obviously, have a lower threshold of pain than others.

   After 45+ years of this, I see no easy solution except making it clear to the group what to expect, and asking everyone to think about how to make the evening pleasant.

Groups which meet regularly can assign a standard name to each type...whether "hoot" or "scat" or "session"...just as long as folks know what to expect.