I think there's a big difference between the situation when other musicians join in with accompaniment that's there already, and when they play with an unaccompanied singer.

If I'm playing an accompaniment when I'm singing, I'd normally be happy with others joining in, but when I'm singing unaccompanied, I wouldn't normally like it. I tend to change my speed of delivery and so forth when I'm singing a song, as part of telling a story or whatever, and it takes a hell of a clever accompanist to improvise along with that - it's much harder than accompanying the accompanist who is already doing that.

There is a big difference between playing a tune and playing an accompaniment, and a lot of excellent instrumentalists don't really appreciate that. The tendency is often to iron out the rhythym into something regular enough to dance to. Which is not traditional - but more important, it gets in the way of the song.