Spot, you're right about bad audiences, and having to do the gig because its your job to perform. But this thread is about When Everything Goes Right.
Norman Kennedy's interview in Sing Out! magazine so many years ago explains it for me: "You're only a link in the chain." Nine times out of ten, my favorite performers, when they talk about it at all, will say that the song/tale/etc is the important thing, not themselves as the performer. I've found this to be true even when the performer had written the piece!
Maybe this is why I've had a similar experience as part of the audience, usually when there's a chorus or refrain to join in on, but sometimes I find myself having "dived deep" (for lack of a better term) into the song or story without physically singing or speaking.
This ties in with your observation, KingBrilliant, because I've always found my best performances to be the ones where you might say I'm part of the audience as well as the performer; that the joy of doing the piece transcends the presence or absence of an audience.