Does anyone have any ideas on how to keep the levels down so that all the instruments can be heard? Or experience of successful sessions of this kind?

I have experienced people playing without listening to themselves, and without concern for the resulting sound of the ensemble. Professional musicians generally play at a compatible volume because it would make them look bad if they were unable to blend in. That said, I've seen people who are essentially professionals take the attitude that it's not their problem and the sound man should look after that. This does make some sense in environments where even the drums are amplified and the performers cannot hear how loud they are up front. I don't think folk and traditional performers are likely to have that problem though. :)

My take is that the problem stems from not playing with your ears. It is in fact quite common for people to be so preoccupied with what they are doing that they don't hear what others are playing. I generally only improvise so obviously I have to listen to others before I listen to myself - so to speak. People who use memorized parts can "barge ahead" without concern for the overall effect.

In the odd situation where I've encountered a problem, I've either recorded with a single mike and played it back so the "mix" can be heard by all, or brought out my sound level meter. The latter is actually a good idea because everyone can take turns adjusting their volume to an agreed upon level. This is much like using a tuner to adjust to an agreed upon pitch.

I used to play recorder in a duo with an electric guitarist. There was never a problem with unmatched volume. I'm sure if you asked that guitarist about it he'd say "no problem - I've got a knob here that I just turn."