I used to use a bit of rubber foam when I had an open-backed banjo, but only when practicing. Fiddle players soemtimes use a spring clothes-peg on the bridge instead of a mute. I'm not sure exactly what goes on, but it "radiates" away some of the higher freqs (or higher freaks?) like a proper clip-on violin mute, only much cheaper. You get some interesting side-effects doing it on a banjo--it may bounce on the head, and will certainly shoot off at the worst possible moment--but a distinct lack of front-effects.

Steve