Or you could lightheartedly apologise for being a spoilsport but say unfortunately it tends to throw you off when people join in? Or, you haven't mastered the art of jamming yet and need to play solo so you can keep your concentration. Or something..
I can't imagine anyone taking umbridge if you tactfully own the fault as your own, so to speak.