I think I would just stop and say 'I'm sorry but I need to set my own rhythm to this and, my fault, I know, but I can't keep to the one you are setting - do you mind if I begin again on my own...'

If they contiued after that, then the gloves would come off....and I'd tell them to play it on their own.