Hey Raggy. I think I would have felt inclined to stop altogether and "Apologise" to the person..."Sorry" but I find it very difficult to play with an "UNREHEARSED" accompaniment, would you please allow me to do this on my own"?
That way you save face for yourself and it is also not a put-down for them (although they may feel like it is).
Best wishes.