As I age, I find myself not trying to be polite any more. If I gotta go, I gotta go. I try to leave between songs and without making a disturbance, and to stand at the entrance to the rooms when I return, until there's a break between songs.
I think that in general, it's rude to move during a song, but OK to move between songs if you must.

We got a hot-tempered Irish priest in our parish in about 2005. The choir members stand during the entire Mass, except for the sermon - and a number of singers would head for the bathroom before his sermon because it was the only break they got during the 1-hour Mass. A couple of times, he noticed; and he would stop his sermon and say that we'd wait until people got back from the restroom. It never happened when I had to go, but I was ready to say, "That's OK, I have irritable bowel syndrome and I really can't adjust to your schedule. You go ahead without me." I am the only person in the parish who would be likely to respond like that. But the bastard retired a couple years ago and I never got the chance. I did yell across the church once about how "fucked up" he was, but only he was there to hear me. Probably that was a good thing.

But anyhow, those who would pass judgment should realize that they may soon be in the same situation. If ya gotta go, ya gotta go.

But if I'm performing, I go to extremes to avoid having to leave in an obvious hurry.

-Joe-