Be careful of what you ask for.
I sang John Barleycorn at an agricultural show in the beer tent a few years ago and one of the organisers came up to me later and accused me of singing the BNP anthem...
Anything and everything can and does get hijacked. Presumably if I sang a MacColl song, that makes me a Stalinist.
On balance, explaining the "disguise and heritage" aspect of looking like one of the black and white minstrels is long winded. Perhaps accepting that 500 years ago, you wouldn't be mistaken for ripping the piss out of the family down the road might help the argument slightly? After all, far more recently than that, we were hanging a monkey for being French.
The swastika has ancient peaceful connotations for a number of religions but you don't see temples in The UK adorned with them*. Common sense can exist even in irrational places, it seems.
*That said, when in New Zealand ten years ago I saw the cathedral in Christchurch, since made derelict by earthquake, did have swastikas carved in one area, representing the multi cultural origins of the occupation of the South Island.