"I often wonder why tolerance is so often a one way street ?"

easy... the good guys are tolerant..

the bad guys aren't.....

left or right, religious or non believer..

ranters and zealots [and the sneaky silent noxious colluders..] are life force sucking socially dysfunctional arseholes..

Sounds to me like you just described perfectly the people who attacked the blacked-up Morris-Dancers, PFR. In a civilised society, nobody should be attacking and abusing anyone.

'Tolerance' is accepting the dancers' explanation of the reasons for, and meaning of, blacking-up.

'Intolerance' is attacking and abusing them, despite their honest assurances that blacking-up is in no way intended to mock or offend dark-skinned people but, rather, is a reference to disguising oneself in order to avoid accusations of begging.

Sometimes, it could be argued, 'offence' and 'intolerance' are precisely the same emotion.

IMHO, of course. YMMV.