How then can you categorise this as 'racist'?

When you are an ideologue most of those who disagree with you blur, in your mind, into a single broad category like racist, right winger, Islamophobe etc.