Why not just say that you disagree with his views on immigration, and explain how you disagree?

I have done, several times. The trouble is, I disagree with his views on immigration precisely because they're racist. Yes, it's a very loaded term, but I don't believe any other term fits (see also Oxford English Dictionary).