To oppose that sort of thing is natural, and it has nothing to do with racism

I entirely agree. You're describing opposition to immigration on economic grounds - which isn't racist. WAV is expressing opposition to immigration on cultural grounds - which is.

I don't think it's wise or judicious to label other people as "racists"

If someone expresses what I think are Christian or Marxist or Tory views, I'm quite likely to comment that I think they're a Christian or a Marxist or a Tory. WAV has expressed what I think are racist views.