A non-racist could have lots of reasons for questioning immigration/emigration, such as population density, land rights, trying to have a multiple number of cultures living under the one state law, caring for young Spanish couples who are being priced-out of the property market by wealthy English and Germans - not content to just visit as respectful tourists, etc.

Poem 76 of 230: LAND RIGHTS

If there is a good thing
    From the Second World War
It's that most peoples learnt
    To conquer lands no more.

In Africa, Asia,
    And the Pacific, too:
Post-war independence -
    Steps only bigots rue.

But for some indigenes,
    Outnumbered much-too-much,
It has all come too late
    For liberty, as such.

So 'tis in Australia,
    And America's sites,
Where the best now, I think,
    Is to respect land rights.

From walkaboutsverse.741.com