About the only argument that is probably true , is that the value of property will go down , which in fairness is a concern if you need tt move. But it happens in lots of places. Near us a former old peoples home is housing immigrants, and locals felt threatened, for no good reason , so far at least.