"if the location bothers people, put it somewhere else,"

Very much the same as "if sitting in the front of the bus annoys people, what's wrong with the back of the bus".

In one sense perhaps sensible, location doesn't matter too much in absolute terms. But backing down in the face of bigotry damages a society very deeply.