I don't agree with you, Lox. It really is the other way round!
When I was a kid members of other ethnic groups began to appear in my home town - a bit strange at first but I soon got used to it and it soon became obvious that these were just people - just like me. Fifty years on and I see kids of all ethnic backgrounds getting along together - if anything they appear to be even more tolerant than I was.
But would it be the same if those in authority started to demonise representatives of different ethnic groups - as happened in Nazi Germany in the 1930s, or the Deep South in the first half of the 20th Century or in the former Yugoslavia in the 1990s? And there are many counter-examples, of multi-ethnic societies, where racism wasn't institutionalised and wasn't an issue (the Ottoman Empire is one such example).
To repeat my position. It's probably true that many people have an in-built fear of strangers - maybe it's even part of 'human nature'(?) - but racism is largely a top-down phenomenon and elites have to work quite hard to get people to seriously discriminate against their neighbours. And when they do so they usually have an economic motive.