If he was talking only about the influx of Irish and Welsh and Scots, would it be racist then?

Yes. He's saying that only one national/ethnic/cultural group belongs in England, and that the presence of other groups is detrimental to English culture. In other words, he's dividing people into racial groups and saying that those racial groups should stay separate.