"Most of us seem to broadly agree that unlimited immigration would not be good, and it is just the level that we are disputing?"

Also I would wager that most of us accept that a total absence of immigration would not be a good thing either for the economy or for our culture.