"The only "restriction" that can be put on EU migrants is the need for an EU passport."
No it isn't and the the fact you are studiously ignoring is that it works both ways and British people can step outside of 'fortress Britain' to seek work and expand their horizons, should they wish - one of the greatest advantages of the E.U. to the British people.
Of course, they could be left to stew in their own juice in Britain's boot camps and on the unemployment lines - (1.6m at present time - and this doesn't include those forced to take unpleasant and inappropriate work in order to stay alive).
One of the great differences between British and Irish youth is that so many of the latter have chased work all around the globe, acquired skills, languages and experience that I have seldom encountered in those back home - and I'm not talking about the ones who have had the benefit of a higher education and are looking for careers.
The end result is a broader-minded, far more internationally tolerant group of young people.
Emigration for work should never be forced on anybody, but it certainly is an advantage when offered as an opportunity.   
".that amounts to hundreds of millions of people who have the option to come here if they wish"
"Research conducted by the Migration Policy Institute for the Equality and Human Rights Commission suggests that, between May 2004 and September 2009, 1.5 million workers migrated from the new EU member states to the UK, but that many have returned home, with the result that the number of nationals of the new member states in the UK increased by some 700,000 over the same period.Migration from Poland in particular has become temporary and circular in nature."
SECRET STATISTICS
Jim Carroll