No, we didn't/don't have control of immigration as the EU insists on "freedom of movement"

Yes we do.

Firstly, non EU immigration has always been higher than EU immigration. That will not alter one iota.

Secondly, the rights of the UK to control it's borders are not undermined by the freedom of movement for EU citizens. I suggest you read the following to get the facts right.

Can the UK Control Its Borders if It Remains in the EU?

The complaint about restricting the rights to work and live in the EU by UK citizens is still valid as leaving the EU will make it more difficult. Not impossible, just difficult.