Pah! A mere 5. I took 7 tests before I passed ....

I think your instructor is right and it is largely confidence. Certainly, my later tests were worse than the early ones.   In one of my tests I stalled in the middle of a junction and had to get the examiner to drive me out of it.

I've now been driving some 20 years and never had an accident or so much as a parking ticket. I think what happened in my case was I stopped trying to pass the test and just drove. That did it.