I saw Peter denounce that 'drugs' theory pretty convincingly at a concert in the 70's ... but with his later conviction on sex with a child, I am sure there are folks who'd see a more sinister sub-plot to the song.

I think it's plain and simple; a song about growing up, leaving behind the toys and fantasies of a child. It's a lovely song and doesn't need analysis or correcting!