The point is that someone trained to play only from written notation is in much the same position as someone who can only speak when reading aloud from a script.

Ideally we should all be able to do both; speaking as a near-illiterate where staff notation is concerned, though, I am glad to be able to hear a tune, play it, and improvise on it, without having to mess about with sheets of paper. We learn to speak by hearing, and learning to read is a useful thing that comes later. The same is, or should be, true of music.