What Wittgenstein was attacking was the idea of the mind as being full of unvocalized language, your thought processes being a continuous monologue in a sort of internal language which would occasionally get translated into outward utterances. The Private Language Argument was only one of several different approaches he took to showing that that theory didn't make a lot of sense.

His starting point (insofar as his later work has one) was the way language arises as a means of structuring more complex social interactions than a group of people could achieve without it - cooperating on a building job, in the first example in the Philosophical Investigations.

Idioglossic languages are different from the private languages envisaged by epistemologists, since they're often in principle teachable and hence only "private" in a very weak sense. (There is a deeply unpleasant example in Flora Rheta Schreiber's book "The Shoemaker", in which a paranoid schizophrenic killer forced his whole family to chant his own verbigerations derived from Latin Catholic ritual before going off to torture and murder a family chosen at random).