Meanwhile, back at the subject, there's a seriously long article by Rob Slade, towards the end of Risks Digest 34.83*:

Capability Maturity Models and generative artificial intelligence

[ The first three steps (of five) in his framework are "chaos", "repeatable" and "documented". Slade argues, in detail, that LLMs aren't past the first step, in which "[w]e don't know what we're doing. Not really". ]

[ ... ] But nobody's got a really good handle on this. (The way you can tell that people really don't have a good handle on this, is that the large language model companies are spending so much money, all over the world, lobbying governments to try and prevent the governments from creating regulations to regulate generative artificial intelligence. If the genAI companies knew what they were doing, they would have some ideas on what kind of regulations are helpful, and what kind of regulations would help make the industry safer, and what kind of business and revenue regulations might affect. But they don't actually know what they're doing, and therefore they are terrified that the governments might (probably accidentally) cut off a profitable revenue stream, or even just a potentially useful function for generative artificial intelligence.)

* Apologies for nonstandard site. The official RISKS site has been having hiccups for some months.