A big issue with LLMs:
It makes some people believe they are more competent than they are.
Example: this company sent me some spam. At first I thought it was a scam. Then I dug into the company.
And I realised: oh. No. This company is not a scam per se. No, these people actually believe their own bullshit.
The CEO ran a web server as root. It got owned. He posted about it on LinkedIn. He used Claude to "fix" it. It was not fixed.
Now you see, before LLMs, if you didn't understand a domain, you'd hit a wall very fast. Documentation would use terms and concepts you didn't understand. Tools would require interpretation. You'd have to ask somebody more experienced. That friction produces a very useful insight: "Oh. I don't know this domain very well." So you'd find someone who did, or learn.
The thing about LLMs is that they can remove an enormous amount of that friction. That friction is not noise, it is signal. That friction is telling you that you need to learn the domain you are working with. Removal of this friction is a very, very bad thing.
The LLM can hold your hand, do a lot of heavy lifting for you, use a lot of tools, run a lot of commands, and make things appear to work.
And the thing is, all of the individual outputs along the way might be pretty good and it might look correct.
The issue is in aggregate. Before you know it, you've built a house out of sand.
And worse yet, people can emerge from the experience with a dangerously incorrect higher-order conclusion: "I am now a domain expert in this."
They are not a domain expert. They understand absolutely nothing about it. The LLM did everything for them and it's wrong. And worse yet, they haven't reviewed it, because they lack the competency to review it. Therefore, they assume it is fine. But it's all incorrect. A true expert would see that it is wrong, and correct it, or push back on the LLM, or do it themselves.
And this is how you wind up running web servers as root and getting owned.