I have recently had a glimpse of the other side of this. One of my other colleagues is highly dependant on Claude for everything, and has been working with a couple of contract programmers on creating some new telescope automation software for a couple of their projects. I got asked to help a bit on the hardware side and came away low key horrified.
While the systems they had created mostly worked most of the time it was apparent that none of them had any idea how it worked and could not do anything new with it without getting Claude to do it for them. At one point I asked what I thought was a trivial question, "how do I see the serial numbers of the currently connected hardware?" and the answer was "I'll ask Claude." What followed was 10 minutes of the chatbot trying random stupid things with a running commentary of fake post-hoc justifications masquerading as a thought process until it by chance stumbled upon something that seemed to work and some numbers came out. I pointed out that a quicker check was needed, so Claude was prompted again and eventually spat out a shell script that appeared to do the same thing.
This brute force automated trial and error approach to producing software had produced some superficially impressive results in a short period of time, but the whole thing seemed incredible fragile, utterly dependant on continued unfettered access to Claude in perpetuity and completely unmaintainable otherwise.
I remain convinced that If you generate code by throwing-shit-at-the-wall-and-seeing-what-sticks at scale then the end result will inevitably be a big mess.
