As we have concluded, we have a problem of bad guys using GenAI to find weak spots in the infrastructure. Closing our eyes to it doesn't sound that smart, and before we come up with systems that are structurally more resilient, we need to act _fast_. IMO a responsible vendor[3] needs to monitor for issues or even proactively try to detect them (even with help of a)). The question is what follows and what are is the emerging system dynamics. All that follows is a hunchwork -- don't take it uncritically :)
1. Using GenAI to implement solutions and to communicate problems, without trying to have a deeper understanding of the issue, deepens the problem by making the system more complex[4] and more prone to obscure issues not easy to understand by people. In other words, the resulting system is harder to reason about and make structurally safe, defeating the purpose. Of course, since issues are harder to fathom by people, we are left relying more on the automation creating, from the perspective of GenAI vendor, a very lucrative protection racket.
2. The alternative is to take a slower path. After the problem is reported the time is spent to understand the issue and solve it by a person. This way we avoid expanding the cognitive ceiling (or cognitive horizon if you like) and it stays within possible human grasp. That leads to tighter systems and better understood mitigations, sometimes maybe in killing features that can't be safely implemented without skyrocketing the system complexity. This slower path leads to a more sustainable and secure software, so in the long term it carries you farther.
As of the immediate holes (like the recent PHP thing) -- I don't really know what is the good immediate approach, certainly PHP foundation being underfunded and underpinning the internet is part of the problem though. Perhaps some middle ground - slap a fix, then rewrite it from scratch? But that, again, requires funding. Wealth and corporate taxation could alleviate the issue :).
[3] Note that there is a huge difference between a vendor who makes money on their product, and a person who does things in their spare time - either fund the latter, or don't expect them ask "how high" when the team zero asks them to jump.
[4] See https://maevyn.codes/decoherence -- a fantastic piece of writing.
2/2