you can't "just have a really good test suite" your way out of this. security issues are caused by violation of properties you never even thought of, by non-obvious and often indirect interactions of things nobody realised were connected. your tests will not catch this
most test suites are actually not very fine constraints on the system under test; the idea that "any program that passes the tests is acceptable" is nonsense, even more so when you're also generating the test suite
I personally know about 1 person who I think is good enough at testing that I'd let them make this argument
multiple things can be true at the same time. it is true that un- or underpaid people having to deal with complex emergencies under tremendous pressure utterly sucks. it is also sucks that that pressure is a strong incentive to deploy under-reviewed code
"you could do this with AI, therefore AI is the only viable way of doing it" is not true, it is a trap that asserts the only way to deal with problems is blind panic, to surrender control and responsibility for your own work
@jcoglan in some ways it feels arguably isomorphic to the whole "the only solution to violence is a good guy using violence" debate
@jcoglan I completely agree with you, but this PHP exploit is both dangerous and disclosed in a pretty bad manner.
As a sysadmin, this is an exploit at the scale of Shellshock, and it should have been managed much more carefully.
I can't blame admins of extremely large installations reach for the fastest and biggest gun they have at this moment.
We have patched a ton of systems (w/o AI) and reinstalling some right now.
@jcoglan and I’m guessing they wouldn’t
@jcoglan I saw this abstract (warning: i did not read the paper) recently touching on LLM-based decompilation projects assuming that you can be certain the decompilation worked if it passes tests which apparently this study shows is not the case.
Oct 22, 2026, 2:44 PM@jcoglan Well-said. This comes up a lot in embedded programming: you cannot "test in" quality. (Quality here is an abbreviation for reliability, predictability, performance, value/cost, etc.) Tests can *verify* quality that was designed and socialized and implemented, but tests cannot add quality.
The old "necessary, but not sufficient" statement, you know? If we let ourselves believe that tests == quality, we have fucked up at a core core level and are lying to ourselves.
@jcoglan I agree but I also think not having any test suite at all for a project that's switching to vibe coding anyway is going to be worse...