Aug 3, 2026, 5:10 AMIs the test suite an audit or a target? Is there such thing as a comprehensive test suite for software?
"when an AI agent's instruction is "go make the failing test pass," the test suite stops being an audit and becomes the target. A human team passing 46,000 tests will pick up understanding as a byproduct of the work. An agent loop passing them just proves the tests pass. It's the same green checkmark, but it carries less information than it used to."
@alexanderbird And we all know what happens when something becomes a target, don’t we?
Aug 3, 2026, 2:41 PM@thirstybear Yup!!
This happened at work this week (hybrid environment where some folks code with "AI" tools and some don't). We were discussing some of the large (2kloc!) files cropping up, and chose to add a fitness function / architecture test related to file size -- we constrain the max file size at a few percentiles (p95, p90, p50, or something like that).
Then someone kicked off a process to automatically split files into chunks, ratcheting down the file size test.... and now we have the 2kloc spread between 10 files with implicit dependencies between them (Python mixins that only work if they're all mixed in together), and the IDE ctrl+click no longer navigates between dependencies.
Not an improvement.
But the metric looks better!
🙈