Nov 4, 2026, 10:38 PMAlso people fundamentally do not understand any part of detection. They don't know why they're detecting, what they're looking for, nor how to do it.
And I've tried for years to explain that more is worse and nobody listens, so I stopped trying.
Still gonna write my book as a vanity project someday.
Nov 4, 2026, 10:50 PMAlso "catching problems before they happen" is a complete fantasy. It is not possible. The best you can hope for is to catch problems early enough for your objectives. Because definitionally, if the data indicates a "future" or "growing" or "small" problem. Then, well, it already exists and it's just within your acceptable tolerance for problems.
And no, machine learning does not change this, and LLMs especially don't change this. ML at best, if you have really really good data, will catch more novel problems than hand-tuned detection. Which may or may not be beneficial depending on many factors.
The best system I ever ran, where I had full control over the alerting had three alerts. All of them paged. None of them paged if there wasn't a real, actionable, immediate problem that needed resolution as soon as possible.
This system had an annual six 9s of reliability.
The belief that you need "lots of" detection to make reliable things is horseshit.
Nov 5, 2026, 8:43 PMThe staff engineer on my team has an all too common belief that you want alerts that tell you to "look at stuff when you have time"
When has anyone in tech in the last 15 years ever had a spare second to look at some random alert that fires 80 times a week? Absolutely never. The only correct solution is to tune every alert to make it immediately actionable when it fires - ie only you, the human, can do non trivial steps to remediate.
Nov 5, 2026, 8:45 PMAlso while I'm at it, the entire premise of runbooks is backasswards. If you can specify the steps to take so clearly that you can just follow them written down then you could write them in code and a machine could do it better.
Don't have the facilities to turn your runbooks into code? Sounds like you're ready for the first step up the operational ladder to victory. Making that possible.
Runbooks are programs with the worst possible calling convention: copy paste.
@amy Run books as temporary "Hay this is a known issue, here's how to fix it, we've got a fix coming but its a sprints worth of work" fair enough. But I've never seen them used that way.
They always seem to be another name for tech debt.
Nov 5, 2026, 9:07 PMA dishonorable mention: I have had exposure to so called "automated SRE" systems (LLMs.) First, as someone who has held more titles in my career than I can count, one of them being SRE I think these things are repugnant for their devaluing of the discipline that made the infrastructure LLMs run on possible at all. SRE is the discipline that took computer science and bridged it to the real world mechanics of risk management.
And, I have seen systems like these do a shockingly good job at identifying systems problems that I already knew the answers to. Also comically wrong. But more right than wrong on the whole of it.
So, take that for what it's worth. Don't use them. You are robbing yourself of skill.
@amy Missed opportunity to allow people to use their words to request something and then, using small LLM trained on the individual’s own words used as a semantic layer (abstraction) and have that translated to tech request using an expert recipe with ingredients and instructions/process/workflow (AKA skill.md).
But instead big tech promised omniscient Oracle and delivered destructive and unsustainable, irrelevant idiot
Nov 5, 2026, 10:12 PM@dahukanna that's really what I wish we'd gotten from language models - small specialized things that were deeply personal and reliable in their behavior.
@amy please write that book. I would love to read it. I have not stared into the abyss but I have pondered the nature of measurement, a lot, and I am sitting here nodding delightedly at everything in this thread. 💚