Nov 5, 2026, 8:43 PMThe staff engineer on my team has an all too common belief that you want alerts that tell you to "look at stuff when you have time"
When has anyone in tech in the last 15 years ever had a spare second to look at some random alert that fires 80 times a week? Absolutely never. The only correct solution is to tune every alert to make it immediately actionable when it fires - ie only you, the human, can do non trivial steps to remediate.
Nov 5, 2026, 8:45 PMAlso while I'm at it, the entire premise of runbooks is backasswards. If you can specify the steps to take so clearly that you can just follow them written down then you could write them in code and a machine could do it better.
Don't have the facilities to turn your runbooks into code? Sounds like you're ready for the first step up the operational ladder to victory. Making that possible.
Runbooks are programs with the worst possible calling convention: copy paste.
@amy Run books as temporary "Hay this is a known issue, here's how to fix it, we've got a fix coming but its a sprints worth of work" fair enough. But I've never seen them used that way.
They always seem to be another name for tech debt.
Nov 5, 2026, 9:07 PMA dishonorable mention: I have had exposure to so called "automated SRE" systems (LLMs.) First, as someone who has held more titles in my career than I can count, one of them being SRE I think these things are repugnant for their devaluing of the discipline that made the infrastructure LLMs run on possible at all. SRE is the discipline that took computer science and bridged it to the real world mechanics of risk management.
And, I have seen systems like these do a shockingly good job at identifying systems problems that I already knew the answers to. Also comically wrong. But more right than wrong on the whole of it.
So, take that for what it's worth. Don't use them. You are robbing yourself of skill.
@amy Missed opportunity to allow people to use their words to request something and then, using small LLM trained on the individual’s own words used as a semantic layer (abstraction) and have that translated to tech request using an expert recipe with ingredients and instructions/process/workflow (AKA skill.md).
But instead big tech promised omniscient Oracle and delivered destructive and unsustainable, irrelevant idiot
Nov 5, 2026, 10:12 PM@dahukanna that's really what I wish we'd gotten from language models - small specialized things that were deeply personal and reliable in their behavior.