You can't deploy new environment variables if you can't build. Free and Hobby users were locked out for over 27 hours.
If builds are down, tell users to revoke at the provider level first and give an ETA for deploys. Don't tell them to run while tying their shoelaces together.
5/10
A few other unforced errors in the comms stack:
⚬ Two timelines: Saying you spotted it Oct 21, then later saying Oct 20 without explaining the gap (letting HN/X assume the worst).
⚬ Anchoring: Calling 100,000 workspaces "a subset of users."
⚬ The email: Sending a breach notice at 22:37 UTC that literally cut off mid-sentence.
6/10
Why does this happen to good engineering teams?
Because during an incident, a hundred people pile into the Slack channel and nobody owns the comms response.
Legal wants to say nothing. Engineering wants to explain everything. Meanwhile, an outsider's theory is upvoted to the top of Hacker News while the team rewrites a Google Doc again and again (and again)
7/10
Distributed security is brutally hard. Customers will forgive a permissions bug or a sophisticated attacker.
What permanently updates their trust weights is information latency: what you told them during the crisis, and the exact timestamp of when you told them.
8/10
If you run a platform that holds other people's secrets, put this in place before your bad week:
⚬ Pick one person to own the words (separate from eng).
⚬ Tell customers within an hour of telling vendors.
⚬ Give scope as a ceiling ("up to 1.85%"), never "a subset."
⚬ Warn customers before vendor revocation emails hit their inbox.
9/10
I run crisis comms for tech companies through SELF.
If you want to plan ahead, or you want someone on-call if it hits the fan, I'd love to chat through it with you:
https://www.thisisstudioself.com/capabilities/crisis
10/10
@Daojoan I realize you're describing a specific kind of situation here, but this is honestly one of the best summations of general human dynamics that I've read lately.