Login
You're viewing the functional.cafe public feed.
  • Nov 14, 2026, 2:58 PM

    @boxobark.ing @davidgerard
    OK, so… MIcrosoft have always had a soft spot in their own understanding of their cloud infrastructure. I was a CTO of a company that was using Azure to provide data to a fleet of devices, and one time we rolled out an update that could;t read the result of a query from Azure, so it would send the request again. We only rolled out the update to about 1% of the fleet before the load on the SSH authentication servers overwhelmed the entire Azure west European infrastructure because crypto is expensive.

    Anyway the important thing was that you couldn’t see the issue on any of the customer facing dashboards - if we looked on the client side we could see more than 90% of requests failing but on the Azure side all we could see was lower than expected traffic. When we reported the problem to MS (initially as one endpoint having an issue) it took them many many hours to realise what the actual issue was taking down their entizre network. And they only figured it out because we did some investigating on our side to see exactly where in the connection things were failing. They couldn’t even see their entire service was down (for all customers, not just us) - though they finally twigged when I explained it wasn’t just the one endpoint, but all of them.

    Anyway, this situation looks very similar in its response shape, but mine happened 8-9 years ago, so I wouldn’t be too quick to point the finger at AI here.

    💬 3🔄 7⭐ 19

Replies