Login
You're viewing the fosstodon.org public feed.
  • AkaSci 🛰️AkaSci
    Nov 14, 2026, 10:45 AM

    This is also consistent with why restarting servers at Microsoft did not help.

    As servers were restarted, you can bet the load balancers routed the surge of client connection attempts to the least loaded servers, i.e., the newly restarted ones. The servers remained least loaded since their connection request queue built up and they kept processing timed out connection attempts.

    This is a case where simply unplugging and plugging a device does not work. 🤔

    21/n

    💬 2🔄 4⭐ 12

Replies

  • Taran Rampersad, EIEIOknowprose@mastodon.social
    Nov 14, 2026, 11:00 AM

    @AkaSci the fact that some of us knew about the retry storm while watching them fo it shows just how far Microsoft is behind the curve.

    Networks propagate. Let's tie a bow on it and run Linux. 🙃

    💬 1🔄 1⭐ 1
  • Dianealienghic@timeloop.cafe
    Nov 14, 2026, 3:03 PM

    @knowprose @AkaSci

    Huh so I guess I helped Microsoft recover.

    I turned my exchange account off as I was tired of the error messages and thought the retries were making unhelpful load

    💬 1🔄 0⭐ 1
  • 💬 0🔄 0⭐ 0
  • siliconlurkersiliconlurker@mstdn.party
    Nov 14, 2026, 12:37 PM

    @AkaSci The catch is what “least loaded” measures: few established connections can hide a huge handshake backlog. That makes the load-balancer metric crucial to this hypothesis. A recovering server may need a capped arrival rate, not just a passing health check, before it can safely rejoin.

    💬 2🔄 0⭐ 1
  • MarkDmarkd@hachyderm.io
    Nov 14, 2026, 12:47 PM

    @siliconlurker @AkaSci Restarting servers with a backlog is not a new problem. Any load balancer built in the last three decades has plenty of capabilities available to deal with the problem, such as throw away the connection rather than pass it on to a server once certain thresholds have been met or exceeded.

    💬 0🔄 0⭐ 0
  • jbjb@masto.hackers.town
    Nov 14, 2026, 12:50 PM

    @siliconlurker it could very well be that this situation was never modeled and there’s no recovery plan for it.

    I’ve seen similar in other F500 companies. Hell, even AWS screwed themselves in a similar way.

    @AkaSci

    💬 0🔄 0⭐ 0