“We’re seeing positive results for server stability after applying an upstream authentication/routing configuration component patch to a portion of our infrastructure.”
I am not a professional computer toucher but what, typically, sits upstream of the server?
Also “routing configuration component patch” sounds awfully close to “it was DNS”
“At this point, we are observing marginal success” is a line I am filing away for future reference.
“We’re continuing to monitor activity across regions to evaluate server status and determine actions that can be taken to improve capacity. This includes reviewing server performance and confirming that developed patches have been applied and working as intended after infrastructure reboots are completed. At this point, we have observed marginal success.”
Marginal success.
Looking at it a different way…
Businesses: Congratulations on successfully completing your GTA6 launch day training course.
LinkedIn is still up, FYI.
Not one single iota of understanding of what’s broken or why.
“As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.”
My I’m not hacked message is raising a lot of questions answered by I’m not hacked message.
Also, “once mitigated” is doing a cubic fucktonne of heavy lifting.
“As part of our ongoing efforts to advance the recoverability of our frontend servers, we’re evaluating traffic controls that can be applied to increase capacity and stability.”
L M F A O.
“Additionally, we’re querying for machines in a bad state to perform a force sync for our recent network change, which may improve service availability. However, we are encountering issues with the machines being stuck in a bad state, preventing our changes from syncing as expected.”
It’s so completely fucked we can’t even turn them off and on again.
“We appreciate your organization’s patience and understanding while we work with urgency to resolve this incident.”
:tgif:
“We’re continuing our efforts to improve server stability to improve traffic flow. This includes adjusting traffic protocols, redistributing traffic, and applying strategic traffic blocking measures.”
Traffic blocking measure proving insanely successful for the past many hours.
“We're observing partial recovery across regions in which we've applied these adjustments; however, these actions aren't resulting in sustained recovery”
It’s bleeding out less now!
I love that we're, what, 12-15 hours deep and it is very clear they have absolutely no fucking idea what has gone wrong, or why. Nor do they have any idea how to fix it.
I love this.
I love computers.
"As part of our investigation into the sustained traffic management imbalances, we’re reviewing traffic flows and affected infrastructure to identify opportunities to optimize routing. This includes verifying the reasons for service failures so we can develop specific actions to improve stability."
Back to square one.
Half-serious question: has there been a noticeable decrease in the amount of spam emails going around today?
Actual headlines on ABC News
#Microsoftdown situation could be described as fluid
@MattHatton official instructions: “if you need to use Outlook, just head down to the pub instead”
“We’ve monitored the status of the changes applied to improve serviceability and confirmed that change is not taking as expected.”
lol and, may I say, lmao
"We’re applying infrastructure capacity improvements where possible in an effort to help support traffic in regions coming online. We’re continuing to monitor the results of changes applied so we can identify strategic actions that can help to improve service health across all affected regions."
1) MOAR TUBES
2) "We are watching the tubes closely"