On reddit tons of people are like “hurr durr it was caused by a configuration error that propogated down causing failed health checks and remaining non-updated servers got overloaded durr” and that’s the “techy” explanation.
Maybe that’s enough for MS IT guys and no shade but, with e.g. crowdstrike we knew pretty quickly exactly what the cause was - an array out of bounds read because of an extra malware definitions file (or something to that effect i do not recall exactly).
It is suspect to me that this outage went on for this long, despite how much it affected, and how the recovery went along so haphazardly it seems. They are probably not done with RCA yet but i also wonder if it was a more negligent fuckup they want to keep under wraps to avoid bad rep and e.g. UK 999 system choosing a different tech corp to work with next time as some european govt agencies have already done.
What do you think?
Infrastructure of this size is just code. It’ll spin up instances, route traffic and balance nicely.
They probably added a trivial timeout to their health check. Say, if an instance doesn’t respond within 60 seconds it gets cycled out.
Probably worked fine on Monday morning when they rolled it out.
It probably took them until Saturday night to remember that it takes two minutes for an instance to launch, so every instance will be cycled out before it tags in.
Just a wild guess about the kind of miss that might’ve caused it. I haven’t read it heard anything.
Microsoft offers voluntary buyouts to 60,000 employees as AI leads to efficiency push
My first thought when reading about it and how much was affected was “what if some of these outages are actually tests?”
It should be noted, though, that I don’t know shit.
Yep. They will never admit it though. Their alleged golden goose can’t be blamed.
Best scenario, in my opinion, they will admit the technical fault with not many details, whatever it is, but not that it existed because AI was forcefully fed to the system in a big scale.
And their long term solution not to let that happen anymore? MORE AI.
A few thousand missing engineers.
Also slop
The root cause would be relying on technology not old enough to drink to manage systems that are old enough to drink.
Makes me want to drink.
I’m going for a drink
Their status updates said they saw that a health check was pulling servers offline that were not unhealthy. The mitigated the health check but could not bring systems online without getting DDOS’d by all of the connections.
May is suggest “Microslop” v “M$”? They banned microslop on one of their forums so they really hate it.
Hackers
I’m of the (completely speculative) opinion that the US just experienced a massive cyber attack (maybe Iran, maybe literally anyone else as USA has a lot of enemies) and that we’re all being kept in the dark about it.
There were thousands of people involved in fixing this. There is no way MS could keep an attack under wraps if they wanted to.
“Never attribute to malice that which is adequately explained by stupidity”
https://en.wikipedia.org/wiki/Hanlon’s_razor
I personally believe it needs to be slightly modified to:
“Never attribute to malice that which is adequately explained by fucking idiots seeking profits.”
I disagree with Hanlon’s razor simply because stupidity and malice are often sold as a bundle.
Yep.
Funny how the FAA comms outage was the next day.
This will be considered Day 1 of the first all-out cyber war.
How would MS and the FAA silence the thousands of people involved in fixing these incidents?
NDAs, money, threats, I do not know, just throwing stuff at the wall right now
I’m not saying it was an attack…but if it were, they would never admit it.
MS wouldn’t have to admit anything. There would be hundreds, if not thousands, of engineers shouting the truth to the heavens.
