As someone who has spent many years working in high load environments, this is not an uncommon pattern.

You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out.

And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more.

This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well.

It could be that the extra usage just pushed them past one of those capacity thresholds.

If I had to guess it's because Github is sitting on top on infrastructure held up by toothpicks and duct tape

GitHub Will Prioritize Migrating to Azure Over Feature Development - https://hackertimes.com/item?id=45517173 - October 2025 (63 comments)

Hotmail used to run on bsd, and they tried to move it to WinNT, back in the day. Same thing. Disaster. And it was held up as a reason to never use winnt for serious work, for years.

Microsoft just always shoots themselves in the foot.

The nobody-got-fired-for-choosing-Microsoft crowd. And they don’t look few and far between.

Maybe this really depends on region, but I can't say this is true. I've experienced tons of hardware failures that have caused instances to throw weird errors, instances to randomly stop, instances to randomly disappear (along with their corresponding EBS volume).

Standard EBS volumes are only 99.8% durable. If you run a lot of instances on EBS, some of them will disappear.

GCP meanwhile has balanced zonal PDs (not even regional, zonal) at >99.999% durability.

I've never lost a disk on GCP, I've never even had an instance stop once without me telling it to stop.

Azure is absolutely a hot mess though. Had a VM a client was paying for backups on. Couldn't restore the backup because they changed generations of VM platforms too many times, literally no way to restore it. Had to get their support to eventually give me a .vhd that magically appeared in a OneDrive share a few days after asking about it.

Probably both. But it still throws a thorn into the theory that LLMs are about to replace software engineers any day now.

You'd think they could LLM-code their way out of this situation easily if LLMs were the software engineer replacement they are being marketed as.

Web search, steaming, high frequency trading, and many other systems are resource demanding, and keep scaling. Whether to handle the surge due to bots or wider adoption. But GitHub can't scale git? It isn't even git failing.

Incompetence in leadership is what makes a tech business technically unable to meet growing demand.

Is there a single Google service that has ever had reliability this bad? Facebook? Instagram? TikTok? Apple? Those companies all run hugely scaled write-heavy platforms. Why is GitHub so uniquely unreliable?

Me and 1 billion of my best friends can upload 5GB 4K videos to iCloud Photos and live stream to Facebook all day long but GitHub who is owned by Microsoft the second largest cloud computing provider on the planet can’t handle 180 million users pushing code and running pipelines on VMs?

I'll give you an example: I pointed Codex at one of my repositories asking it to implement some features. Very quickly, it ballooned a 3 minute CI workflow to 30 minutes per commit, and the number of commits it started making increased 10 fold. So we're looking at a 100x increase in usage just from myself.

You factor that into the fact that all public repo get access to this same compute, basically unlimited minutes, up to 20 parallel tasks and 6 hours per request. Now because of AI, every small little weekend throw away side project is running a full dev ops shop. And the pressure never goes down because the AI just start a cron job to run these systems every single day, whether anyone is even looking at these repos including the authors, who may themselves just be bots.

And the systems are sophisticated: parallel builds across windows, mac, multiple linux distributions, multiple architectures, embedded targets, web targets, full feature matrix, the works. The default MO of these agents is to do as much testing as possible without any regard for resource constraints, and GitHub offers them unlimited resources, so it's like an addict meeting a dealer.

So in a sense, maybe being attached to Microsoft is the problem. Or at least their unlimited wallet is -- Microsoft is the enabler in all of this.

It still just sounds like a really vanilla "horizontally scaling VMs" problem, especially for yesterday's incident that was focused on pipelines.

If this is a "GitHub is giving away more capacity than it has" problem, that's easily solved by rate limiting and queuing.

This is GitHub's explanation:

> During the incident, some Actions Runner Controller (ARC) runner pods became stuck in an idle state. Affected users can delete those pods using kubectl or redeploy their Actions Runner Controller application. ARC will automatically create replacement runners.

> The next releases of Actions Runner and Actions Runner Controller will include an automatic recovery mechanism, preventing the need for these manual steps in the future.

That sounds a lot more like a major architectural flaw and a really obvious oversight.

I don't think this is true. Take LLMs for example, you need GPUs to serve them and these are in short supply at the moment making it hard to meet demand. I don't think this is necessarily the fault of incompetent leadership.

With GitHub, we threw that away. It's centralized on steroids. Now we're all depending on one platform that has to be massively scaled to deal with the massive load of serving almost every major software project on the planet. And when it is down, we all notice.

I see this repeated a lot and IMO it's simply not true.

The main decentralized advantage of Git is that you can continue to do VCS operations without network access or access to the remote host.

Most of what Github does is managing collaboration. The only thing we've "thrown away" by using Github is the email based workflows or directly pushing git branches to various hosts. But unless you're gonna have team members SSH into each other's machines you'd still need a central repo somewhere.

https://www.reddit.com/r/kubernetes/comments/f2bcsu/parodyfu...

Seems pretty conclusive. Very similar story when they bought skype.

It got worse around April/May this year, while that graph stops at February.

I have an Ops background and I strongly suspect they were given a stupid timeline for the Azure migration.

I've got to believe Microsoft have decent Ops people but the management wanted to move faster than was reasonable and screwed it up. Move one thing at a time and double check it all works and you can do a migration like this.

* There's a huge disconnect between the actual product and what was sold to the buyer. It could be that the product was a borderline fraud, or it could be that the product was actually much better quality than the expectation on the buyer's side, but the buyer isn't interested in most of the product.

* Fear spreads in the acquired company that their product will be discontinued or reshaped into something else.

* The pay is good, probably much better than before the company was bought.

* Many internal teams end up lacking real purpose and try to insert themselves into every new internal project only to obfuscate their irrelevance.

This leads to some pathological developments, where the teams previously working on acquired products start doing a lot of useless, for-show work. Internal initiatives sprout like mushrooms after a summer rain, but they are all plagued by very broad (and mostly irrelevant) team involvement, duplication of existing products / services, and fear of being discovered. And while there are plenty of such initiatives, their role is to be a superficial distraction. In reality, everyone is afraid to touch the old code or do any sensible integration because it could lead to blanket firing of a lot of people. The middle-management behavior becomes a sort of exchange of favors, where everyone is afraid that the other can blackmail them into losing their job, and so everyone is trying to be extra nice by offering a slice of a pie to another manager.

What this leads to, in reality, is insane inertia (often despite highly shortened software release cycles), astronomic amounts of unmitigated tech. debt, opacity in communication with management, persecution of those who genuinely want to improve the system.

Based on this, my prediction is that Github will not survive. Just like Skype didn't. Somehow or other, Microsoft will find a way to replace the product with... MS Outlook with a new skin.

I always figured these problems were directly related to the migration.

I'm sorry but this made me laugh out loud. That isn't true at all, this has been going on for years. This conversation[0] from six years ago has discussion about the outages starting to become much more frequent in December 2019. It has never gotten better in that time, it's just continually degraded.

[0] https://hackertimes.com/item?id=22935941

Most people work around it by self-hosting github (which has other problems but uptime aint one).

If you want to have a horrible time, try reading some of the runner code. It's early 2010s-style Windows-First, MSFT C# crud that has trouble not racing several threads to inconclusive status codes.

Its common knowledge that Win11 needs 32gb RAM, but im glad to hear that Microsoft is aware of it.

Looking forward to this 8gb variant :)

So they will make Windows run on 8GB because people can't purchase cheap RAM anymore, and they want to stop people migrating to Linux.

This is not directly caused by AI slop, only indirectly through RAM prices.

With what to show for it? If GH did 10x in volume/git commits, it's all LLM sloppy-pasta. Where's the 10x productivity? Where are the amazing apps?

If your platform can't handle the use patterns of AI, then perhaps don't go around telling everyone to use AI for everything? It's a self-inflicted wound, you could also just not do this. Too bad Microsoft has bet its future on AI not being a giant bubble, huh?

That plus migrating clouds is insanely difficult to manage. They're almost certainly drowning in traffic and trying to keep up.

Jokes aside, yes, quite correct. Even frontier models like Fable have very obvious limits.

I'm really kind of surprised they let us do that - like, why didn't they just have you upload the binaries after building on your local machine?

So, from the sales point of view, Github Actions was, at the minimum, an alright idea. Not brilliant, but quite obvious and expected. From the engineering standpoint, however, this is a disaster on many levels. But, that never stopped Microsoft before. They don't try to win the market by making an objectively better product, their tactics are and always have been to make a product that can claim (with an asterisk) to be able to do a lot of things the customer wanted only to discover afterwards that those promises were phony.

You can do that already with GH Releases. Actions is if you want CI/CD managed by GitHub. And you can also use your own machines via self-hosted runners.

My only guess is there's some circular dependency they want to avoid by keeping Okta.

Though I'm sure some of the blame can go to internal slop code.

I think it is a natural progression in technology. Failure rates were high for initial aircraft designs and safety improved over time, for instance.