Azure History

Share
10 views11/17/2026
The sordid history of Azure development - How Microsoft Vaporized a Trillion Dollars Inside the complacency and decisions that eroded trust in Azure—from a former Azure Core engineer. https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion

0 Comments

Be the first to leave a comment.

▸Show captions (47)
00:00 - 00:05Joined Azure Core in 2023
00:05 - 00:07Cutler's vision for Azure
00:07 - 00:10Zero human intervention
00:11 - 00:16Fully automated. Nobody touches nodes.
00:17 - 00:21Then they shipped it 48 weeks later
00:21 - 00:24So now we touch the nodes
00:29 - 00:31A lot.
00:31 - 00:34Hundreds of times a month
00:38 - 00:40Via 'digital escorts'
00:41 - 00:47$18/hr contractors on Pentagon nodes
00:47 - 00:50Some directed from China
00:50 - 00:56Hegseth: "a breach of trust"
00:58 - 00:59Fair.
00:59 - 01:01Anyway, our new feature
01:04 - 01:06Key exchange
01:06 - 01:08Eleven months in testing
01:13 - 01:14Rolled out to prod
01:14 - 01:16Two Sev-2s within hours
01:16 - 01:20Agent calling an unknown endpoint
01:21 - 01:27Retry. Retry. Retry. Forever.
01:35 - 01:36Logs filled up
01:36 - 01:393 days of logs became 2 hours
01:41 - 01:44I said halt the rollout
01:44 - 01:47Rollback took the weekend
01:47 - 01:49And half the next week
01:50 - 01:53Then nodes hit 100GB of temp files
01:54 - 01:56Leaked files
01:56 - 01:58Easy: run a delete script
01:58 - 02:02But the fleet tooling kept failing
02:05 - 02:07Took three months
02:10 - 02:11To delete files
02:11 - 02:13January to March
02:15 - 02:17Customers? They're fine
02:17 - 02:20Bad nodes? Migrate the VMs
02:21 - 02:28Rogue VMs, zombie VMs, leaks
02:29 - 02:30We cope
02:30 - 02:32Spare capacity. 24/7 firefighting.
02:36 - 02:39What if everyone scales at once?
02:40 - 02:42A run on the bank?
02:50 - 02:55Nobody knows.
02:56 - 02:58So I escalated
02:58 - 03:04All the way to senior leadership
03:04 - 03:09Some said noted. Some got defensive.
03:09 - 03:11Meanwhile, from the top
03:16 - 03:20Rewrite Azure in Rust
03:23 - 03:26Ship OpenAI's bare-metal SKUs
03:31 - 03:36Zero human touch, they said