AWS ME-CENTRAL-1
66 views3/2/2026
Mar 01 9:41 AM PST We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire.
0 Comments
Be the first to leave a comment.
▸Show captions (49)
| 00:00 - 00:03 | We are investigating issues with AWS services in the ME-CENTRAL-1 Region. |
| 00:04 - 00:05 | At around 4:30 AM PST |
| 00:05 - 00:07 | One of our Availability Zones |
| 00:08 - 00:12 | mec1-az2 was impacted by "objects" |
| 00:12 - 00:15 | that struck the data center, creating sparks and fire. |
| 00:17 - 00:19 | That's OK.Our multi-AZ Auto Scaling Group |
| 00:19 - 00:21 | would switch to mec1-az3 soon. |
| 00:24 - 00:26 | Dear CTO... |
| 00:27 - 00:28 | mec1-az3... |
| 00:31 - 00:33 | We just lost S3 in mec1-az3 |
| 00:34 - 00:36 | Another "object" hit it. |
| 00:53 - 00:58 | Everyone except CIO, VP of Engineering and SRE lead, please leave the room. |
| 01:13 - 01:15 | Where is the failover? |
| 01:15 - 01:17 | I ordered a multi-region failover! |
| 01:18 - 01:23 | I wanted our backups in a different city! |
| 01:25 - 01:28 | No more free food until this is fixed |
| 01:29 - 01:31 | YOU HAD ONE JOB. |
| 01:31 - 01:34 | The SLA says 99.99% uptime! |
| 01:34 - 01:37 | I don't care that TWO AZs went down |
| 01:37 - 01:40 | We should've been in us-west-2 already! |
| 01:40 - 01:42 | Local data protection laws does not permit us to send data to the USA |
| 01:42 - 01:46 | What happened to our own servers? We used to host them! |
| 01:46 - 01:48 | We disabled those to save costs last quarter. |
| 01:48 - 01:52 | Don't you know there's no cloud |
| 01:53 - 01:54 | Just someone else's computer. |
| 01:56 - 01:57 | I told you in JANUARY. |
| 01:57 - 02:00 | I said "multi-region." I used those EXACT WORDS. |
| 02:00 - 02:03 | I sent a calendar invite called "Multi-Region Strategy" and nobody showed up! |
| 02:04 - 02:08 | We have enterprise client. |
| 02:08 - 02:13 | We have a 99.99% uptime SLA. Do you know what 99.99% means?! |
| 02:14 - 02:16 | It means 52 minutes of downtime PER YEAR |
| 02:17 - 02:21 | You all better convert your laptops to servers and serve traffic via Starlink! |
| 02:27 - 02:29 | You had one architecture decision |
| 02:30 - 02:34 | AWS literally wrote a whitepaper. They made a framework! |
| 02:34 - 02:36 | To handle multi-AZ failure |
| 02:41 - 02:42 | They have a certification exam. |
| 02:43 - 02:47 | It's literally on page one and you all failed the open book test |
| 02:48 - 02:53 | I knew I shouldn't have hired you guys from Microsoft |
| 02:54 - 02:56 | Just "put the application in MORE THAN ONE PLACE" |
| 02:56 - 02:59 | MORE THAN ONE PLACE! |
| 03:00 - 03:02 | and you COULDN'T DO IT! |
| 03:04 - 03:07 | Don't worry, at least we are not using us-east-1 |
| 03:14 - 03:16 | Our status page is down |
| 03:19 - 03:23 | Because our status page is in the same region. |
| 03:25 - 03:26 | It's over. |
| 03:31 - 03:33 | Perhaps we should have used Google Cloud |
| 03:40 - 03:46 | Google Cloud has fewer services than AWS......but you know what? You can't break what doesn't exist. |
| 03:46 - 03:49 | ...we would have been fine on Google Cloud anyway... |
| 03:53 - 03:56 | It's only some javascript |