| 00:00 - 00:05 | So on Saturday I got a call.. conversions are not coming in, there's a drop in subscribers |
| 00:05 - 00:07 | come quickly, let's find out what's going on. |
| 00:07 - 00:10 | I opened Logz.io.. and guess what |
| 00:11 - 00:16 | There were some 404s in Subscription API |
| 00:17 - 00:21 | Looked a bit carefully... TASK KILLED |
| 00:21 - 00:24 | Tried to log into Nomad |
| 00:29 - 00:31 | there's no Nomad |
| 00:31 - 00:34 | The whole fucking production cluster died |
| 00:38 - 00:40 | No landing pages |
| 00:41 - 00:47 | No products, no subscriptions no cs, absolutely nothing |
| 00:47 - 00:50 | Everything disappeared |
| 00:50 - 00:56 | nothing, absolutely nothing |
| 00:58 - 00:59 | The whole shit went babanas! |
| 00:59 - 01:01 | We got the backups up |
| 01:04 - 01:06 | They failed!! |
| 01:06 - 01:08 | They failed 4 times! |
| 01:13 - 01:14 | All of them failed |
| 01:14 - 01:16 | Then the lead devops had to leave |
| 01:16 - 01:20 | He took his shit and left us to sort out everything |
| 01:21 - 01:27 | but only after he deleted all the logs |
| 01:35 - 01:36 | So.... |
| 01:36 - 01:39 | we have no idea what happened |
| 01:41 - 01:44 | Well, it's okay, we said |
| 01:44 - 01:47 | let's deploy Kubernetes |
| 01:47 - 01:49 | . |
| 01:50 - 01:53 | . |
| 01:54 - 01:56 | . |
| 01:56 - 01:58 | . |
| 01:58 - 02:02 | . |
| 02:05 - 02:07 | . |
| 02:10 - 02:11 | . |
| 02:11 - 02:13 | . |
| 02:15 - 02:17 | . |
| 02:17 - 02:20 | . |
| 02:21 - 02:28 | . |
| 02:29 - 02:30 | . |
| 02:30 - 02:32 | . |
| 02:36 - 02:39 | . |
| 02:40 - 02:42 | . |
| 02:50 - 02:55 | . |
| 02:56 - 02:58 | . |
| 02:58 - 03:04 | . |
| 03:04 - 03:09 | . |
| 03:09 - 03:11 | . |
| 03:16 - 03:20 | . |
| 03:23 - 03:26 | . |
| 03:31 - 03:36 | . |
