| 00:00 - 00:03 | We set up Debezium to capture changes from Postgres |
| 00:04 - 00:05 | then configured Kafka Connect |
| 00:05 - 00:07 | with the sink connector |
| 00:08 - 00:12 | We deployed a 5-node Kafka cluster with Schema Registry |
| 00:12 - 00:15 | and wrote 2,000 lines of YAML to wire it all together |
| 00:17 - 00:19 | We're running CDC from |
| 00:19 - 00:21 | 47 Postgres tables into ClickHouse |
| 00:24 - 00:26 | Mein Fuhrer... |
| 00:27 - 00:28 | The pipeline... |
| 00:31 - 00:33 | Someone changed a column type in Postgres |
| 00:34 - 00:36 | The entire CDC pipeline is down. ClickHouse has been out of sync for 3 days |
| 00:53 - 00:58 | If you've never debugged a Debezium connector at 3 AM, leave now |
| 01:13 - 01:15 | What were you thinking? |
| 01:15 - 01:17 | Who runs a CDC pipeline with NO schema evolution strategy?! |
| 01:18 - 01:23 | You think watching one Kafka Summit talk makes you a streaming engineer?! |
| 01:25 - 01:28 | We should have just stayed on Postgres... |
| 01:29 - 01:31 | You HAVE to install every CNCF sandbox project |
| 01:31 - 01:34 | you find on Hacker News! |
| 01:34 - 01:37 | I could have used a managed Snowflake pipeline |
| 01:37 - 01:40 | for a tenth of the headache |
| 01:40 - 01:42 | Even a cronjob with pg_dump scales better than this |
| 01:42 - 01:46 | That's what the Kafka vendor wants you to believe, idiot! |
| 01:46 - 01:48 | Mein Fuhrer, we can add Flink to handle the schema changes |
| 01:48 - 01:52 | ANOTHER JVM service with its own cluster to manage?! |
| 01:53 - 01:54 | What's next, Spark? |
| 01:56 - 01:57 | Maybe Airflow on top? |
| 01:57 - 02:00 | I was told to use 'infrastructure as code' for the connectors |
| 02:00 - 02:03 | But the connector configs are 500 lines of JSON that NO ONE understands |
| 02:04 - 02:08 | We wasted 4 months migrating from Kafka to Redpanda to avoid the JVM |
| 02:08 - 02:13 | And we STILL need Debezium which IS a JVM application! |
| 02:14 - 02:16 | The sink connector crashes every time ClickHouse does a merge |
| 02:17 - 02:21 | And the Schema Registry rejects every other schema change |
| 02:27 - 02:29 | Why is syncing two databases so damn hard? |
| 02:30 - 02:34 | Why is there no sane way to just say 'here's my schema, wire everything up'? |
| 02:34 - 02:36 | Even our YAML configs have YAML configs |
| 02:41 - 02:42 | I'm done with Debezium and Kafka Connect |
| 02:43 - 02:47 | Sir... a junior engineer found something called MooseStack |
| 02:48 - 02:53 | Another framework? I swear to God if it needs a JVM... |
| 02:54 - 02:56 | No JVM. You define your schema in TypeScript or Python |
| 02:56 - 02:59 | Run one command and it sets up ClickHouse, Redpanda, APIs, everything |
| 03:00 - 03:02 | ...one command? |
| 03:04 - 03:07 | moose dev. Hot reload. Schema migrations handled automatically |
| 03:14 - 03:16 | You're telling me this existed the ENTIRE TIME? |
| 03:19 - 03:23 | I spent six months on this pipeline... six months of YAML |
| 03:25 - 03:26 | Six months of Java stack traces at 2 AM |
| 03:31 - 03:33 | Delete the Kafka cluster. Delete all of it. |
| 03:40 - 03:46 | We're rewriting everything in MooseStack. TypeScript. No JVM. Schema as code. |
| 03:46 - 03:49 | Like civilized people. |
| 03:53 - 03:56 | If only someone had told me about this six months ago... |
