| 00:00 - 00:03 | We set up Debezium to capture changes from Postgres |
| 00:04 - 00:05 | then configured Kafka Connect |
| 00:05 - 00:07 | with the sink connector |
| 00:08 - 00:12 | We deployed a 5-node Kafka cluster with Schema Registry |
| 00:12 - 00:15 | and wrote 2,000 lines of YAML to wire it all together |
| 00:17 - 00:19 | We're running CDC from |
| 00:19 - 00:21 | 47 Postgres tables into ClickHouse |
| 00:24 - 00:26 | Mein Fuhrer... |
| 00:27 - 00:28 | The pipeline... |
| 00:31 - 00:33 | Someone changed a column type in Postgres |
| 00:34 - 00:36 | The entire CDC pipeline is down. ClickHouse has been out of sync for 3 days |
| 00:53 - 00:58 | If you've never debugged a Debezium connector at 3 AM, leave now |
| 01:13 - 01:15 | What were you thinking? |
| 01:15 - 01:17 | Who runs a CDC pipeline with NO schema evolution strategy?! |
| 01:18 - 01:23 | You think watching one Kafka Summit talk makes you a streaming engineer?! |
| 01:25 - 01:28 | We should have just stayed on Postgres... |
| 01:29 - 01:31 | You HAVE to install every CNCF sandbox project |
| 01:31 - 01:34 | you find on Hacker News! |
| 01:34 - 01:37 | I could have used a managed Snowflake pipeline |
| 01:37 - 01:40 | for a tenth of the headache |
| 01:40 - 01:42 | Even a cronjob with pg_dump scales better than this |
| 01:42 - 01:46 | That's what the Kafka vendor wants you to believe, idiot! |
| 01:46 - 01:48 | Mein Fuhrer, we can add Flink to handle the schema changes |
| 01:48 - 01:52 | ANOTHER JVM service with its own cluster to manage?! |
| 01:53 - 01:54 | What's next, Spark? |
| 01:56 - 01:57 | Maybe Airflow on top? |
| 01:57 - 02:00 | I was told to use 'infrastructure as code' for the connectors |
| 02:00 - 02:03 | But the connector configs are 500 lines of JSON that NO ONE understands |
| 02:04 - 02:08 | We wasted 4 months migrating from Kafka to Redpanda to avoid the JVM |
| 02:08 - 02:13 | And we STILL need Debezium which IS a JVM application! |
| 02:14 - 02:16 | The sink connector crashes every time ClickHouse does a merge |
| 02:17 - 02:21 | And the Schema Registry rejects every other schema change |
| 02:27 - 02:29 | Why is syncing two databases so damn hard? |
| 02:30 - 02:34 | Why can't I just define my schema and have everything wired up automatically? |
| 02:34 - 02:36 | Even our YAML configs have YAML configs |
| 02:41 - 02:42 | I'm done with Debezium and Kafka Connect |
| 02:43 - 02:47 | I will rewrite the entire pipeline myself. In Python. |
| 02:48 - 02:53 | All I wanted was to get data from Postgres into ClickHouse |
| 02:54 - 02:56 | Instead I'm running a JVM zoo with 6 services |
| 02:56 - 02:59 | just to move rows from one database to another |
| 03:00 - 03:02 | I'm putting so much effort into this... |
| 03:04 - 03:07 | Maybe I should just give up and query Postgres directly |
| 03:14 - 03:16 | Who needs analytics anyway |
| 03:19 - 03:23 | I just wanted schema as code. One command. Hot reload. Is that too much to ask? |
| 03:25 - 03:26 | Apparently it is. |
| 03:31 - 03:33 | Six months of my life. Gone. Debugging offset resets. |
| 03:40 - 03:46 | Sir... a junior engineer just Slacked me a GitHub link |
| 03:46 - 03:49 | Something called... MooseStack? |
| 03:53 - 03:56 | ...six months. |
