<!-- nornir:class: idiot -->
# CDC for beginners — "what changed?", explained slowly
## What problem is this solving
You have a table. Yesterday it had a million rows. Today it has a million and
five, one row got edited, and two got deleted. You want a screen that shows those
eight events — not the million rows.
That list of events is a **change feed**, and producing it is called **CDC**
(change data capture).
## The naive way, and why it is bad
Read the whole table today, read the whole table yesterday, subtract. That works
and it is terrible: you pay for a million rows to learn about eight, and you
cannot tell an *edit* from a delete-plus-insert.
Table formats like **Apache Iceberg** keep every commit as a numbered
**snapshot**, so you can ask "what happened between snapshot 41 and snapshot 42"
and get back only the changes. That is what knut does.
## Three ways to ask, and knut picks one
Annoyingly, there are three different mechanisms depending on what is on the
other end:
1. **Spark 4.2's own CDC.** New in Spark 4.2 (July 2026). You write SQL:
```sql
SELECT * FROM my_table CHANGES FROM VERSION 41 TO VERSION 42
```
Spark hands back the changed rows plus three extra columns saying what kind of
change each was.
2. **Iceberg's CDC.** Older, and what Spark 4.1 and earlier have. You call an
Iceberg stored procedure that builds a temporary view of the changes, then
`SELECT` from that view.
3. **No Spark at all.** knut can read Iceberg's files directly with the `skade`
library — pure Rust, no Java, no cluster. It works out the changes from
Iceberg's own bookkeeping files.
You do **not** choose. knut asks the server which it is, tries the fast path, and
uses whatever actually works.
## The plot twist
Spark 4.2's native CDC is real and it works — but only for tables whose
*connector* has been taught to speak it. **Iceberg has not been taught yet.**
Iceberg does not even publish a Spark 4.2 build at the time of writing.
So if you point knut at a Spark 4.2 server holding Iceberg tables, it will:
1. see "4.2 — good, native CDC exists";
2. actually try it (a zero-row test query);
3. get an error back;
4. **say so, loudly**, and use Iceberg's procedure instead.
That last step is the important one. A tool that quietly used the fallback and
said nothing would leave you believing you were on a code path you were not on.
knut writes `fell_back: true` plus the engine's own error message into its log
**and** into the state file it keeps on disk.
## What comes out
One row per change event, with five columns:
| column | what it holds |
|---|---|
| `op` | what happened: `insert`, `delete`, `update_before`, `update_after`, `update` |
| `before` | what the row looked like **before** — empty if it is a brand-new row |
| `after` | what it looks like **after** — empty if the row was deleted |
| `snapshot` | which commit this change belongs to |
| `ts` | when that commit happened |
So a delete has a `before` and no `after`. An insert has an `after` and no
`before`. An edit has both — that is your diff, ready to paint red/green.
They arrive **in batches, as they are produced**, so a change pane can start
showing rows immediately instead of freezing until the whole scan is done.
## Why "don't read columns you don't need" matters here
If your screen only shows a person's `id` and `name`, there is no reason to read
their address, their eight payload blobs, and everything else. Telling knut which
columns you need is called a **projection**, and telling it which rows you care
about (`country = 'SE'`) is a **predicate**.
knut already did this for whole-table reads. It did **not** do it for change
feeds — every CDC read pulled every column, always. That is now fixed, and
`knut.cdc_pushdown_read` measures the difference.
One caveat the honest version has to include: on the no-Spark path, knut trims
the columns *after* reading the file, not before. So it saves memory and
downstream work, not disk reads. Making it skip the disk reads too is a change
inside `skade`, not inside knut.
## The one rule that must never break
**A change feed must never silently lose a delete.**
Everything else is a performance question. If a delete goes missing, your graph
keeps a node that no longer exists and nobody finds out. So:
* an unrecognised change-type is an **error**, not a skipped row;
* a `null` change-type is an **error**;
* when a filter cannot judge a row (because a deleted row's columns are blank —
which is normal), knut **keeps** it rather than guessing;
* there is a test that inserts, edits and deletes rows in a real Iceberg table
and fails if the delete does not come out the other end.