<!-- nornir:class: reasoning -->
# CDC — the research trail and the roads not taken
Sister to `cdc-design.md`, which names only the chosen design. This file holds
the sources the design rests on and the alternatives that were rejected.
## Sources (checked 2026-07-22, not recalled)
**Spark 4.2 native CDC**
| claim | source |
|---|---|
| CDC is in 4.2.0, as a `CHANGES` clause + DataFrame/PySpark/**Spark Connect** APIs, batch and streaming, plus Auto CDC in Declarative Pipelines | Spark 4.2.0 release notes, `spark.apache.org/releases/spark-release-4-2-0.html` (SPARK-55668, SPARK-56249) |
| Umbrella issue, **Resolved**, fix-version **4.2.0**, 12 subtasks; example `SELECT * FROM my_table CHANGES FROM VERSION 10 TO VERSION 20`; changelog schema requires `_change_type`, `_commit_version`, `_commit_timestamp` | ASF JIRA **SPARK-55668** |
| The DSv2 connector API + analyzer resolution + the SQL clause | ASF JIRA **SPARK-55948** |
| DataFrame API **and Spark Connect** support | ASF JIRA **SPARK-55949** |
| PySpark `changes()` | ASF JIRA **SPARK-55950** |
| Exact grammar (`changesClause`, `streamChangesClause`, `relationPrimary` → `#changelogTableName`) | `apache/spark` tag **`v4.2.0`**, `sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4` lines 928–939, 1119 |
| The connector interface, `@since 4.2.0`; `CHANGE_TYPE_INSERT = "insert"`, `..DELETE = "delete"`, `..UPDATE_PREIMAGE = "update_preimage"`, `..UPDATE_POSTIMAGE = "update_postimage"`; `_commit_version` is Long **or** String; `_commit_timestamp` TIMESTAMP; the pushdown contract | `apache/spark` `v4.2.0`, `sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Changelog.java` |
| Engine-side post-processing (carry-over removal, update detection, netChanges), batch and streaming | `apache/spark` `v4.2.0`, `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala` |
**Where the docs are wrong-by-omission:** the 4.2.0 SQL reference does *not*
index a CHANGES page. `docs/latest/sql-ref-syntax.html`'s Data Retrieval list has
no CDC entry and `docs/latest/sql-ref-syntax-qry-select-changes.html` returns 404.
Anyone checking the docs site alone would conclude the feature does not exist. It
does; the page is missing. This is why the grammar was read from the source tag.
**Iceberg's side**
| claim | source |
|---|---|
| `create_changelog_view` produces `_change_type` (INSERT/DELETE/UPDATE_BEFORE/UPDATE_AFTER), `_change_ordinal`, `_commit_snapshot_id`; supports `net_changes` and `identifier_columns` | Iceberg Spark Procedures docs, `iceberg.apache.org/docs/latest/spark-procedures/` |
| Latest release is **1.11.0** | `iceberg.apache.org/releases/` |
| **No `iceberg-spark-runtime-4.2_2.13` exists.** The published set at 1.11.0 is `{3.4,3.5,4.0,4.1}_2.13` (+ older 2.12) | Maven Central search API, `g:org.apache.iceberg AND a:iceberg-spark-runtime-4.2_2.13` → `numFound: 0`; `a:iceberg-spark-runtime*` → the list above |
| `apache/iceberg` `main` has `spark/v3.5`, `spark/v4.0`, `spark/v4.1` and no `v4.2` | GitHub contents API on `apache/iceberg:spark` |
| **#14984 "Spark: Add support for 4.2.0" is OPEN**; **#16751 "Spark: Add Spark 4.2 module (copy of Spark 4.1)" was closed unmerged** on 2026-06-13 | GitHub issues/PR API |
| **No Iceberg source implements `org.apache.spark.sql.connector.catalog.Changelog`** | GitHub code search, `org.apache.spark.sql.connector.catalog.Changelog repo:apache/iceberg` → `total_count: 0` |
knut's own tree had already recorded the runtime half of this:
`containers/spark-iceberg/Containerfile` ("no 4.2 runtime yet") and
`crates/knut-pipelines/src/server.rs` (`DEFAULT_IMAGE_41`, with 4.2.0 GA dated
2026-07-14 as a named alternate). The *connector* half — that Iceberg implements
no `Changelog` at all, so the clause would not resolve even with a runtime — is
new here, and it is the fact that forced detection to be a **probe** rather than
a version comparison.
## Alternatives rejected
**Choose the backend from config.** A `cdc_backend: spark_native | iceberg` key in
the mapping YAML. Rejected: it makes the operator responsible for a fact only the
server knows, and it fails in the worst way — a mapping that says `spark_native`
against a 4.1 server errors every round, and one that says `iceberg` against a
future CDC-capable connector silently keeps the slow path forever. Detection is
free (one version request + one zero-row probe, cached per table).
**Choose from the version string alone.** `>= 4.2 → native`. Rejected by the
Iceberg finding above: the engine having CDC and *this table* having CDC are
different questions, and today the answers differ. A version-only check would
have shipped a path that errors on every real table.
**Trust the probe's absence of an error as "it works".** The probe is
`… CHANGES FROM VERSION 0 LIMIT 0`. It proves the clause *resolves* — the
analyzer found a `Changelog` on the connector — which is exactly the question.
It does not prove the data is correct; the round-trip test does that.
**One `_change_type` vocabulary, imposed.** Normalise everything to Iceberg's
UPPER-CASE spelling at the source and delete the Spark one. Rejected: the wide
batches flow to the listener untouched (that is what makes the pushdown a pure
narrowing), so the spellings genuinely coexist in the pipeline. Making
`ChangeOp::parse` accept both, and canonicalising only at the *renderable* edge,
keeps one vocabulary without rewriting anyone's data.
**A `Vec<ChangeRow>` struct for the UI.** Rejected: the whole pipeline is Arrow,
FalkorDB's loader takes `RecordBatch`, and a row-struct would force a transpose
per batch and lose zero-copy. The `Struct` columns give the UI the same
per-row ergonomics with none of that.
**`futures::Stream` for the feed.** Rejected: `futures` is an optional dep
(feature `flight`) and the core is deliberately dependency-light. `tokio` is
already a hard dep, so `tokio::sync::mpsc` gives bounded, backpressuring delivery
for free. If a `Stream` impl is ever wanted it is a wrapper, not a rewrite.
**A second incremental reader in knut.** Rejected outright (L5): skade owns
`read_delta` / `read_changelog` / `project_columns` and knut consumes them.
Where skade's surface is missing something (a projection on the delta scan), that
is logged as skade-side work rather than worked around with a private copy.
**A second row-filter implementation for CDC.** Nearly shipped, then rejected.
`89d7e31` had just landed `mapping::apply_row_filter`, and the CDC path needed one
behavioural difference: a row whose filter column is NULL must be **kept** (that
is a widened `DELETE`), whereas the scan-planner semantics every other mode must
match is **drop**. Twinning the kernel lowering to get that would have been two
predicates that could drift. Instead `FilterNulls { Drop, Keep }` was added to the
one implementation. Same code, one explicit policy argument, and the difference is
now a named concept instead of an accident.
## The bug this work uncovered
`mapping::apply_row_filter` ends in `arrow::compute::filter_record_batch`, which
treats a **NULL mask entry as false** — the row is dropped. Since a CDC `DELETE`
row carries nulls in every non-identity column, a mapping with any `filter:` in
`mode: cdc` **silently dropped every delete**. It was live on `main` and no test
covered it. See `cdc-design.md` §f4 and the RED-when-broken test
`cdc::post_filter_keeps_unjudgeable_rows_where_the_scan_planner_would_drop_them`.
That is the concrete argument for why "never lose a delete" is the invariant the
CDC module is organised around, rather than a slogan: the failure is one default
argument away, it produces no error, and the graph just quietly keeps a node that
no longer exists.