<!-- nornir:class: reasoning -->
# skade incremental read (`read_delta`) — the snapshot diff
> `Table::read_delta(from, to)` reads **only the rows appended between two
> snapshots** — not the whole table. `skade/src/read.rs`.
> *Diagram: lineage walk → per-snapshot op guard → manifest-list classify
> (Data→inserts / EqualityDeletes→identity-deletes / PositionDeletes→full-reload)
> → `DeltaPlan` → consumers polling `commit_seq()` then `read_delta(cursor, latest)`.*
## Why it exists
iceberg-rust 0.9.1 (what skade pins) has **no incremental / changelog scan**.
`TableScanBuilder` only offers `snapshot_id(id)` (time-travel to one snapshot) —
there is no `from_snapshot_id`/`to_snapshot_id` and no changelog procedure (that
is Spark-only). So an Iceberg consumer that wants a *streaming* delta (read each
new snapshot's new rows, not re-scan everything) has nothing to call. `read_delta`
fills that gap, built from the manifest layer, pure Rust.
The first consumer is **knut-bifrost's** Iceberg → FalkorDB streaming ingest:
poll the table, `read_delta(cursor, latest)`, inject only the new rows into the
graph, advance the cursor. The second is **modgunn (Móðguðr)**, whose Engine B
world-delta watcher polls `commit_seq()` then `read_delta(cursor, latest)` over
the shared warehouse to track every change to the dependency world — modgunn is
a leaf consumer of skade (never of nornir), so this is its only change-sensing
path. (`modgunn/.nornir/modgunn-master-plan.md` §1 CDC seam.)
## How it works (manifest diff)
1. **Lineage walk** — from `to`, follow `parent_snapshot_id()` back to `from`
(exclusive), collecting the new snapshot ids oldest-first.
2. **Operation guard** — if any new snapshot's `summary().operation` is not
`Append` (overwrite/delete/replace), set `needs_full_reload` and return no
batches; an append-only file delta would leave stale rows behind, so the
caller re-reads `to` in full.
3. **Operation set** — `Append` (inserts) **and** `Delete` (identity/equality
deletes) are both expressible as an additive delta. `Overwrite`/`Replace`
rewrite rows in place — those set `needs_full_reload`.
4. **Manifest-list scan** — load `to`'s manifest list. Each `ManifestFile` has
`added_snapshot_id` + `content`. For manifests added by a snapshot in the new
set, parse the avro and classify each `Added` entry by its **data-file
content type**:
- `Data` (in a `Data` manifest) → `added_files` (rows to insert).
- `EqualityDeletes` → `delete_files` (one `EqualityDeleteFile { path,
equality_ids }`; the identity columns that define the deleted rows).
- `PositionDeletes` → `needs_full_reload` (delete-by-offset can't be mapped
to a graph identity, so re-read).
5. **Read** — `added_files` → insert `RecordBatch`es. `delete_files` are read on
demand by `read_equality_deletes(table, plan)` → `(equality_ids, batches)`
per file (the deleted-row identities). Both through the table's `FileIO`.
Returns `(Vec<RecordBatch>, DeltaPlan)`; the `DeltaPlan { snapshots,
added_files, delete_files, needs_full_reload }` lets a caller see/assert what
the delta resolved to — inserts **and** deletes.
## CDC deletes — both sides, pure Rust
iceberg-rust 0.9.1's `Transaction` only implements `fast_append`, which rejects
non-`Data` content (`validate_added_data_files`), so it cannot write delete
files. skade now writes them at the metadata layer instead:
`Table::delete_equality(keys, equality_columns)` (`skade/src/delete.rs`) writes
an equality-delete parquet (the identity columns of the removed rows) + a v2
`Deletes` manifest + a manifest list that carries the existing manifests
forward, then commits an `AddSnapshot` (`operation = delete`) +
`SetSnapshotRef` through `RedbCatalog::commit_table`. The result is a real
Iceberg equality-delete snapshot, read back by `read_delta` as a
`delete_file` — a complete CDC delete with **no Spark**.
## Limitations (honest)
- **Identity (equality) deletes, not position deletes.** `delete_equality`
writes equality-delete files (delete by business-key value), which a graph
upsert keyed on that id can resolve. *Position* deletes (delete by file+row
offset) and `Overwrite`/`Replace` set `needs_full_reload`.
## Tests
`skade/tests/delta.rs` (no container, real local warehouse):
- `read_delta_reads_only_appended_rows` — A(3)+B(2): the A→B delta reads only
`{4,5}`, one file, cursor=B; a caught-up cursor reads nothing.
- `read_delta_spans_multiple_snapshots` — A→B→C: the A→C delta reads `{3,4,5,6}`
(both new snapshots), two files, oldest-first lineage.
- `read_delta_surfaces_equality_deletes` — A(3) then a real equality-delete of
id=2: the A→B delta has **no inserts**, **one `delete_file`**, no full reload;
`read_equality_deletes` reads back the deleted identity `{2}`. The delete is
*produced* by `Table::delete_equality` — the inject side is pure Rust too.
```bash
cargo test --test delta # in skade/skade
```
## SnapshotUtil parity in the vendored fork (`vendor/iceberg-arrow58`)
`read_delta` above resolves its snapshot lineage by **delegating** to the
vendored `iceberg-arrow58` fork's `TableMetadata::snapshots_added_between`
(swarm #38, 2026-07-29) — it no longer reimplements the ancestor/parent-chain
walk inline. The fork carries the **general** snapshot-range resolution as
first-class `TableMetadata` methods — the metadata-layer primitives Java's
`SnapshotUtil` exposes and that stock iceberg-rust does not ship in a released
form:
| method | Java parity | what it returns |
|---|---|---|
| `snapshot_ancestor_ids(id)` | `SnapshotUtil.ancestorIds` | ancestor ids **newest→oldest**, inclusive of `id`; cycle-guarded; unknown id → empty |
| `snapshots_added_between(from, to)` | `SnapshotUtil.ancestorsBetween` | the `(from, to]` ids **oldest→newest** (application order); **errors** if `from` is not on `to`'s lineage |
| `is_append_only_between(from, to)` | `BaseIncrementalAppendScan` precondition | `true` iff every snapshot in `(from, to]` is `Operation::Append` |
Tested in `vendor/iceberg-arrow58/src/spec/table_metadata.rs`
(`snapshot_range_resolves_append_lineage`, `overwrite_in_range_is_not_append_only`)
— an append lineage resolves, an `overwrite` spliced into the range flips the
append-only check to `false`, and both guards (unknown `to`, a `from` off the
lineage) are asserted.
**Wiring (swarm #38 — dedup collapsed).** `read_delta` now calls
`metadata.snapshots_added_between(from, to)` for the `(from, to]` chain instead
of walking `parent_snapshot_id` itself; this gives the primitive its first
production caller and removes the duplicate lineage walk (one codepath).
`read_delta`'s `needs_full_reload` gate stays a **local explicit loop** over the
resolved chain — it is Delete-tolerant (`Append | Delete` are both expressible
as an additive/CDC delta; only `Overwrite`/`Replace` force a full reload) and is
therefore deliberately **wider** than the fork's `is_append_only_between`
(Append-ONLY). Wiring that stricter primitive into the gate would wrongly
full-reload a pure-`Delete` window and shift observable behaviour for the
knut-bifrost + modgunn CDC consumers, so it is intentionally *not* used there.
The public signature of `read_delta`, the `DeltaPlan` shape, and `read_changelog`
are unchanged; the delta suite proves the observable result is identical
(`read_delta_delegates_to_snapshots_added_between` asserts
`plan.snapshots == snapshots_added_between(...)` across an append range and an
overwrite range).
**Behaviour-identity note / two edge-case refinements.** For every input a real
CDC consumer passes (`from` = a prior snapshot that IS an ancestor of `to` on
the branch, no expired ancestor mid-window) the delegation is byte-identical to
the old inline walk. It refines exactly two unreachable-in-practice edges: a
`from` that is **not** an ancestor of `to` now *errors* (the old walk silently
returned the whole lineage — the #2337 footgun, latent re-stream-everything bug),
and a **dangling** (expired) parent now truncates the walk rather than erroring.
Both match the fork's Java-parity semantics and are not exercised by any consumer
or test. Flagged for Rickard as a (strictly-more-correct) design decision, not a
forced behaviour change.
**Consumer verification (swarm #38).** modgunn builds green against the changed
skade (native `path = "../skade/skade"` dep) and its delta suite passes
(`cdc_delta_scans_only_newly_added_components` et al. drive `table_delta` →
`read_delta`). knut-bifrost consumes the **published** skade by deliberate policy
(the removed `[patch.crates-io] skade` note in `knut/Cargo.toml`); a local
path-patch to verify against the changed skade is blocked by an unrelated
dual-`iceberg-arrow58` version collision (knut pins published `0.9.8`, the local
fork vendors `0.9.1`) — not a semantic shift. Since `read_delta`'s public
API/behaviour is unchanged, knut-bifrost is unaffected.
### Why the fork carries this — the upstream iceberg-rust assessment (2026-07-29)
Checked apache/iceberg-rust before implementing (the "check upstream first" rule):
- **#1469 "support incremental scan between 2 snapshots"** and **#1470** (its PR):
both **closed as not-planned / stale**. The original request that named
`SnapshotUtil`, `IncrementalScan`, `BaseIncrementalAppendScan` — abandoned.
- **#2241 "Add helpers to scan snapshot ancestors"** — **closed, done via #2285**:
upstream `main` did land ancestor helpers. But they are **not in the
`0.9.1`/arrow-58 line the fork pins**, so the fork provides its own (the arrow-58
pin is load-bearing for RecordBatch interop across the skade stack, §Cargo).
- **#2337 "feat: Incremental append scan"** — **OPEN, unmerged**, awaiting review.
Its own description documents a **footgun**: if `from_snapshot` is not on the
lineage it *silently* returns all ancestors of `to`. The fork's
`snapshots_added_between` **errors** in exactly that case — a deliberate
divergence, not an oversight.
- **#2152 "incremental reads between snapshot-ids"** and **#1636 "(incremental)
changelog scan for CDC"** — both **open**, unimplemented upstream.
**Fork plan / what's genuinely missing (reported, not yet in the fork):**
1. **File-planning `IncrementalAppendScan`** — turning the resolved id range into
a planned set of `added` data files (the scan layer above these metadata
primitives). skade already has an equivalent at its own layer in
`read_delta`'s manifest walk; upstreaming/adopting #2337's scan once it merges
is the consolidation path.
2. **Changelog / net-changes scan (#1636)** — CDC over the changelog view
(INSERT/UPDATE_BEFORE/UPDATE_AFTER/DELETE). skade covers the append+equality-
delete slice today via `read_delta` + `delete_equality`; the full changelog
procedure is Spark-only upstream and remains a frontier for the fork.
The must-have for the Iceberg↔Spark streaming gap's **native incremental path**
— resolve a correct, lineage-guarded snapshot range and know whether it is
append-only — is the piece implemented here.