<!-- nornir:class: design -->
# Table compaction — rebuild-and-swap — design
> **Class:** `design` · **Function:** `compaction` · **Uses:** `commit`, `write`,
> `catalog`. Only the shape that runs; the observed pathology it fixes (unbounded
> snapshot/metadata growth under continuous `fast_append` ingest → OOM) is the
> motivation, not an alternative — there is no second design here.
## f0 — why a rebuild, and the shape it takes
iceberg-rust 0.9.1's only writer is `fast_append`: each commit adds one snapshot
and one manifest and **carries the whole manifest list forward**. A long-lived,
continuously-appended table therefore grows its snapshot log, manifest list, and
tiny-file count without bound — thousands of snapshots, GBs of metadata, and
eventually process OOM (observed on the garmr dogfood warehouse: 13 GB / 5869
snapshot files, RSS peak 7.2 GB).
Compaction fixes this by **rebuilding** the table rather than editing its history:
stream the current rows into a fresh table written as a handful of large zstd
files in **exactly one** commit, then **atomically swap** that table into the live
name. The compacted table has one snapshot regardless of size, so the growth
arithmetic converges — no separate `expire_snapshots` pass is needed. Peak memory
is one Parquet row group; a large table is never materialised whole.
## f1 — `Warehouse::compact_table` (the orchestration)
`warehouse.rs` · `compact_table(name, prune) -> CompactReport`
1. Load the live table; reject partitioned tables (a stream can span many
partition keys, and skade appends are single-partition).
2. Create a uniquely-named scratch table `<ns>.<table>__c<nanos>` with the same
Arrow schema.
3. `rebuild_into`: scan the old snapshot to Arrow and feed the stream to
`append_stream_props` (§f2) with ZSTD + a 128Ki row-group target → one commit,
one snapshot on the scratch table. An optional `prune` closure transforms each
batch en route (e.g. drop rows already sealed to cold storage — the one safe
moment to shrink, since the writer is single and the rewrite is total).
4. On any failure in step 3: drop the scratch table and `remove_dir_all` its
partial data dir, then return the error — nothing leaks to the next restart and
the live table is untouched.
5. `catalog.swap_table` (§f3) atomically repoints `name` at the scratch metadata.
6. Return `CompactReport { snapshots_before, rows, rows_pruned, old_data_dir }`.
**Caller obligations after return** (documented on the fn): reload any held
`Table` handle for `name` (the old handle points at retired metadata), and delete
`CompactReport::old_data_dir` only after a grace window longer than the longest
in-flight query (readers resolved just before the swap keep streaming the old
files).
**Concurrency contract:** no concurrent *writer* to `name` (garmr calls this on
its single append actor); a racing append copies the entry snapshot and would be
silently lost. Concurrent *readers* are fine — the swap is gap-free.
## f2 — `append_stream_props` (the one-commit stream writer)
`write.rs` · `append_stream_props(catalog, table, batches, map, props) -> (IceTable, rows_in, rows_out)`
Rolls the incoming batch stream into large Parquet files via a
`RollingFileWriterBuilder` + `DataFileWriterBuilder`, applies the optional `map`
per batch, recasts to the table's Arrow target, and commits **all** the rolled
data files in a **single** `fast_append` transaction. Unlike per-group `append`
(one snapshot + manifest each), the result gains exactly one snapshot. Empty
input (or a `map` that drops everything) commits nothing and leaves the table at
its current state. Peak memory is one row group. Unpartitioned tables only.
## f3 — `RedbCatalog::swap_table` (the atomic pointer swap)
`catalog.rs` · `swap_table(live, temp, new_snapshot) -> Option<String>` — one redb
write transaction, `Durability::Immediate`:
1. Read `temp`'s metadata_location (error if absent); read `live`'s prior location.
2. Remove `temp`'s pointer; insert `temp`'s location under `live` — **no instant
where `live` resolves to nothing** (a two-step rename would let a crash leave
`live` absent, which `table_or_create` would silently re-create as an empty
table → data loss).
3. Prune the commit log for **both** keys over the full `[id 0 .. id -1]` suffix
range (retired entries point at metadata about to be GC'd, and an unpruned log
— plus the Ragnar index rebuilt from it — is the same slow-OOM class compaction
exists to fix), then re-seed `new_snapshot` under the live key so log and
pointer stay consistent.
4. `commit()` (one fsync), then L1 pointer-mirror write-through under the lock.
Pointer-only, like `rename_table`: no data files are moved or deleted — the caller
owns physical reclaim (`CompactReport::old_data_dir`).
## f4 — the reader race (datafusion providers)
A table listed a moment ago can be removed by a concurrent swap (the scratch
table) or a drop before a session loads it. Both eager providers — the `sql`
module's per-session registration (`warehouse.rs`) and the vendored
`IcebergSchemaProvider` (`vendor/iceberg-datafusion-arrow58/src/schema.rs`) — now
**skip** a `TableNotFound` on load instead of failing the whole session: an
unrelated `SELECT … FROM events` must not 404 on a transiently-absent scratch
table.
## f5 — bounded handle cache
`table_cache.rs` · `DEFAULT_TABLE_HANDLE_CACHE_CAPACITY` dropped `16_384 → 512`.
Each cached handle pins its parsed `TableMetadata` (O(snapshots)) plus a
scan-populated `ObjectCache`, and under continuous ingest every commit inserts a
new metadata_location — so a generous cap steadily pins gigabytes (observed on
garmr's 8 GB SOC host). 512 ≈ tens of MB worst case while still covering every hot
table.
## Observability
`compact_preserves_content_and_collapses_snapshots` (RED-when-broken,
`skade/tests/compaction.rs`): 20 tiny appends → 20 snapshots; after
`compact_table` the reloaded table's snapshot log is **exactly 1**, the many tiny
files are coalesced, and the row content is **byte-identical** (id → score-bits +
name multiset) before vs after — and again after a warehouse reopen (durable,
gap-free). `compact_with_prune_drops_selected_rows_only` asserts the `prune` hook
drops exactly its selected rows and keeps the rest byte-identical, with
`CompactReport.rows` / `rows_pruned` accounting for the split. Reverting the swap
turns both red.