<!-- nornir:class: design -->
# Fast-commit populate — the commit-side levers — design
> **Class:** `design` · **Function:** `populate` · **Uses:** `write`, `commit`,
> `catalog`, `atomic`, `builder`. The one shape that runs. This is the umbrella
> over the three commit-side populate levers and the ergonomic surface that makes
> the fast path the *default*; the per-mechanism detail lives in
> [`ingest-multi-design.md`](ingest-multi-design.md) (cross-table single commit)
> and [`committer-overlap-design.md`](committer-overlap-design.md) (the
> un-serialized committer).
## f0 — the encode is done; the commit is the frontier
skade's bulk encode is already memory-bandwidth-bound and all-core-saturated
(`write` ≈ 35 M rows/s, 12/12 cores — the core-saturation law is satisfied). The
warehouse-populate analysis (`workspace_nordisk/bencher_agents_problems.md`)
confirmed the remaining populate cost is **not** in the encode but **downstream
of it**: the snapshot-commit fsync and the serial committer that issues it. So
the levers here never touch the encode — they cut the number of fsyncs, the cost
of each fsync, and the time the committer spends blocking on them:
| # | Lever | Cuts | Entry point |
|---|-------|------|-------------|
| 1 | **Cross-table single commit** | N table commits → **1** redb txn (**1 fsync**) | `Warehouse::ingest_multi` / `skade::ingest_multi` |
| 1b | **Bulk-import durability** | the per-commit **fsync itself** (`Eventual`/`None`) | `Warehouse::open_with_durability` |
| 2 | **Overlapped committer, recommended default** | the **serial** commit-fsync tail (hidden behind write I/O) | `Table::populate` |
Each is additive: every per-table path (`append`, `ingest_parallel`,
`ingest_pipelined`) is unchanged and stays available. The L2 contract holds — the
embedded API and its ~35 M rows/s encode do not regress; these are new lanes.
## f1 — lever #1: one commit for many tables (`Warehouse::ingest_multi`)
Every `fast_append` is one Iceberg snapshot = one redb write txn = **one fsync**.
A populate that writes several tables as one logical unit (nornir's symbol scan
writes ~4 tables/scan) therefore pays **one fsync per table**. `ingest_multi`
all-core-encodes and writes every table's data files exactly as `ingest_parallel`
would (byte-identical files), then **stages** each table's fast append
(`Transaction::stage_fast_append` — manifests written, catalog pointer untouched)
and flips all of them in **one** all-or-nothing redb transaction via
`RedbCatalog::atomic_release_raw`. N tables → **one fsync**; each table still
gains exactly one new snapshot; a mid-flush failure commits nothing.
`Warehouse::ingest_multi(Vec<(Table, groups)>, &WriteProps)` is the ergonomic
form — it supplies the warehouse's own catalog, so callers pass their `Table`
handles directly instead of threading `&dyn Catalog` + `IceTable` through the
free `skade::ingest_multi`. Full mechanism + failure contract:
[`ingest-multi-design.md`](ingest-multi-design.md).
## f2 — lever #1b: shed the fsync for bulk import (`open_with_durability`)
redb's write durability is a per-database setting (`WriteDurability::{Immediate,
Eventual, None}` in `skade_katalog::builder`). `Immediate` (the default, and what
`Warehouse::open` uses) fsyncs on every commit — correct for steady-state
serving. A **bulk import** is a different regime: the whole warehouse is
rebuildable from source, so the fsync is pure tax. `Warehouse::open_with_durability`
surfaces the knob at the warehouse level:
* `Eventual` — persists shortly after `commit` returns; a hard crash may lose the
last few commits but the DB stays internally consistent.
* `None` — no per-commit fsync; the fastest, durability best-effort.
This composes with lever #1: `ingest_multi` cuts the **count** of commits,
durability cuts the **cost** of each. Caveat (redb): pages are only reclaimed at
higher durability, so a long-lived process writing *exclusively* at `None` grows
the catalog file — use `None` for a bounded import, then reopen at `Immediate`
for durable serving. `swap_table` overrides durability to `Immediate` regardless,
so a compaction swap is never at risk.
## f3 — lever #2: the overlapped committer is the recommended default (`Table::populate`)
`ingest_parallel` writes every file with a **single serial writer**;
`ingest_pipelined` overlaps the FileIO writes (the async `gatling::io::run_ordered`
engine, ≤ `channel_depth` in flight, re-sequenced to submission order) but keeps a
**serial commit tail** (write-all → commit-all). `ingest_pipelined_overlap`
un-serializes that tail: a concurrent committer consumes the ordered write stream
so the `fast_append` fsyncs of early file-groups overlap the writes of the later
ones (mechanism: [`committer-overlap-design.md`](committer-overlap-design.md)).
`Table::populate(groups, files_per_commit)` **is** the overlapped path driven at
a machine-tuned `default_channel_depth()` (2× cores, clamped `[8, 64]` — a wider
default than the old hand-picked `8`; the bounded channel + submission-order
reorder buffer keep memory in check regardless). It is the entry point new
callers should reach for — both the I/O round-trips and the commit-fsync tail are
hidden behind the encode/write stream, and the result is byte/row-identical with
the same snapshot grouping as the lower-level paths. `ingest_parallel` (strict
serial writer) and `ingest_pipelined` (serial committer) remain for callers that
need a specific topology or the proven serial-commit contract.
For a **multi-table** populate, `populate` (single-table) composes with lever #1:
encode+write each table, then one `Warehouse::ingest_multi` flush.
## f4 — observability + tests
The durable data-commit (fsync) count is a process-global counter,
`skade_katalog::redb_commit_count`, bumped once per redb pointer-advance txn
(`note_data_commit`). The tests read a delta on it to **prove** the collapse
rather than infer it:
* `ingest_multi.rs` — an `ingest_multi` of N tables advances the counter by **1**
vs **N** for N separate `ingest_parallel` calls; all-or-nothing; byte-identical
(RED-when-broken: regress to a per-table loop and the delta jumps to N).
* `populate.rs::warehouse_ingest_multi_wrapper_is_one_fsync` — the same **1-vs-N**
contrast through the `Warehouse::ingest_multi` ergonomic wrapper.
* `populate.rs::populate_default_matches_parallel` — `Table::populate` is
byte/row-identical to `ingest_parallel` (recommended default changes scheduling,
never data) and `default_channel_depth()` is clamped to `[8, 64]`.
* `populate.rs::open_with_durability_bulk_import_roundtrips` — a populate under
`None`/`Eventual` lands every row with the correct snapshot count (the
fsync-shedding knob never loses or corrupts data).
Because `redb_commit_count` is process-global within a test binary, every
committing test in `populate.rs` holds one `COMMIT_COUNT_GATE` so a measured
delta is never polluted by a parallel test's commits.
## f5 — benches
Host-stamped, plain-number arms in `bench/examples/nornir-bench.rs`
(authoritative numbers are the Loki bencher's):
* `skade.ingest_multi` — per-table (N fsyncs) vs cross-table (1 fsync) on a
symbol-scan-shaped flush; reports the measured `multi_fsyncs`/`separate_fsyncs`
and the fsync reduction.
* `skade.committer_overlap` — serial vs overlapped committer, commit-bound shape.
* `skade.ingest_strategies` — `ingest_parallel` vs `ingest_pipelined`.
* `skade.bulk_import_durability` — a many-commit populate at `Immediate` vs `None`;
same data + snapshots, only the per-commit fsync tax differs.