<!-- nornir:class: design -->
# Parallel single-snapshot append — `append_parallel` — design
> **Class:** `design` · **Function:** `append_parallel` / `append_parallel_with`
> / `append_parallel_props` (`Table::append_parallel`) · **Uses:** `write.rs`,
> `znippy-zoomies/gatling_forkjoin`, `parquet`. The write-path complement to the
> footer-stat fold (`write-stat-parallel-design.md`): that one took the serial
> *footer parse* off the write; this one takes the serial *encode* off a lone
> append.
## f0 — the hot cost: a lone append encodes on one core
`append` (`skade::append`, `Table::append`) encodes the **whole** write into ONE
Parquet file on ONE core — a serial zstd/encode that leaves the other ~11 cores
idle. By the core-saturation law that idle is a bug: a single large append (the
exact shape nornir's Mimir populate hits — one big zstd-compressed batch) strands
eleven cores behind one encoder. `ingest_parallel` already fans the encode across
cores, but it requires the caller to pre-split the rows into per-file groups; a
caller with one big batch had no all-core single-commit path.
## f1 — the fix: split row-aligned, encode all-core, commit one snapshot
`write.rs` · `append_parallel_props(catalog, table, batches, props) -> Result<IceTable>`
1. **Split** — `split_batches_row_aligned` cuts the input into up to one group
per core using **zero-copy** `RecordBatch::slice`s (no row is copied or
re-materialized; slicing only chooses where the file boundaries fall). The
concatenation of the groups in order is exactly the input rows in input order.
2. **Encode** — the groups are encoded across all cores on the ONE gatling
fork-join engine (reusing `ingest_parallel_props`); compression runs *inside*
that stage, so zstd/snappy is parallelised for free.
3. **Commit** — `files_per_commit = groups.len()`, so **all** the files land in a
**single** `fast_append` snapshot — matching a lone `append`'s one-snapshot
result, not N snapshots.
File count is chosen from machine parallelism and row count:
`nfiles = (total_rows / APPEND_PARALLEL_MIN_ROWS).clamp(1, cores)` — one file per
core, capped so every file carries at least `APPEND_PARALLEL_MIN_ROWS` (8 192 ≈
the small end of a Parquet row group).
## f2 — functionally equivalent, NOT byte-identical (L2)
Unlike the footer-stat fold (which is byte-identical), `append_parallel` changes
the **physical layout**: the snapshot holds `nfiles` data files instead of one,
so a full scan reads the rows in a different file order. It is *functionally*
equivalent to `append` — the same rows with the same per-column values land in
the same one snapshot, so any order-independent read (row multiset, `count`,
`sum`, filtered / point lookups, SQL aggregates) returns an identical result.
Callers that require the exact single-file byte layout (or a stable scan order)
keep calling `append`.
**Small-write fallback (byte-identical).** Below `APPEND_PARALLEL_MIN_ROWS` total
rows, `nfiles` collapses to 1 and the call takes the plain single-file
`append_props` path — byte-identical to `append` — because an extra Parquet
footer + manifest entry per split file would outweigh the encode win on a tiny
write.
## f3 — what is asserted (RED-when-broken)
`skade/src/write.rs` `#[cfg(test)]`:
- **`append_parallel_matches_append_functionally`** — a large `append_parallel`
and a serial single-file `append` of the same batch are read back and compared
as row multisets: identical rows, identical per-column values (order-independent).
- **`append_parallel_fans_into_multiple_files_and_falls_back_when_small`** — a
large append must fan into **>1** data file (if it regressed to one, the encode
serialized on one core), while a sub-threshold append must fall back to a
**single** file. Both halves guard the split/fallback decision directly.
## f4 — bench arm (LIGHT; needs the Loki HEAVY pass)
`bench/examples/nornir-bench.rs` · `skade.append_parallel_encode` A/Bs the two
placements over a zstd batch (`SKADE_APE_ROWS`, default 200 k rows): lane A is the
serial single-file `append` (encode pins ONE core), lane B is `append_parallel`
(encode fans across all cores). It reports `serial_rows_per_sec` vs
`parallel_rows_per_sec`, the `speedup`, and the **core-saturation** of each lane
(`serial_cores_busy` ~1, `parallel_cores_busy` toward `cores_total`); the parallel
lane is expected faster with far more cores busy.
**These are LIGHT (contended-box) measurements** — indicative, not authoritative.
The parallel lane runs measurably faster than the serial lane, but the exact ratio
is not pinned in-repo; the authoritative figure is owed from the serial **Loki
HEAVY** pass (issue #88) on an idle box, as with the rest of the write-path arms.
Consistent with the ⛔ root law, the fan-out uses the ONE gatling fork-join engine
— no rayon.