# arrow-ipc metadata-tail benches — the bench-first gate
<!-- nornir:class: reasoning -->
> **LAW (bench-first):** the throughput benches for the *current* arrow-ipc
> write path must exist and be **baselined before** any write-path optimisation
> is written, so a new impl's win is a measurable delta against a real number.
> This doc records the five baseline benches, the numbers they produced, and the
> gate the three planned optimisations (A/B/C) must beat.
>
> Spec: `workspace_nordisk/znippy-arrowipc-analysis.md` §2. Benches live in
> `xtask/examples/nornir-bench.rs` (the same nornir `Bencher` harness as every
> other znippy bench); baseline rows are appended to `bench_history.jsonl` at the
> workspace root. **No optimisation is implemented in this pass — only the
> benches + their baseline.**
---
## Why these five exist
The existing end-to-end benches (`ArchiveBench`, `StreamBench`) measure
`compress_mbs` / `decompress_mbs` but **fold the metadata seal into compress
time** and never touch the random-access read path. The whole single-threaded
metadata tail — sort + lookup re-materialisation + fst trie + manifest + footer —
and the per-call re-read on `locate_file` are therefore **invisible** in today's
numbers. These five benches make each of those stages a stable, isolated metric.
The blob pipeline (left, the `1→N→1` no-barrier gatling flow) is already
near-optimal and well-benched — see `.nornir/gatling.md` and the
`assets/gatling_flow.svg` skade-style diagram for its full 1→N→1 saturation
story. The **red box** is what this baseline measures: the single-threaded seal
tail that runs *after* the gatling pipeline has drained, plus the cold read path.
---
## The five benches
| # | id (family) | What it isolates | Headline metric | Drives optimisation |
|---|---|---|---|---|
| B1 | `znippy.subindex_write_{s,m,l}` | `ArrowIpcSink::push_subindex` over N base-schema rows — Arrow IPC serialise + the `accumulate_lookup` copy. No blob I/O, no trie. | `subindex_ms`, `subindex_mrows_s` | (D) streamed sub-index write; (A1) shedding base cols |
| B2 | `znippy.footer_seal_{s,m,l}` | `Box::new(sink).finish()` — sort all rows + 2nd full materialisation (lookup sub-index) + fst trie + manifest + footer + `sync_all`. **The headline bottleneck.** | `seal_ms`, `seal_mrows_s` (+ `sort_ms` sub-cost) | **(A) incremental / parallel seal** |
| B3 | `znippy.random_lookup_cold_{s,m,l}` | K random `locate_file()` calls on a sealed N-row archive — today re-reads manifest + both reserved sections + a full `decode_lookup` per call. | `lookup_ns_p50`, `lookup_ns_p99`, `lookups_per_s`, `manifest_reads_per_lookup` | **(B) cached `ArchiveReader`** |
| B4 | `znippy.prefix_meta_{s,m,l}` | `get_files_meta_with_prefix()` at ~1% / 10% / 100% selectivity — trie range-seek + `decode_lookup`. | `prefix_ms_{1,10,100}pct` | (B) cached reader |
| B5 | `znippy.decompress_subset_{1,10}pct` | Selective restore: extract a random 1% / 10% of files via `get_file` (locate + pread + decompress + verify) — the workload `decompress_mbs` says nothing about. | `extract_ms`, `extract_files_per_s`, `bytes_out_mbs` | (B) cached reader |
**Row scales** (the seal scales with *rows*, not bytes): `S = 10k`, `M = 100k`,
`L = 1M` (heavy, gated behind `NORNIR_BENCH_SKIP_HEAVY` exactly like
`StreamBench`). B1/B2 drive `ArrowIpcSink` directly with synthesized `BlobMeta`
rows (one distinct, namespace-scattered path per row — worst case for the sort +
fst). B3/B4/B5 build a *real* sealed v0.7 archive via `compress_stream` so the
true read path (manifest + lookup + trie + blob pread + blake3 verify) is
exercised. The random subset uses a deterministic LCG so runs are reproducible.
Each bench `run()` **inject-asserts**: e.g. `random_lookup` fails unless all K
targets are located; `prefix_meta` fails unless the empty prefix returns exactly
N rows. A `#[cfg(test)] mod arrowipc_bench_tests` drives each bench at the S
scale and asserts the reported metrics are real and internally consistent (the
"tests inject values, not just no-crash" LAW).
---
## BASELINE (2026-06-18 · threadripper-3975wx · 32 cores · znippy 0.9.0)
**B1 — `subindex_write`** (push one base-schema sub-index of N rows)
| scale | rows | subindex_ms | subindex_mrows_s | index_bytes_per_row |
|---|---:|---:|---:|---:|
| S | 10 000 | 1.08 | 9.29 | 108.3 |
| M | 100 000 | 9.85 | 10.15 | 108.1 |
| L | 1 000 000 | 195.66 | 5.11 | 108.1 |
**B2 — `footer_seal`** (the headline bottleneck: sort + 2nd materialisation + trie + manifest + footer + fsync)
| scale | rows | seal_ms | sort_ms | seal_mrows_s | seal_appended_bytes |
|---|---:|---:|---:|---:|---:|
| S | 10 000 | 11.02 | 1.11 | 0.91 | 1 141 455 |
| M | 100 000 | 114.06 | 14.02 | 0.88 | 10 905 042 |
| L | 1 000 000 | 1 545.15 | 357.77 | 0.65 | 108 258 914 |
*Seal throughput collapses ~0.91→0.65 Mrows/s from S→L; the sort alone is 357ms
of the 1.5s L seal. This is the number idea (A) must beat.*
**B3 — `random_lookup_cold`** (K random `locate_file` calls, fresh handle each — today's re-read-everything path)
| scale | rows | lookups | lookup_ns_p50 | lookup_ns_p99 | lookups_per_s | manifest_reads_per_lookup |
|---|---:|---:|---:|---:|---:|---:|
| S | 10 000 | 2 000 | 838 628 (0.84 ms) | 953 887 (0.95 ms) | 1 177 | 3 |
| M | 100 000 | 500 | 19 228 069 (19.2 ms) | 23 343 734 (23.3 ms) | 51.4 | 3 |
| L | 1 000 000 | 100 | 281 045 524 (281 ms) | 350 218 230 (350 ms) | 3.5 | 3 |
*Per-call latency grows **linearly with N** (0.84 ms → 281 ms, ~335× for 100× the
rows) because every cold call re-decodes the entire lookup sub-index. Idea (B)'s
cached reader should make this ~flat (one fst `get` + array slice) and drop
`manifest_reads_per_lookup` 3 → 0 after open.*
**B4 — `prefix_meta`** (`get_files_meta_with_prefix` at ~1% / 10% / 100% selectivity)
| scale | rows | prefix_ms_1pct | prefix_ms_10pct | prefix_ms_100pct |
|---|---:|---:|---:|---:|
| S | 10 000 | 1.99 | 1.03 | 1.77 |
| M | 100 000 | 19.79 | 18.19 | 30.95 |
| L | 1 000 000 | 284.09 | 293.34 | 393.99 |
*Even a 1%-selective prefix pays ~the full-scan cost (the whole lookup is decoded
before the range is sliced) — visible as `prefix_ms_1pct ≈ prefix_ms_100pct` at
each scale. Idea (B) decouples it.*
**B5 — `decompress_random_subset`** (selective restore via `get_file`, M corpus = 100k files, extraction count capped at 2 000)
| arm | files_total | files_extracted | extract_ms | extract_files_per_s |
|---|---:|---:|---:|---:|
| 1pct | 100 000 | 1 000 | 18 195.00 | 55.0 |
| 10pct | 100 000 | 2 000 | 37 003.15 | 54.0 |
*~54 files/s because each `get_file` is a cold `locate_file` (full lookup decode)
+ pread + decompress + blake3 verify. The lookup re-read dominates; idea (B)
should lift `extract_files_per_s` by ~the same factor it cuts `random_lookup`.
(`bytes_out_mbs` rounds to 0.0 — the synthetic payloads are tens of bytes each;
`extract_files_per_s` is the comparable metric for this selective-restore path.)*
> Run notes: `S/M/L = 10k/100k/1M` rows. `random_lookup` and
> `decompress_subset` are O(K·N) on the cold path, so K (lookups) and the
> extraction count are capped as N grows (S=2000, M=500, L=100 lookups;
> ≤2000 extractions) — the per-call/per-file metric is N-normalised, so fewer
> samples at L is fine and the wall-time stays bounded.
These are the numbers in `bench_history.jsonl` (the `arrow-ipc baseline` run,
date `2026-06-18`). They are the gate. An optimisation is accepted only if:
- **(A) incremental / parallel seal** drops `footer_seal.seal_ms` (raises
`seal_mrows_s`), most visibly at M / L scale, with **no** regression > 10% on
the existing `compress_mbs` / `decompress_mbs` (the WORKSPACE.md release gate).
- **(B) cached `ArchiveReader`** drops `random_lookup.lookup_ns_p50` (the design
expects ~1–2 orders of magnitude), collapses
`random_lookup.manifest_reads_per_lookup` from **3 → 0** after open, and
proportionally drops `prefix_meta.prefix_ms_*` and raises
`decompress_subset.extract_files_per_s`.
- **(C) append-mode + explicit `seal()`** reuses (A)'s incremental seal; it is
measured later by an `append_seal` variant of B2 (full N-row archive + delta of
K rows; `append_seal_ms` vs full `seal_ms`). Baselined here as the full seal.
---
## How to run
```bash
# Target dir is pinned to fast disk via .cargo/config.toml.
# Full suite (planet skipped); the five families below are part of it:
NORNIR_MACHINE=threadripper-3975wx NORNIR_BENCH_SKIP_PLANET=1 \
cargo run -p xtask --release --example nornir-bench
# Skip the 1M-row (_l) heavy scale for a fast focused pass:
NORNIR_BENCH_SKIP_PLANET=1 NORNIR_BENCH_SKIP_HEAVY=1 \
cargo run -p xtask --release --example nornir-bench
# Harness self-test (inject-assert, no timing):
cargo test -p xtask --release --example nornir-bench arrowipc_bench_tests
```
---
## Sequencing (from the analysis)
1. **These benches first (this commit)** — implement + baseline. ✅
2. **(B) cached `ArchiveReader`** — lowest risk, biggest read win, zero format
change. Validate against B3/B4/B5.
3. **(D) small seal cleanups** (no per-row `String` clone, streamed sub-index) —
validate on B1/B2.
4. **(A) incremental / parallel seal** — validate on B2.
5. **(C) append + `seal()`** — last; add the `append_seal` bench alongside.
---
## A/B AUDIT (2026-06-18) — `ArrowIpcSink` (A) vs `ArrowIpcSinkAppend` (B)
> **The question:** does the new resume/append/seal feature hurt the *normal*
> write speed? (*"speed inte drabbas"*) The new capability is a near-identical
> **clone** of the original writer, `znippy_common::ArrowIpcSinkAppend`, that adds
> append/resume (`open_existing` → append blobs + rows → re-seal) and the
> first-class native blob-append (`append_files`, the `compress --append`
> primitive #23 noted was missing). The ORIGINAL `ArrowIpcSink` hot path is
> **untouched** (byte- and perf-identical = the A baseline). The clone's
> fresh-write path is a line-for-line copy, so it should be a **zero-cost
> superset** on the normal path.
**Design (clone, not refactor).** A new module `znippy-common/src/meta_sink_append.rs`
holds `ArrowIpcSinkAppend` — field-for-field identical to `ArrowIpcSink`, same
`push_subindex` + `finish` (same sub-index serialise, same sorted lookup, same
fst trie, same manifest, same `ZNPYMIDX` footer). The only added surface:
`open_existing(path)` (reads the full manifest incl. reserved sections, finds
`blob_end` = lowest `index_offset`, recovers existing data rows from the sorted
lookup section, positions the cursor at `blob_end` so new bytes overwrite the
rebuildable tail), `blob_end()`/`recovered_rows()` accessors, and `append_files()`
(native: codec-compress new files → append blobs past `blob_end` → re-seal merged
old+new rows). `finish()` re-emits recovered rows as a data sub-index (so the
ordinary reader still lists them) and `set_len`-truncates so no stale footer
lingers. One minimal additive extraction in `index.rs`: `read_znippy_full_manifest`
(public wrapper over the existing private `read_full_manifest`) — read-only, the
original write path unchanged.
### B1 — `subindex_write` (push one base-schema sub-index of N rows): A vs B
| scale | rows | A `subindex_ms` | B `subindex_ms_b` | Δ% | A `mrows/s` | B `mrows/s` | bytes (A=B) |
|---|---:|---:|---:|---:|---:|---:|---:|
| S | 10 000 | 1.20 | 1.21 | +0.8% | 8.33 | 8.24 | 1 082 696 |
| M | 100 000 | 18.82 | 15.61 | −17.1% | 5.31 | 6.41 | 10 814 088 |
| L | 1 000 000 | 214.73 | 217.95 | +1.5% | 4.66 | 4.59 | 108 126 664 |
### B2 — `footer_seal` (the headline seal: sort + 2nd materialise + trie + manifest + footer + fsync): A vs B
| scale | rows | A `seal_ms` | B `seal_ms_b` | Δ% | A `mrows/s` | B `mrows/s` | total_bytes | byte-identical |
|---|---:|---:|---:|---:|---:|---:|---:|:---:|
| S | 10 000 | 10.55 | 10.76 | +2.0% | 0.95 | 0.93 | 2 224 151 | ✅ |
| M | 100 000 | 125.16 | 125.76 | +0.5% | 0.80 | 0.80 | 21 719 130 | ✅ |
| L | 1 000 000 | 1 730.10 | 1 753.09 | +1.3% | 0.58 | 0.57 | 216 385 578 | ✅ |
*`total_bytes` is **identical** A=B at every scale (asserted byte-for-byte by the
`clone_fresh_path_is_byte_identical_to_original` unit test): the clone produces
the exact same v0.7 container. The Δ% on `seal_ms` is single-shot timing noise
around 0 (M went the other way at L vs M); none exceeds the gate.*
### PARITY VERDICT — **PASS** ✅ (WORKSPACE.md ≤10% no-regression gate)
The append-capable clone is a **zero-cost (≈noise) superset** of the original on
the normal, non-append write path:
- **Output is byte-identical** — same bytes, same length, A=B, all scales.
- **Throughput within the gate** — worst seal delta **+2.0%** (S), M **+0.5%**,
L **+1.3%**; worst subindex delta **+1.5%** (L). All **well under the ≤10%
gate**. No regression to report.
- The `compress_mbs`/`decompress_mbs` end-to-end path is **completely unaffected**
— the original `ArrowIpcSink` it uses is untouched (the clone is opt-in, only
reached via `ArrowIpcSinkAppend::new`/`open_existing`/`append_files`).
### NEW capability MEASURED — `resume_append` (open existing → append 1 000 blobs → re-seal)
| scale | rows_before | rows_added | `reopen_ms` | `append_ms` (end-to-end) | `reseal_mrows_s` | blob_bytes_added |
|---|---:|---:|---:|---:|---:|---:|
| S | 10 000 | 1 000 | 1.40 | 22.41 | 0.49 | 57 094 |
| M | 100 000 | 1 000 | 31.20 | 172.31 | 0.59 | 57 094 |
| L | 1 000 000 | 1 000 | 323.54 | 1 754.42 | 0.57 | 57 094 |
*Resume cost is finally **measured**, not just functionally tested. `reopen_ms` is
the open-existing cost (read manifest + recover the sorted lookup of N rows);
`append_ms` is the full end-to-end (reopen + append 1k blobs + re-sort & re-seal
the merged N+1k rows). The append re-seal is **≈ a full fresh seal of N+K rows**
(L: append_ms 1754ms vs full footer_seal_l 1730ms) because re-sealing rebuilds the
sorted lookup + trie + manifest over the whole row set — expected, and the honest
cost of an immutable, content-addressed sealed format. The append blob write
itself is trivial (57 KB); the cost is the metadata re-seal, which scales with
total rows exactly like B2. A future (A) incremental seal would lower both B2 and
this in lockstep.*
> Benches: `xtask/examples/nornir-bench.rs` — `SubindexWriteB`/`FooterSealB`
> (`*_b_{s,m,l}`) + `ResumeAppend` (`resume_append_{s,m,l}`), heavy `_l` gated by
> `NORNIR_BENCH_SKIP_HEAVY` like A. Inject-assert tests:
> `arrowipc_bench_tests::{ab_clone_write_seal_within_parity_gate,
> resume_append_recovers_and_reseals}` + the structural parity/round-trip unit
> tests `meta_sink_append::tests::{clone_fresh_path_is_byte_identical_to_original,
> native_append_roundtrips_old_and_new_files}`. Row appended to
> `bench_history.jsonl` as the `arrow-ipc-ab` suite, literal date `2026-06-18`.
> Run: `ZNIPPY_BENCH_ONLY=subindex_write,footer_seal,resume_append
> NORNIR_MACHINE=threadripper-3975wx cargo run -p xtask --release --example nornir-bench`.