# znippy-zoomies — benchmarks (the crushing numbers)
> **Class:** `benchmarks`. The measured head-to-head numbers, HEAVY runs only (no
> small/smoke mode — see [`reasoning.md`](reasoning.md)). Host CPU, core count and
> peak memory are recorded **per rival**. How to reproduce: [`guide.md`](guide.md) and
> [`decompressors-guide.md`](decompressors-guide.md).
## f0 — the decompressor bake-off (the star)
One large deterministic corpus is compressed into **every** rival's format, then
decompressed back by **every** decompressor on the box — znippy-zoomies' parallel
`lgz` / `lbunzip2` against the legacy serial tools *and* the other parallel C
decoders, all at full cores + max memory. The harness warmup-primes each decoder and
keeps the **min of N≥3** timed passes, so re-runs reproduce rather than chase
single-shot noise. Each system is one row; the contested metrics are the columns.
**Bold = the winning cell for that metric.** The table is regenerated from the latest
warehouse `BenchRun` by the marker below.
**The honest scope (read this before the numbers).** znippy-zoomies is the fastest
decoder *within the two legacy formats it targets* — **not** the fastest decoder in the
field. Concretely, on this corpus:
- **gzip:** `znippy-gz` beats serial `gzip` and parallel `pigz` per core — a clean
win *within the format*, though gzip decode is Amdahl-/bandwidth-bound and does not
scale to all 12 cores (see the core-scaling verdict below).
- **bzip2:** `znippy-bz2` leads the bz2 field measured here — it beats serial
`bzip2`/`pbzip2` by ~13× per core and **trades the lead with the mature C `lbzip2`**
(808 vs 764 MB/s in the v0.1.15 min-of-3 render; a later same-host run had it a
photo-finish the other way — see the **Reproducibility caveat** below). Against the
stock serial/parallel bzip2 tools the win is decisive and repeatable; against C
`lbzip2` it is a genuine same-format tie, not a decisive lead.
- **modern codecs:** `zstd`, `lz4`, `xz`/`pixz`, `plzip` decode their own formats
**1.5–2.6× faster than our `lgz`**. Those formats were designed for fast decode; gzip
and bzip2 were not. Our value is multi-core decode of the formats you *already have*.
**Grouping rule (the apples-to-apples fix).** Every table below pairs a decoder **only**
with rivals in its **own format** and its **own core-mode**: single-core tools sit with
single-core tools, all-core tools with all-core tools. A 12-core `decompress_mbs` is
**never** set against a 1-core rival and called a win — that apples-to-pears framing was
a real defect and it is gone. Cross-format and cross-mode reading is led by
**`percore_mbs`** (throughput per host core), the one metric that stays honest when the
core counts differ.
**v0.1.15 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB (multi-member
gzip = 128 × 8 MB independent members; bzip2 `-9`) · min-of-3 · 12 cores · 2026-07-10**
### 🍎 DEFLATE family — `znippy-gz` vs `gzip` vs `pigz`
**Single-core** — serial `gzip` and the pinned `znippy-gz_st` (one gatling worker):
<!-- nornir:gen:start:benches best=decompress_s:low cols=decompress_mbs,percore_mbs,decompress_s,peak_mem_mb compare=znippy-gz_st,gzip self=znippy-gz_st -->
**v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13**
| workload | decompress_mbs | percore_mbs | decompress_s | peak_mem_mb |
|---|---|---|---|---|
| znippy-gz_st | **655.62** | 54.63 | 1.56 | 239.80 |
| gzip | **684.25** | 57.02 | 1.50 | 1.90 |
<!-- nornir:gen:end:benches -->
**Multi-core** — parallel `pigz` and full-core `znippy-gz`:
<!-- nornir:gen:start:benches best=decompress_s:low cols=decompress_mbs,percore_mbs,decompress_s,peak_mem_mb compare=znippy-gz,pigz self=znippy-gz -->
**v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13**
| workload | decompress_mbs | percore_mbs | decompress_s | peak_mem_mb |
|---|---|---|---|---|
| znippy-gz | **5760** | 480.03 | 0.18 | 416.60 |
| pigz | **589.32** | 49.11 | 1.74 | 2.50 |
<!-- nornir:gen:end:benches -->
The `znippy-gz` / `znippy-gz_st` cells are held for the next bench-machine wave (the
local warehouse carries no `lgz` `BenchRun`; a render on the bench box fills them from
the marker params above). The **core-scaling probe** measured `znippy-gz` at
**~2.7 / 12 cores** decoding this corpus to `/dev/null` (`lgz corpus.gz > /dev/null`, the
serial-stdout path) and **~7.8 / 12 cores** on the same input written with `-o file` (the
positional-`pwrite` path), at **~75 MB/s/core** — ahead of both `gzip` (60.35/core) and
`pigz` (51.30/core). See the core-scaling verdict below for why gzip decode is bandwidth-
rather than core-bound.
### 🍐 bzip2 family — `znippy-bz2` vs `bzip2` vs `lbzip2` vs `pbzip2`
**Single-core** — serial `bzip2` and the pinned `znippy-bz2_st`:
<!-- nornir:gen:start:benches best=decompress_s:low cols=decompress_mbs,percore_mbs,decompress_s,peak_mem_mb compare=znippy-bz2_st,bzip2 self=znippy-bz2_st -->
**v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13**
| workload | decompress_mbs | percore_mbs | decompress_s | peak_mem_mb |
|---|---|---|---|---|
| znippy-bz2_st | **129.29** | 10.77 | 7.92 | 75.60 |
| bzip2 | **60.02** | 5 | 17.06 | 5.10 |
<!-- nornir:gen:end:benches -->
**Multi-core** — parallel `lbzip2`, `pbzip2`, and full-core `znippy-bz2`:
<!-- nornir:gen:start:benches best=decompress_s:low cols=decompress_mbs,percore_mbs,decompress_s,peak_mem_mb compare=znippy-bz2,lbzip2,pbzip2 self=znippy-bz2 -->
**v0.1.19 · AMD Ryzen 9 7900 12-Core Processor · 61 GiB · corpus 1024 MB · min-of-3 · 12 cores · 2026-07-13**
| workload | decompress_mbs | percore_mbs | decompress_s | peak_mem_mb |
|---|---|---|---|---|
| znippy-bz2 | **744.39** | 62.03 | 1.38 | 80.60 |
| lbzip2 | **759.55** | 63.30 | 1.35 | 90.60 |
| pbzip2 | **60.29** | 5.02 | 16.98 | 41.80 |
<!-- nornir:gen:end:benches -->
**Reproducibility caveat — the bz2 lead over C `lbzip2` is real but *close* and
occupancy-sensitive (read this before quoting "808 vs 764").** The `808.39` / `764.48`
cells above are the v0.1.15 min-of-3 render, normalised to a flat **12 nominal cores**
(`percore_mbs` = `decompress_mbs` / 12 exactly — a nominal core count, not a measured
`cores_busy`). A *later* heavy bake-off on the **same Loki host and the same corpus**
measured the head-to-head **closer, with the sign flipped**: `znippy-bz2` **695 MB/s @
11.41 cores_busy** vs `lbzip2` **768 MB/s @ 11.81 cores_busy**. Nothing in the decode path
changed between the two runs — the `lbzip2` / `gatling` decode source is **byte-identical**
since the commit that produced these numbers (`33ff958`, "beat system lbzip2 on multicore
decode"; the only later commits are this docs regroup and an *optional*, off-by-default
`nornir-testmatrix` feature-dep repoint, neither touching the hot path). The delta is
therefore **not a code regression** but run-to-run variance: bzip2 multi-core decode is
**L3-latency-bound** (diagnosed in `33ff958`), so throughput tracks *measured* core
occupancy, and the two runs simply disagree on occupancy (a flat nominal 12 vs a contended
11.41 / 11.81). On a fully idle box the committed min-of-3 has `znippy-bz2` ahead; on a
loaded box the more core-hungry scaler gives up more throughput than the mature C tool and
can trail it. **Honest reading: `znippy-bz2` and C `lbzip2` are within run-to-run
variance on this corpus — a genuine same-format photo-finish, not a decisive lead.** When
quoting the bz2 result, cite the per-core figure *and* the measured `cores_busy`, not a
bare "808 vs 764".
*Columns — `decompress_mbs`: decode throughput (MB/s) · `percore_mbs`: decode throughput
per host core · `decompress_s`: decode wall time · `peak_mem_mb`: peak resident set.
`znippy-gz` / `znippy-bz2` are ours; every other row is a stock rival. The tables
scope to **decode** metrics — the only ones that differ between decoders of the same
format. Local render note: the warehouse on this box has no `BenchRun`, so these regions
are hand-filled with the numbers the bench-machine committed; `nornir docs render
znippy-zoomies` on the bench box reproduces them from the marker params.*
*Format properties (identical within a format family — set by the reference compressor,
not the decoder): gzip-family `compress_mbs` ≈ 256 MB/s at `ratio` 4.64; bzip2-family
≈ 23 MB/s at `ratio` 7.50. They describe the archive, not the decode race above.*
**How to read it — the wins.** The two formats parallelise very differently, and we
say so plainly. **`znippy-bz2` is the genuine all-cores scaler and trades the bz2 lead
with C `lbzip2`:** bzip2's block decode is CPU-heavy, so the parallel path reaches **~8.5×
at 12 cores** (~71 % efficiency, ~8 cores kept busy) — it decodes `bzip2 -9` at **67.37
MB/s/core vs `bzip2`'s 5.02 and `pbzip2`'s 5.04**, a **~13.4× lead** over the stock
serial/parallel bzip2 tools, and at **808 MB/s it edges even the mature C `lbzip2`
(764 MB/s, 63.71/core)** in the v0.1.15 min-of-3 render — the single biggest,
*same-format* gap over the *stock* tools in the run. Against C `lbzip2` specifically the
result is a run-to-run photo-finish (a later same-host run had `lbzip2` ahead 768 vs 695
at 11.81 vs 11.41 measured cores) — see the **Reproducibility caveat** above; the decode
path did not change between the runs. **`znippy-gz` is a more modest parallel win, and Amdahl-/bandwidth-
bound by design:** gzip's DEFLATE decode is already so fast that little compute remains
to spread across cores (see the verdict below). It still lands the fastest gzip decode
measured at **~75 MB/s/core vs `gzip`'s 60.35 and `pigz`'s 51.30**, but that is a
format-limited speedup, *not* an all-cores one.
**Core-scaling verdict — why `lgz` uses ~2.7 cores here, and whether it *should* use
all 12.** The honest answer is *both*: real, already-built headroom the bench under-shows
*and* a hard format ceiling.
- **Not a single-stream stall on this corpus.** The gzip corpus is 128 **independent**
8 MB members (`split -b 8M … | gzip`), so it is embarrassingly parallel — the classic
"one LZ77 32 KB window serialises everything" argument does **not** apply here. `lgz`
already exploits every parallel escape hatch in code: parallel multi-member scan
(`speculative::find_gzip_members_parallel`), `Z_FULL_FLUSH` boundary split
(`deflate_scan` + `chunk`), and a trailer-verified speculative block-boundary guess
(`speculative::speculative_decode`) for streams with none. It is not being lazy.
- **The 2.7 figure is the `> /dev/null` (stdout) path.** The bench runs
`lgz corpus.gz > /dev/null`, which forces the single-writer ordered-sink path (stdout
cannot seek). `lgz`'s own `-o file` path decodes the *same* input at **~7.8 / 12 cores**
via positional `pwrite` (each worker writes its member at a prefix-sum offset, no
serial collector). So on this corpus the realistic ceiling is **~7.8 cores, not 2.7** —
the gap is a measurement artifact of piping to `/dev/null`, and it is fixable for free
by benching the `-o` deployment shape.
- **~7.8, not 12, is the *inherent* gzip ceiling.** gzip decode is memory-bandwidth-
bound: serial `gzip` already reconstructs the whole 1 GB corpus in 1.41 s (724 MB/s),
so there is only ~1.4 core-seconds of inflate work, most of it DRAM-write-bound LZ77
copies that saturate the memory controller at a handful of cores no matter the thread
count. This is exactly why `znippy-bz2` *does* reach 11.7/12 (bzip2 blocks are
compute-heavy) while `znippy-gz` cannot — the difference is per-byte compute
intensity, not effort.
- **Context — `lgz` already leads the field.** `pigz`, the reference parallel gzip,
reaches only **1.27 / 12 cores** on this same corpus (its decoder is essentially
serial). `lgz` at 2.7 (stdout) already ~2.1× that, and at 7.8 (`-o`) ~6× — no gzip
decoder parallelises better, and a *plain* single-member `gzip foo` stream is
serial-by-format for everyone, `pigz` included.
**Verdict:** Rickard is right that the bench leaves cores idle it need not — surfacing the
`-o` path lifts `lgz` from ~2.7 to ~7.8/12 with zero code change, so "use more cores" is
a legitimate, actionable ask *for the measurement*. But "all 12 cores" is not physically
reachable for gzip decode: it is bandwidth-bound, unlike bzip2's compute-bound blocks
that genuinely do hit 11.7/12. lgz already implements both parallel paths and already
out-scales every gzip tool measured.
**How to read it — the losses (stated plainly, not buried).** Two things these tables
do **not** show, and we won't pretend otherwise:
1. **Modern codecs decode faster than our `lgz`, full stop** — `lz4`, `pixz`, `xz`,
`zstd` and `plzip` all reconstruct their own formats **1.5–2.6× faster**. They were
engineered for fast decode; gzip and bzip2 were not.
2. **`lgz` is bandwidth-bound, not all-core** — as the verdict above states, gzip decode
tops out around ~7.8/12 cores even on the ideal multi-member input; only the bzip2
family (`znippy-bz2`) is a true all-cores scaler here.
So the defensible claim is narrow and true: **multi-core decode for the ubiquitous
gzip/bzip2 formats every tool already reads**, on data you already hold and can't
re-encode — where the stock tools leave the box idle. For bzip2 that is a decisive win
over the stock serial/parallel tools and a same-format tie with the mature C `lbzip2`
(a run-to-run photo-finish — see the Reproducibility caveat); for gzip it is the fastest per-core
decode in a format that simply cannot fill 12 cores. It is *not* "fastest decoder
overall", and the numbers above are the reason we don't say that.
**Memory is recorded per rival** — the `peak_mem_mb` column is the GNU `time -v` peak
RSS of each decode. Ours buys its throughput with a larger working set (the ring of
in-flight slots: `znippy-gz` 906.8 MB, `znippy-bz2` 655.2 MB) — a deliberate
space-for-cores trade on a 61 GiB host, called out honestly rather than hidden.
## f1 — the compressor bake-off (`ldeflate`, the write side)
Everything above this line is **decode**. `ldeflate` is the workspace's one write-side
crate — the parallel DEFLATE/zlib engine, plus the gzip, ZIP and JAR deflators built on
it — and it is benched on a different question, so it gets its own section rather than a
column in f0.
**Why there is no rival column.** f0 races every decoder on the box to reconstruct *one
given archive*, which is a fair fight because the input is fixed. A compressor's output
is not fixed: `gzip -6` and this crate produce different bytes, so "MB/s" alone ranks
nothing. The contested number for a compressor is what its **schedule** costs against
the *same crate's* serial path over the *same* objects at the *same* level — which is
why the baseline row here is `compress_objects_serial`, not `gzip`. `ldeflate 0.1.0` is
published on crates.io (2026-08-03).
**One engine, four containers.** `plan::compress_objects` is the only scheduler in the
crate. gzip, ZIP and JAR all route through it and differ **only in framing** —
`Container::wrap` puts a header and a trailer around a body the planner already
produced. Total new scheduling code across the three containers: **none** (LAW 5). The
planner picks one of three unit kinds per object, by size:
| unit | when | why |
|---|---|---|
| `Unit::Batch` | object < `coalesce_below` (**64 KiB**) | one task drains many objects **reusing one encoder** — `reset()` at ~1.6 µs instead of `Compress::new` at ~4.1 µs, ~2.5 µs saved per object |
| `Unit::Whole` | `coalesce_below` ≤ object < `split_above` | one task, one object |
| `Unit::Segment` | object ≥ `split_above` (**4 MiB**) | one object across many tasks, cut at `Z_FULL_FLUSH`, so a 400 MB blob cannot serialise a worker |
**93.8% of real git objects take the first path** — measured over 48 917 objects from
eight repositories on this box, not assumed. That is why coalescing is the axis that
matters for git and splitting is the axis that matters for LFS, and why **one `Config`
serves both**: the classification is per object, not per call.
**`limits.rs` is the other half of that story, and it is a scar.** Both `segment` and
`member` are *amplifying* parameters — the caller passes two small integers and the crate
allocates a structure proportional to their quotient. On 2026-08-03 a test crossing
`len = 3_000_000` with `member = 1` (three million one-byte gzip members) reached ~497 GB
anon RSS on a 499 GB box; the OOM killer fired and took a shared tmux service and three
unrelated sessions down with it. `MAX_PIECES = 262 144` now **refuses** such a request,
naming what was exceeded. An `Err`, never a clamp: quietly raising `piece` until the
count fits would change the segment plan, therefore the output bytes, therefore the
determinism guarantee — without the caller ever learning. Every test and bench in the
crate now runs under `(ulimit -v 25165824)`.
### The honest framing — read this before the scaling table
**Above 8 threads the speedup column measures the box, not the code.** oden is shared,
and `ldeflate` takes every core it can get: cores actually held tracked `32 − other
tenants` almost exactly across a seven-run batch. So the same binary over the same corpus
produced 32-thread speedups from **10.38× to 23.20× — a 123.5% spread** — while the CPU
cost of the schedule held to **1.025×–1.054× of serial**. The first column is a statement
about machine occupancy; the last is a statement about this crate.
An earlier headline — **28.30× at 32 threads, 88% efficiency, 2266 MB/s** — is *not
refuted*: it is consistent with, and slightly conservative against, the re-runs, which
carried forward predict 29.2–31.2× at that occupancy. But it was presented as a property
of the crate when it is a property of that run's occupancy, and **it cannot be reproduced
on a shared box**. It is quoted here only as retired framing. The durable claim is the
CPU one.
### Scaling — re-measured 2026-08-04
**oden · AMD Ryzen Threadripper PRO 3975WX · 32 physical cores · SMT off · 499 GB RAM ·
30 770 REAL git objects, 715.4 MB, median 986 B, 93.7% under 64 KiB · level 6 · min-of-3 ·
load 40–47 during the run · 2026-08-04**
<!-- nornir:gen:start:benches best=mb_per_sec:high cols=mb_per_sec,cores_busy,speedup_vs_serial,efficiency_pct compare=deflate.repack_git_objects_st,deflate.repack_git_objects_mt1,deflate.repack_git_objects_mt2,deflate.repack_git_objects_mt4,deflate.repack_git_objects_mt8,deflate.repack_git_objects_mt16,deflate.repack_git_objects_mt self=deflate.repack_git_objects_mt -->
| workload | mb_per_sec | cores_busy | speedup_vs_serial | speedup ÷ cores held |
|---|---|---|---|---|
| deflate.repack_git_objects_st (serial oracle) | 72.5 | — | 1.00 | — |
| deflate.repack_git_objects_mt1 | 69.2 | 0.93 | 0.95 | 1.02 |
| deflate.repack_git_objects_mt2 | 137.1 | 1.84 | 1.89 | 1.03 |
| deflate.repack_git_objects_mt4 | 258.5 | 3.55 | 3.57 | 1.01 |
| deflate.repack_git_objects_mt8 | 424.4 | 5.86 | 5.86 | 1.00 |
| deflate.repack_git_objects_mt16 | 621.1 | 8.48 | 8.57 | 1.01 |
| **deflate.repack_git_objects_mt** (32) | **842.7** | **12.15** | **11.63** | **0.96** |
<!-- nornir:gen:end:benches -->
**Read the last column, not the second-to-last.** This run asked for 32 threads and got
**12.15 cores** — other tenants held the rest — so 11.63× is what 12.15 cores buys, and
the crate converted them at **0.96–1.03 speedup per core held at every width from 1 to
32**. The 32-thread row is inside the carried 10.38–23.20× band and its efficiency against
cores actually held (95.7%) is inside the carried 94.9–97.6% band. Two independent
sessions, two different corpora, same conclusion.
**Carried from 2026-08-03** (`.nornir/ldeflate-design.md` Finding 4b — *not* re-measured
here): 49 497 objects / 1684.7 MiB, **n = 7 independent runs** of min-of-5, each under
`sudo -n systemd-run --scope -p MemoryMax=32G` and run strictly serially.
| threads | speedup min–max (carried) | spread | eff vs cores HELD | **cpu-s vs serial** |
|---|---|---|---|---|
| 1 | 0.99–1.00× | 1.0% | 99–100% | **1.001–1.014×** |
| 2 | 1.96–2.01× | 2.6% | 98.0–100.5% | **0.997–1.018×** |
| 4 | 3.87–3.94× | 1.8% | 97.2–99.0% | **1.009–1.029×** |
| 8 | 7.55–7.74× | 2.5% | 95.7–97.8% | **1.022–1.045×** |
| 16 | 13.76–15.19× | 10.4% | 95.4–96.9% | **1.032–1.048×** |
| 24 | 14.65–21.94× | 49.8% | 95.0–97.3% | **1.027–1.052×** |
| 32 | 10.38–23.20× | **123.5%** | 94.9–97.6% | **1.025–1.054×** |
Speedups at 1–8 threads reproduce to ≤2.5% only because ≤8 cores were always free; the
spread grows with thread count exactly as the request starts to exceed what was
available. **The one column that reproduces at every width is the CPU one.**
### `precompressed.rs` — never compress what is already compressed
The largest avoidable cost on the write side is paying an entropy coder to hand back its
own input. A git object store is full of packfiles; a JAR is full of PNGs. The gate is a
**hint → extension → magic-byte probe**, in that order, and it is deliberately **not** an
entropy estimate: an estimate is a heuristic with a tunable threshold, costs a scan
proportional to the data, and misjudges in both directions. Every probe here is exact — a
fixed byte pattern at a fixed offset, at most `SNIFF_PREFIX_LEN` = 16 byte compares, no
threshold, no sample, no allocation. The hint is honoured in **both** directions, so a
caller who knows better than the magic bytes wins either way.
Acting on the verdict is free because of `stored_deflate` — an **RFC 1951 §3.2.4
stored-block emitter**: a 5-byte block header per 64 KiB and a `memcpy`. It is *not*
`flate2` at level 0, which still builds and resets a compressor and still walks the input
through the encoder state machine. That distinction is where the CPU saving actually
comes from.
**oden · 24 REAL git packfiles + pack indexes · 113.8 MiB · level 6 · min-of-3 · decided
on CPU seconds · load 66 · 2026-08-04 (re-measured)**
| probe | cpu-s per GiB | wall MB/s | ratio |
|---|---|---|---|
| off — deflate everything | 22.85 | 184.8 | 1.0041 |
| **on — store what is already compressed** | **0.99** | **724.6** | 0.9999 |
**→ 23.1× less CPU, 95.7% saved, output +0.41% larger.**
Decided on **CPU seconds, not wall time**, because the box was at load 66 and CPU-seconds
consumed by our own process is far less sensitive to other tenants than the wall clock is.
The `wall MB/s` column is indicative only — the carried 2026-08-03 run of this identical
arm read 267.5 / 2075.5 MB/s on a quieter box while its `cpu-s per GiB` column read
**22.40 / 0.99**, i.e. the load-robust metric moved by under 2% and the wall column moved
by 3×. That is the whole argument for the choice of metric.
**Read the ratio, not the multiplier.** Across four runs on 2026-08-03/04 the CPU
multiplier wandered **19.4× – 23.1×** and the saving held at **94.8% – 95.7%**. The
multiplier is unstable because its *denominator* is ~1 cpu-s/GiB — a small absolute
number that a shared box perturbs easily. **The stable claim is ~95% of the CPU saved for
+0.41% of size**, and the size penalty came out **byte-identical (+0.41%) on every single
run**. It independently reproduces znippy's own 92–95% figure from a different codebase.
**The negative controls, because a 23× that never collapses is not evidence about a
probe.** Same harness, same two arms, 2026-08-04:
| control | corpus | result |
|---|---|---|
| **green** — the claim | 24 packfiles, 113.8 MiB | 20.38× less CPU, 95.1% saved, size **+0.41%** |
| **red 1** — probe has nothing to find | git loose objects, 1422.4 MiB | **1.03×**, 2.8% saved, size +0.11% |
| **red 2** — probe vetoed by `ContentHint::Compressible` | the *same* 24 packfiles | **1.04×**, 4.1% saved, size **+0.00%** |
Red 1 collapses because loose objects are commits, trees and source text, so
`looks_compressed` returns false and both arms do identical work — the residual 1.03× and
+0.11% is a handful of genuinely-compressed blobs in those repos, which is the honest
answer rather than a leak. Red 2 is the tighter control: **identical bytes**, only the
policy differs, and it collapses to 1.04× with the output byte-identical to the
compress-everything arm. The 23× is a property of the probe, not of the two arms
differing in some other way.
### What the compressor costs in ratio
| axis | ratio cost |
|---|---|
| coalescing small objects | **zero** — not one output byte changes, asserted as a test |
| claim size (1 / 4 / 16) | **zero** — and within 0.3% on CPU |
| splitting at the default 1 MiB segment | 2.2%, and only on objects ≥ 4 MiB |
| storing already-compressed input | +0.41% on real packfiles |
*Carried from 2026-08-03 except the last row.* Determinism is what makes the first two
rows sayable at all: the work plan is a pure function of `(object sizes, Config)` and
never reads the core count, so output is byte-identical at 1 worker and at 32 —
`parallel_equals_serial_at_every_worker_count` asserts exactly that, and
`segment_size_actually_changes_the_bytes` proves the assertion is not vacuous by showing a
different config *does* move the bytes.
### The LFS answer is not "parallelise harder"
| content, one 64 MiB object, one core | deflate L6 | SHA-1 | hash ÷ deflate |
|---|---|---|---|
| compressible | 398.4 MB/s | 2021.1 MB/s | 5.1× |
| **incompressible** | **48.2 MB/s** | **2027.3 MB/s** | **42.1×** |
*Carried from 2026-08-03.* Level-6 DEFLATE over already-compressed content achieves a
ratio of **1.000 — literally nothing — while running 42× slower than a hash over the same
bytes**. Splitting makes it 11× faster at achieving nothing. Re-measured on 2026-08-04 the
`deflate.lfs_incompressible` arm read **357.5 MB/s at ratio 1.000**, which is the same
result: the number to act on is the ratio, and the action is to route that content to
`precompressed`'s store path *before* a compressor sees it. **Making waste cheaper is not
making progress**, and someone will otherwise bench this crate on LFS-shaped data, find it
does nothing, and conclude the design failed when it is working exactly as intended.
### Reproduce
```sh
# the publishable arms (nornir bencher — never criterion). These rows warehouse in
# the SAME BenchRun as the decompressor bake-off above.
NORNIR_BENCH_DEFLATE_ONLY=1 cargo run --release --example nornir-bench
# the tuning loop, including the precompressed arm (which does NOT warehouse)
CARGO_TARGET_DIR=/home/rickard/scratch/cargo/zoomies-deflate \
cargo run --offline --release -p ldeflate --example measure -- precompressed
```
`NORNIR_BENCH_DEFLATE_ONLY=1` skips the heavy bake-off and emits **no `BenchRun`
contract** — the numbers go to stderr, so a compressor fix can be measured in a minute
without a preview run reaching a published table. The scaling arm carries its own gate:
`check_deflate_scaling_real` fails the run below 4× speedup or 4 cores held, and
`deflate_scaling_gate_trips_on_serial` feeds it a serial implementation's numbers
(speedup 1.0, 1 core) and asserts it trips — **verified red on 2026-08-04 by dropping both
floors to 0.0, which made the test fail with its own message**, then reverted and
re-confirmed green. Whole suite: **86 tests, green serially under `(ulimit -v 25165824)`,
2026-08-04.**
## osm-katana — read the regime before the numbers
`osm-katana` (OSM-XML/PBF → GeoParquet, all-core on the gatling pipeline) has **two very
different benchmark regimes**, and only one of them is a throughput measurement.
**Smoke run (small synthetic corpus) — NOT a scaling result.** The default bake-off runs
a tiny synthetic OSM-XML corpus that exists to exercise the emit path *on any box*. It
finishes in ~0.2 s, so the numbers are dominated by process start-up, thread-pool spawn
and file open — fixed overhead that **min-of-3 cannot remove**. Every tool shows ~1 core
busy because the parallel path never engages on ~24 MB; a lean serial C tool naturally
"wins" a micro-input where thread-spawn cost exceeds the work. **Do not read 12-core
scaling — or any throughput ranking — from this table.**
| workload (smoke, ~24 MB, ~0.2 s) | convert_mbs | cores_busy | convert_s | peak_mem_mb |
|---|---|---|---|---|
| osm-katana | 115.26 | 1.34 | 0.21 | 308.40 |
| osmium | 121.23 | 1.31 | 0.20 | 36.80 |
| osmconvert | 131.04 | 0.87 | 0.18 | 267.20 |
**Representative run (real all-core face) — the region-clip planet cut.** `osm-katana`'s
actual scaling shows only at planet scale, where there is enough work to fill the box.
The committed warehouse `BenchRun` (Sweden cut out of the full planet with our own tool,
resolved geometry: 19.35 M ways + 14.5 M relations) is:
| workload (Sweden-clip planet, steady-state) | convert_mbs | cores_busy | convert_s | peak_mem_mb |
|---|---|---|---|---|
| osm-katana | 48.45 | **10.94 / 12** | ~3450 (57.5 min) | 15000 |
That is the honest ~11/12-core figure (`perf(osm-katana): parallelize … ~3→12 cores`,
`f6dc01b`; run committed in `6518559`). There is **no head-to-head at this scale**:
`osmium` / `osmconvert` have no planet→clipped-GeoParquet equivalent, so the region-clip
row is `osm-katana`-only and we do not manufacture rival numbers for it. The multi-hour
planet run is **not** reproduced on this box (it is a heavy run); these are the committed
warehouse numbers. A lighter but still representative all-core figure would need a
region-sized corpus that runs several seconds all-core — a bench-machine job, not a
laptop one.
## osm-katana — per-phase core occupancy
Throughput alone cannot say *why* the box sits idle, so the bench now reports the
converter's own per-phase occupancy (`--log` JSONL → one bench row per phase). The
numbers below are **PREVIEW** — a 319 MB PBF, not a publishable heavy run — and the
harness enforces that: a light run never emits a `BenchRun`, so these can never reach
the warehouse or a README table. They are here because the *shape* is the finding.
**oden · AMD Ryzen Threadripper PRO 3975WX · 32 physical cores · SMT off · 499 GB RAM ·
input 319 MB PBF · min-of-3 · 2026-07-22**
| phase | wall_s | cores held | busy % of box | collector % |
|---|---|---|---|---|
| pass1 | 1.145 | 16.35 | 51.1 | 55.3 |
| sort_and_stree | 0.450 | 4.74 | 14.8 | 0.0 |
| pass2 | 1.346 | 20.70 | 64.7 | 13.5 |
| **convert** (whole run) | **2.963** | **16.45** | **51.4** | — |
How to read it:
* `convert` sits *below* pass1 and pass2 ⇒ core-time is lost **between** the bursts,
not inside them. That is the `pass1 → sort_and_stree → pass2` barrier.
* `sort_and_stree` at 4.74 / 32 is the barrier itself: pass 1 must be fully drained
before the tail coord chunk can be sorted, and pass 2 cannot start until the STree
exists.
* pass1's `collector_pct` 55.3 with `busy_pct` 51.1 ⇒ **collector-bound**. The single
in-order sink copies every packed coord buffer into the sorter *and* writes the whole
`nodes.parquet` — 565 MB + 533 MB on one thread for this input — while 31 workers
wait behind it.
### What that measurement already bought
The first thing the per-phase view exposed was not in the pipeline at all: every phase
was reporting a wall time that was an exact multiple of 100 ms. `Phase::done()` joins
the phase's sampler thread *before* it reads the clock, and the sampler only noticed the
stop flag between 100 ms sleeps — so each of the four phase boundaries charged up to a
full poll interval of dead time with the entire worker pool already idle. Replacing the
polled flag with a condvar:
| | before | after |
|---|---|---|
| convert wall (319 MB PBF), bench harness min-of-3 | 3.654 s | **2.963 s** (−18.9 %) |
| average cores held | 13.60 | **16.45** |
| box occupancy | 42.5 % | **51.4 %** |
| convert wall, interleaved A/B min-of-11 (see caveat) | 3.218 s | **2.995 s** (−6.9 %) |
| liechtenstein (3.3 MB) convert, 5/5 runs | 0.500 s | **0.192–0.211 s** (−58 %) |
| liechtenstein occupancy | 4.4 % | **11.1 %** |
**Contention caveat.** oden was carrying other agents' workspace builds for most of
this session (load average swinging 18 → 159), so the 319 MB figures are quoted two
ways: the bench harness's min-of-3 from a quiet window, and a stricter
old-binary/new-binary **interleaved** A/B (11 pairs, round-robin so both binaries see
the same turbo/contention windows) whose best pair is the −6.9 % above. The new binary
won 9 of 11 pairs. The 3.3 MB figure needs no caveat — it repeated to the millisecond
across 5/5 runs, because on a small input the four 100 ms boundary stalls *were* the
runtime.
Output unchanged across the change — `osm-katana verify --digest` gave the same
multiset digest for all three tables (`nodes` `19b60a8f97bfc25e` / 35316700 rows,
`ways` `04bb35929ff74b2d` / 4170200, `relations` `00eff866fcbfa1f7` / 93800).
### A negative result worth keeping
`CoordFileWriter::write_raw` — the pass-1 collector's coord path — walks
`chunks_exact(16)` rebuilding each `[u8; 16]` element by element, which reads like an
obvious 565 MB-per-run win over a bulk `memcpy`. It is not. Isolated at the real shape
(35316700 records over 370 segments, 3 passes):
| | time | throughput |
|---|---|---|
| element-wise `extend` | 147.7 ms | 3.83 GB/s |
| `copy_nonoverlapping` | 147.0 ms | 3.84 GB/s |
0.5 % — the `TrustedLen` specialisation already compiles the extend to the same thing,
and both sit at memory bandwidth. The `unsafe` version was written, measured and
**reverted**: not faster, so it does not ship. The pass-1 collector's real cost is the
533 MB `nodes.parquet` stitch, not the coord copy.
### The disguised-pool sweep (2026-07-22) — what each conversion actually cost
ROOT LAW #0 names `std::thread::scope` + `s.spawn` as **the disguise**: a hand-rolled
scoped work pool is a rayon replacement and is forbidden just as hard, because it dodges
a `thread::spawn` grep while being the identical violation. Twelve such pools were left
in the tree; this pass converted them all except the root crate's `psort`/`xml`/`vtd`.
Every conversion is quoted honestly — a law conversion lands even at parity, but parity
is what it is called.
All numbers on **oden** (AMD Ryzen Threadripper PRO 3975WX, 32 physical cores, SMT off,
499 GB), interleaved old/new A/B so both binaries see the same contention windows.
oden was shared with other agents all session; single before/after numbers are not
quoted anywhere below.
| site | shape it became | wall | other |
|---|---|---|---|
| `geo2arrow.rs` | `ordered::run_ordered_sink` | **3.88 → 3.09 s** (−20 %) | **peak RSS 4.49 → 1.55 GB** (−65 %) |
| `node_store.rs` fast path (×2) | `gatling_run` + `gatling_for_each`, positional writes | 0.248 → 0.240 s quiet; **0.467 → 0.311 s contended** | 3 syscalls per range, not per record |
| `verify.rs` (×2) | `gatling_for_each_balanced`, LPT by row-group size | 0.16 → 0.15 s | 13.5 → 14.5 of 32 cores |
| `side_outputs/search.rs` | `gatling_for_each_balanced`, LPT by row-group size | 9.29 → 9.19 s | output became **reproducible** (see below) |
| `node_store.rs` `merge_sorted_chunks` | `gatling_for_each_balanced`, LPT by partition count | not on any live path | positional write replaces per-record `write_all` |
| `writer.rs` (×4, test-only) | `gatling_map_balanced` | n/a | the encode-saturation guard now measures the shipped engine (15.1×) |
| `stree.rs` / `stree32.rs` | `gatling_scanlines`, band rows | 0.037 → 0.037 s | see the band negative result below |
**`geo2arrow` was also a memory bug.** The old code claimed its bounded channel kept
memory in check. It did not: row groups were carved into *contiguous per-worker
partitions* while the writer emits in strict global order, so nothing worker 1..N−1
produced could be written until worker 0 finished its whole partition — and the writer
kept draining the channel into an unbounded `pending` HashMap to unblock the decoders.
`run_ordered_sink`'s permit pool cannot do that: a permit is released only after an
in-order emit. 104 M rows / 214 row groups → 6.03 GB Arrow IPC, `md5` identical from
both binaries.
**The search index was silently non-reproducible.** `gatling_features` concatenated each
worker thread's private `Vec` in *handle* order, so the dense feature ids — and every
posting list, and the whole `.idx` — depended on which thread won which row group. Three
runs of the old binary on liechtenstein gave three different files; three runs of the new
one are byte-identical. Same 61 082 features, same 18 558 tokens, every posting list
equal under the id relabelling. (A second hole in the same file: `postings` is a HashMap
and was serialised in HashMap iteration order — the same class of bug as the
`serialize_tags` HashMap. It is key-sorted now, via `gatling_sort_unstable_by`.)
**A second negative result worth keeping — the stree band.** The obvious gatling unit for
the per-layer block fill is ONE BLOCK, and that is what was written first. It is **2×
slower**: a block is 64 bytes of fill and the widest layer of a 100 M-key tree is 12.5 M
blocks, so the shared `fetch_add` and its cache line become the entire cost.
| unit | build, 100 M keys |
|---|---|
| old scope pool, one static chunk per thread | 0.037 s |
| `gatling_scanlines`, one block per unit | 0.081 s (+119 %) — **rejected** |
| `gatling_scanlines`, band of `n/(4·threads)` blocks | 0.037 s — **shipped** |
The lesson generalises: gatling's self-dispatch is not free, and the right unit is the
one where the work per `fetch_add` dwarfs the `fetch_add`. Finer is better only until it
is not — measure it.
**What did NOT move.** `convert` itself is untouched by this pass (its phases were
already on gatling), and the interleaved liechtenstein A/B says so: pass1 0.080 → 0.078 s,
pass2 0.107 → 0.104 s, convert 0.197 → 0.197 s. The one visible change is
`sort_and_stree` occupancy, 4.43 → 5.48 of 32 cores at the same 0.009 s wall — the stree
band conversion showing up. Output gate: `verify --digest` multiset digests unchanged
(`nodes 16a8386cf4b7ad76` / `ways 02ed64602fadfe31` / `relations 119bffec8cc479f0`) and
`node_coords.arrow` compares equal under `cmp`.
**Left standing, deliberately.** `xml_to_pbf.rs`'s `blob_writer_loop` is ONE named,
long-lived thread doing blocking file I/O off the gatling collector, fed by a bounded
mpsc channel. That is IO-offload, not CPU fan-out — the law's own carve-out — and it is
commented as such at the call site.
### Reproduce (PREVIEW lens)
```sh
NORNIR_ZOOMIES_CORPUS_MB=0 NORNIR_ZOOMIES_WORK=/tmp/zb \
NORNIR_OSM_KATANA_INPUT=~/work/liechtenstein-latest.osm.pbf \
cargo run --release --example nornir-bench
```
`NORNIR_ZOOMIES_CORPUS_MB=0` means "no decompressor corpus" — the decoder roster is
empty and only the osm-katana arm runs, so a fix can be measured in seconds instead of
minutes. Same code path, same rows, same metric keys as the heavy run. See
[`osm-katana-guide.md`](osm-katana-guide.md#bench).
## static search tree — `stree32` vs `binary_search`
Measured with `-C target-cpu=native`, the in-test micro-bench, 1M hit queries
(`src/stree32.rs`, `stree32_bench` / `stree32_bench_ram`). The win tracks the
cache→RAM cliff and rises toward Ragnar's headline ~40× at 1 GB:
| Store | binary_search | STree32 best | speedup | vs STree64 (i64) |
|---|---|---|---|---|
| 16 MB u32 (4M keys, cache→RAM) | 317.0 ns/q | 33.6 ns/q | **9.4×** | 0.97× (parity) |
| 512 MB u32 (134M keys, RAM-bound) | 950.0 ns/q | 45.5 ns/q | **20.9×** | **1.43×** |
The u32 density win flips on once RAM-bound: in cache both widths are ~equal, but at
512 MB the i64 tree's extra level costs a real cache miss, so `STree32` is **1.43×
faster than `STree64`** over the *same* keys — the whole reason to restore the 32-bit
width. Reproduce:
```sh
cargo test -p znippy-zoomies --release stree32_bench -- --nocapture # in-cache sweep
cargo test -p znippy-zoomies --release stree32_bench_ram -- --ignored --nocapture # 512 MB RAM-bound
```
## engine + scanners — the criterion benches
Three criterion benches ship in [`benches/`](../benches/) (`harness = false`,
`Cargo.toml:58`): `gatling_roundtrip` (the no-barrier streaming engine, split → N
decode → in-order collect → sink), `xml_scan` (VTD-style OSM-XML scan), and `json_scan`
(zero-copy NDJSON scan). All three run all-cores by design; the operational contract is
flat 100 % core utilisation (the "all cores busy" tell in [`guide.md`](guide.md)).
```sh
cargo bench -p znippy-zoomies --bench gatling_roundtrip
cargo bench -p znippy-zoomies --bench xml_scan
cargo bench -p znippy-zoomies --bench json_scan
```
The `json_scan` bench also pins the no-rayon port: after moving the NDJSON line-scan
off `rayon::into_par_iter` onto `gatling_forkjoin::gatling_for_each`, the ported scan is
if anything *faster* than the rayon version it replaced (no global-pool contention).