# Benchmarking: why `nornir bench`, not `cargo bench`
<!-- nornir:class: reasoning -->
znippy has exactly **one** benchmark suite, and it does not run through cargo's
bench harness. This chapter records why, and how the single source of truth is
wired, so nobody re-adds a parallel `criterion` or `#[bench]` path.
## Has nornir forked its own way? Yes — deliberately.
nornir does **not** use cargo's `#[bench]` / `cargo bench` / criterion harness at
all. From `nornir/src/bench/api.rs`, the contract is tiny:
- Each bench is a small struct implementing nornir's `Bencher` trait, registered
with `register_bench!` / `register_bench_ordered!`. The `inventory` crate
collects them **at link time** — there is no hand-maintained registry.
- `examples/nornir-bench.rs::main()` is literally `nornir::bench::api::run_main_json()`.
It runs every registered bencher and prints **one `BenchRun` JSON line** to stdout.
- The release pipeline subprocesses `cargo run -p xtask --release --example
nornir-bench --quiet`, parses that stdout line, and writes it into the Iceberg
`bench_runs` table. The README tables are then rendered from the warehouse, not
by hand.
> **⚠ Correction, 2026-08-03 — there is no `git_sha` column.** This chapter used
> to say the row is "tagged with release_id + per-repo git SHA + dep-graph
> snapshot", and so does `nornir/.nornir/bencher.md`. READ:
> `nornir/src/warehouse/iceberg_schema.rs:127-140` gives `bench_runs` exactly
> seven fields — `run_id, repo, ts_micros, date, version, machine, cores` — and
> `build_bench_runs_batch` (`nornir/src/warehouse/iceberg.rs:2440-2456`) writes
> exactly those seven. **No `git_sha`, no `release_id`.** A bench number is keyed
> by `(repo, machine, version, timestamp)`; SHA attribution is indirect, through
> `release_lineage` / `regression_trace`. Do not design a suite around per-SHA
> keying it will not get.
### Why a fork, not `cargo bench`/criterion
1. **dev-deps.** An *example* may use `[dev-dependencies]` (osm-katana,
znippy-compress, the plugins); a `bin` may not — and nothing about nornir is
allowed to leak into a published crate. That is the whole reason the entry
point is an example, not a `bin` or a `[[bench]]`.
2. **Different goal than criterion.** criterion does statistical micro-sampling
with its own human-readable output. nornir wants *single-shot, real-workload*
numbers (multi-GB archives, the planet OSM pipeline), historized by git SHA,
gated for >10% regression, and rendered into the README. criterion's model and
cargo's bench output are not machine-historizable the way nornir needs.
## The xtask pattern (this repo)
znippy is a multi-crate workspace, so it uses nornir's recommended shape:
```text
xtask/ publish = false → heavy deps never reach crates.io
├── Cargo.toml nornir + osm-katana + znippy-* under [dev-dependencies]
├── src/lib.rs doc-only; exists so the package has a target the example builds against
└── examples/nornir-bench.rs the ONE bench source: registers every Bencher, calls run_main_json()
```
Run a release-gate-equivalent locally:
```bash
cargo run -p xtask --release --example nornir-bench # full suite (JSON to stdout)
NORNIR_BENCH_SKIP_PLANET=1 NORNIR_BENCH_SKIP_HEAVY=1 \
cargo run -p xtask --release --example nornir-bench # skip the multi-GB workloads
```
The old `before_cratering.sh` and `xtask/src/main.rs` runners (which duplicated
each other) and the throughput tests in `tests/tests/perf_bench.rs` /
`compress_dir_bench.rs` were retired — their unique workloads were ported into
`nornir-bench.rs` so nothing was lost:
- `archive_text_500mb` / `archive_binary_500mb` / `archive_random_500mb` /
`archive_mixed_repo` / `archive_huge_2gb` — synthetic codec throughput via
`compress_stream` (heavy ones gated behind `NORNIR_BENCH_SKIP_HEAVY`).
- `archive_real_wheels` — the Python-wheel corpus alongside jars/crates.
- `maven_extract` — native (ljar) vs fallback metadata extraction, µs/call.
- `read_only_uring_10k` — the raw io_uring open+read+close NVMe ceiling that every
`compress_dir` number is measured against (no codec).
- `stree_8mb` / `stree_128mb` / `stree_1gb` — static-search-tree id lookup
(single `find_exact` + batched `batch_stream`) vs a `binary_search` baseline,
swept across the cache→RAM cliff. Reproduces Ragnar's "40× faster than binary
search". **Benched as `Bencher` impls, NOT criterion** — even though upstream
uses a criterion-style harness, nornir bencher stays the single source of
truth (no `[[bench]]`/`criterion` connection needed). `stree_1gb` is gated
behind `NORNIR_BENCH_SKIP_HEAVY`. Full write-up, numbers, and the two perf
fixes: [stree.md](stree.md).
Maven *correctness* tests stayed in `tests/tests/maven_bench.rs`; only the timing
functions moved.
## The `git` package format (`znippy.git_*`) — where its workloads live
Six arms measure the `git` package format: `git_oid_lookup_{100k,1m,8m}`,
`git_graph_{linear,merges}_200k`, `git_reach_10k`, `git_sections_300k` and
`git_archive_60k`. The registrations are in `xtask/examples/nornir-bench.rs` like
every other arm, but the **workloads are not** — they live in
`plugins/native/znippy-plugin-git/src/bench_kernels.rs` behind the crate's `bench-kernels`
feature, and the `Bencher` impls are thin callers.
That is deliberate, and it is the direct consequence of the worktree trap above:
**no git worktree of znippy can build `xtask` at all**, so a workload written in
`nornir-bench.rs` could only ever be *executed* from `/home/rickard/git/znippy`.
Written in the crate it compiles anywhere, `cargo test -p znippy-plugin-git
--features bench-kernels` guards it, and there is still exactly one
implementation (LAW 5). The feature is **off by default** — it is what drags in
the `fst` rival and `znippy-compress`, and none of that belongs in a shipped
archiver.
`git_oid_lookup` is the arm that matters: `__gunnar_oid__` chose an `stree`
keyspace over the stock `fst` trie and `lookup_batch_pipeline::<8>` was the whole
justification, so the arm times binary search, `stree` serial, `stree` batch and
an `fst::Map` built over the same oids — after cross-checking that all four
return the **same row for every query**. The verdict is reported as a neutral
`*_x` ratio rather than asserted, so a loss is visible instead of failing the
run. The archive arm writes its fixture to `/home/rickard/scratch/…` (md1), never
`/tmp`, which on oden is RAM and would have measured the page cache.
## The `cargo bench` muscle-memory — wired
`cargo bench -p xtask` runs the exact same suite, via a thin `[[bench]]` shim. A
`[[bench]]` is a separate binary, so it must link the same `register_bench!`
registrations — `xtask/benches/nornir.rs` gets that by `#[path]`-including the
example source (benches may use dev-deps, like examples), so `inventory` collects
every bencher into the bench binary too:
```rust
// xtask/benches/nornir.rs
#[allow(dead_code)]
#[path = "../examples/nornir-bench.rs"]
mod suite;
fn main() -> anyhow::Result<()> {
nornir::bench::api::run_main_json()
}
```
```toml
# xtask/Cargo.toml
[[bench]]
name = "nornir"
harness = false
```
Caveats: it still emits **nornir's JSON**, not pretty criterion output, and nornir
itself keeps calling `cargo run --example` (it never calls `cargo bench`). So the
shim is only the familiar command with nornir semantics — not a second bench
system. See nornir's `.nornir/bencher.md` → *Surfaces* for the canonical writeup.