<!-- nornir:class: history -->
# Bencher
Internal reference for nornir's **Bencher framework** — the contract a child
workspace (holger, znippy, …) implements so a `nornir release` can drive its
benchmarks, historize them in the warehouse, gate on regressions, and render
them into the README. This file lives in `.nornir/` and is a durable design
note (not a rendered artifact); it is also a chapter of `nornir docs book`.
Source: **`edda/crates/nornir-bench`** — `api.rs` (the trait + the INNER runner
that runs inside the bench example), `lib.rs` (the run/result envelope),
`history.rs` (append-only JSONL), `assets.rs`, and `runner.rs` (the OUTER runner:
find the example, build it, run it, parse the `BenchRun`). The bencher was
extracted out of `src/bench/` into its own `nornir-bench` crate, which then moved
to edda; `nornir::bench` is a re-export of `nornir_bench`, so the module paths
(`nornir::bench::…`) are unchanged for callers. Config:
[`src/config.rs`](../src/config.rs) `BenchSpec`. Renderer:
[`src/docs/sections.rs`](../src/docs/sections.rs) `render_bench`.
**Where the RUNNER lives (2026-08-03).** `run_bench_example` used to be ~300 lines
in [`src/release/pipeline.rs`](../src/release/pipeline.rs); that file now
re-exports `nornir_bench::runner` and adds nornir's two policy answers (which
cargo, and whether the run is heavy). The verb that drives it belongs to the
**forge**: `dwarves bench run <product>`. `nornir bench run <repo>` still works
and prints where it moved. Both spawn the identical runner, so rows written
through either end are the same row.
Use the forge verb from a **worktree**: it takes a name OR a path, and keys the
`bench_runs` row on the `origin` remote's name. `nornir bench run <repo>` resolves
the checkout by NAME against the workspace root, so issued from
`gunnar-realbench` it measures `<workspace>/gunnar` instead — a different working
tree, silently, while reporting success.
## What it is
A tiny, project-agnostic harness. Each child repo writes its own benches as
small structs and `cargo run`s them in isolation; nornir, as orchestrator,
subprocess-spawns that runner, parses one JSON line off stdout, and writes the
result into Urðr (Iceberg) tagged with the release id + per-repo git SHA + dep
graph snapshot. nornir never links the child's bench code — the only coupling
is **stdout JSON**, the same contract shape as `cargo metadata`.
## Bench vs test — the classification (LAW)
`test` and `bench` are **separate subsystems** (separate runners, registries,
warehouse tables, and viz tabs — see [`bench-live-pane.md`](bench-live-pane.md)).
A plain `nornir test` / "Run full matrix" runs **no benches**; the only overlap
is the `bench-smoke` *aspect* (`cargo bench --no-run` — compile-only, lands in
`test_results`, never executes a benchmark).
But some **tests in practice do bench-grade work** (compress/decompress large
corpora, build a graph over N generated crates, scan/write large tables). Those
are what spin the fans. Classify every heavy/`#[ignore]`'d test into one of three
buckets:
- **A — bench-type, a bench already exists** → only **downscale the test** (no
clone). Example: znippy `ljar`/`lzip` `maven_artifacts.rs` timing tests already
have `jar_throughput`/`zip_throughput` criterion benches; the test just needs to
fetch 1 JAR instead of 26.
- **B — bench-type, no bench yet** → **clone it into a bench** (capture the
throughput/timing metric so it historizes in `bench_history` and gates on
regression) **and** keep a **downscaled** correctness test. Example: nornir
`iceberg.rs::skade_ab_write_read` (env knobs `AB_COMMITS`/`AB_ROWS`),
`scenarios.rs::l3/l4_layered_dag_*` (a `dep_graph_build` bench), facett
`tests/perf.rs::*` (render/ingest throughput, no harness yet).
- **C — NOT bench-type** → keep as a heavy test, downscale if trivial, **no bench**.
The heaviness is dominated by network (model/JAR download), daemon boot, or a
one-shot correctness assertion with no scale (LLM/ONNX model loads, daemon
smoke, git-clone security scans).
**Rule of thumb:** if it could emit a meaningful **rows/s · MB/s · ms-for-N**
number, it is bench-type (A or B). If the slow part is a download or a boot, it
is C. The full per-repo verdict for the whole nordisk constellation lives in
[`bench-test-classification-audit.md`](bench-test-classification-audit.md) — the
canonical work-list for cloning/downscaling.
A new beginner-facing version of this is
[`bench-vs-test-for-idiots.md`](bench-vs-test-for-idiots.md).
## Heavy tests are OPT-IN (impl change, 2026-06-17)
**Heavy = anything outside the fast, hermetic default — large SCALE *or* external
INFRA.** Concretely the `#[ignore]`'d arms: large corpora / scale matrices
(scale), and containers (podman/Docker), VM boots (QEMU), model downloads,
network, and live-server RPCs (infra). Empirically ~90% of today's `#[ignore]`
tests are infra, not scale — so "container counts as heavy" is already the
reality (`heavy_enabled()` gates *all* `#[ignore]`). They now run **only on
explicit request — default OFF everywhere** (release, `nornir test`, and the viz
"Run full matrix"). A **fixture-gate** (skip gracefully when the
container/fixture/server is absent) is the *secondary* safety even under
`--heavy`. They burn minutes and spin the
fans, and a flaky heavy/network arm turning a release red is the opposite of a
gate; the always-run synthetic tier IS the gate.
- Opt IN with the **global `--heavy`** flag (sets `NORNIR_RUN_HEAVY=1`), or the
back-compat `NORNIR_HEAVY_TESTS=1`.
- `--skip-heavy` / `NORNIR_SKIP_HEAVY` is now a **no-op** (off is already the
default) — kept so existing scripts/CI don't break.
- Single source of truth:
[`nornir_testmatrix::runner::heavy_enabled()`](../crates/nornir-testmatrix/src/runner.rs).
Both the release runner (`run_cargo_test`) and the matrix runner (`run_matrix`,
which now adds `--include-ignored` / nextest `--run-ignored all` only when heavy
is on) read it, so CLI, release, and viz behave identically.
**Heavy is a TEST-only concept — benches are NOT heavy-gated.** Tests run
*implicitly* (every release gate, every viz "Run full matrix"), so they need a
default-off switch. Benches run only when you *explicitly* `nornir bench run` —
you already chose the work, and a bench's whole point is to measure the heavy
workload. Gating benches behind `--heavy` would also silently gut bench-based
regression gating (`no_regression` / `required_results`) now that `release`
defaults to no-heavy. A bench that needs network or a giant fixture gates the
**existing** way: `nornir.toml`'s `network_required` + a fixture-present check
that degrades to a skip — never a heavy flag.
### Heavy-skip + chains — the idiot guard
Skipping a heavy producer can orphan a dependent (heavy test #1 unpacks OSM,
light test #2 reads it → #2 breaks when #1 is skipped). **LAW: a test must build
its own inputs lazily (on-demand fixture), never depend on another *test* having
run first.** A genuinely shared expensive step is a fixture helper (any caller
triggers it) or an **ordered bench stage** (`register_bench_ordered!` +
prerequisite check + `NORNIR_BENCH_STOP_ON_ERROR`), never a sibling `#[test]`.
The full chain model — a `test_chains` warehouse table, the `no_light_on_heavy`
gate, skip-propagation, and the chain-graph viz — is designed in
[`test-bench-chains.md`](test-bench-chains.md).
## The contract (locked — see plan.md §4)
A child opts in with **three** things and nothing else (nornir stays invisible
to production builds of the child):
1. `nornir = { path = "../nornir" }` under **`[dev-dependencies]` only**.
2. One `examples/nornir-bench.rs` — at any of **three** accepted locations:
- `<repo>/examples/nornir-bench.rs` (single published package), or
- `<repo>/xtask/examples/nornir-bench.rs` (workspace with an `xtask` member —
`cargo run -p xtask …`), or
- `<repo>/bench/examples/nornir-bench.rs` (**detached `bench/` crate** with its
own `[workspace]`; the runner runs cargo *inside* `bench/`). Use this
when the bench harness + its heavy dev-deps (e.g. `nornir`, scenario libs)
must stay out of the published crate without adding an xtask workspace member.
(Example: nornir-catalog.)
The file is literally:
```rust
fn main() -> anyhow::Result<()> { nornir::bench::api::run_main_json() }
```
It is an *example*, not a bin, because examples may use dev-deps and bins
cannot.
3. One small struct per bench implementing [`Bencher`] + `register_bench!`:
```rust
struct JarDecompress;
impl nornir::bench::api::Bencher for JarDecompress {
fn id(&self) -> &'static str { "ljar.jar_decompress" } // <repo>.<scenario>
fn run(&self) -> anyhow::Result<nornir::bench::BenchResult> { /* measure */ }
}
nornir::register_bench!(JarDecompress);
```
`inventory` collects every registration at link time — no hand-maintained
registry list. `register_bench_ordered!(N, …)` pins execution order for
pipelines where stage N feeds stage N+1 (e.g. osm-katana `bz2 → pbf → geo`).
`cargo run --release --example nornir-bench` then prints one
[`BenchRun`](#data-model) JSON line. That is the whole interface.
> **⚠ For agents authoring benchers — read this (2026-06-07).** Your contract is
> **unchanged and stays JSON**: write `impl Bencher` + `register_bench!` + the
> one-line `run_main_json()` example; it emits **one BenchRun JSON line**. That is
> all you do — nothing about authoring a bencher changed.
>
> What *did* land is on **nornir's** side, not yours: `nornir bench run` now has a
> **client mode**. With `NORNIR_SERVER` set, after running your example locally it
> ships the result to a running `nornir-server` over the `Bench.Submit` **gRPC RPC**
> (protobuf) instead of appending in-process; without it, it appends locally (the
> default). Either way **your example still just prints the JSON line** — the child
> process never speaks gRPC; nornir does the forwarding. So: **keep using
> `run_main_json()`**, do not make your example a gRPC client.
>
> The RPC path is **lossless**: the proto `BenchRun` carries results **and** test
> outcomes, and the server persists into the iceberg `bench_runs` table (the same
> source of truth the local path writes), so a remotely-submitted bench is fully
> queryable (`bench_history`, gates, `query_bench_runs`). The JSONL is just the
> on-disk receipt.
## Data model
Free-form metrics are the key design choice — one envelope serves every repo
(`crates/nornir-bench/src/lib.rs`):
| type | shape | notes |
|------|-------|-------|
| `BenchResult` | `{ name: String, metrics: Map<String, Value> }` | `metrics` is **flattened** JSON — any keys the bench wants |
| `BenchRun` | `{ date, timestamp, version, machine, cores, results: Vec<BenchResult>, tests: Vec<TestOutcome> }` | the envelope the runner stamps automatically |
| `TestOutcome` | `{ name, passed, duration_ms?, message? }` | tests that ran beside the benches — a red one fails the release too |
Because `metrics` is open, a result can carry whatever a scenario measures —
including **several measurements side by side** (this is what makes comparative
"VS" benches representable; see below). Metric-naming convention (plan.md §4):
suffix scalars `_mbs` / `_ms` / `_pct` so renderers + the regression gate can
reason about them.
### `required_results` — the corpus contract
`[repo.<n>.bench] required_results = [...]` in `nornir.toml` lists the
`BenchResult.name`s a release **must** produce. `BenchSpec::validate`
(`src/config.rs`) fails the release if any are missing, so a corpus can't
silently shrink. It validates **names**, not metric keys — a VS bench is *one*
named result carrying paired metrics. Today:
```toml
[repo.holger.bench]
required_results = ["rust_crate_st","rust_crate_mt32","java_jar_st","java_jar_mt32","pip_wheel_st","pip_wheel_mt32"]
[repo.znippy.bench]
required_results = ["text_500mb","binary_pattern_500mb","random_500mb", …]
```
## Execution & ordering (`run_main_json`)
Collects every `register_bench!`'d entry, sorts by `(order, id)`, runs them
**sequentially** (bench machines should be quiescent — no parallel scheduling),
wraps each in a `TestOutcome` (a panicking/erroring bench degrades to a recorded
red test rather than killing the run — unless `NORNIR_BENCH_STOP_ON_ERROR=1`),
stamps the envelope (`date`, `timestamp`, `version` from `CARGO_PKG_VERSION`,
`machine` from `NORNIR_MACHINE`, `cores` from available parallelism), and prints
JSON. Convention knobs seen in the wild: `NORNIR_BENCH_QUICK=1` for fast dev
iteration vs full-fidelity CI corpora.
## Progress logging — name every unit as it runs (0.4.10 "Gjallarhorn")
A full bench suite can run for minutes; before 0.4.10 the terminal sat silent
the whole time. The bencher now *announces* each unit (`crates/nornir-bench/src/progress.rs`,
driven from `crates/nornir-bench/src/api.rs`) — Heimdall's horn for the bench loop. Around
every bencher it emits three line shapes (the `(mt|st)` mode label distinguishes
the multi-core pass from the single-core `_st`, affinity-pinned pass):
```
bench: znippy.compress (mt) starting…
bench: znippy.compress (mt) → 1234 ops/s (0.42s) [cores 6.80/16 · host-busy 7/16]
bench: holger.ops (st) ✗ failed: <error> (0.05s)
```
The `→` summary prefers a throughput metric (`*ops_sec` → `N ops/s`); a unit
with no headline metric still logs its id, mode, and elapsed. The trailing
`[cores C/N · host-busy B/N]` is **two different numbers and only the first is
about the bench**: `cores` is this process's own peak occupancy (task-clock ÷
wall from `/proc/self/stat`), `host-busy` is how many of the box's cores were
>50% busy *for any reason*. On a loaded machine `host-busy` reads N/N whatever
the bench does — see *Live CPU/mem telemetry* below. Omitted on hosts without
`/proc` (no sampler data). A failing bench logs the
`✗` line and (per the contract above) degrades to a recorded red rather than
killing the run.
**Live stderr streaming.** `dwarves bench run` / `nornir bench run` / `release
run` spawn the child bench binary with its **stderr inherited** (not captured):
`nornir_bench::runner::run_bench_example`
pipes only stdout — the channel carrying the final `BenchRun` JSON we must parse —
and lets stderr flow straight to the operator's terminal. (Previously it used
`Command::output()`, which buffered *both* streams until the child exited, so a
multi-minute bench looked hung and no `bench:` progress appeared until the end.)
With stderr inherited, each `bench: …` line — including its `cores-busy/N` — is
visible live, so a long-running bench is always identifiable.
**Two sinks, both best-effort and decoupled from the viz crate:**
1. A stable, greppable **stderr** line, always on, prefixed `bench:`.
2. When `$NORNIR_VIZ_ACTIONLOG` is set, an **appended** line — in the exact
`HH:MM:SS.mmm <seq> [BENCH] …` format the viz action-log writer uses — so a
running benchmark streams **live into the viz's 🐞 action-trail overlay**
(amber `[BENCH]` lines). The bencher only ever *appends*; the viz owns
truncation at launch (the file is the single shared source of truth). A missing
env var or unwritable file is silently ignored — stderr already carried it, and
`crates/nornir-bench` never takes a dependency on the feature-gated `viz` crate.
Guide — watch a bench run live in the viz:
```sh
# terminal A — launch the viz on the shared action-log file
NORNIR_VIZ_ACTIONLOG=/tmp/nornir_viz_actions.log nornir viz
# then toggle the 🐞 log panel in the toolbar
# terminal B — point the bench at the SAME file and run it
NORNIR_VIZ_ACTIONLOG=/tmp/nornir_viz_actions.log nornir bench run znippy
# each bench: … line appears amber in the viz 🐞 panel as it runs
```
The overlay tails the file's tail (`action_log::tail_lines`), so it surfaces the
`[BENCH]` lines written by the *separate* bench process, not just the viz's own
in-memory ring — see [viz-introspection.md](viz-introspection.md).
## From run → warehouse → gate
`run_pipeline` (and `nornir bench run <repo>`) does:
1. `run_bench_example(root)` — detect `examples/nornir-bench.rs` (single-package),
`xtask/examples/nornir-bench.rs` (workspace), or `bench/examples/nornir-bench.rs`
(detached `bench/` crate — cargo runs inside `bench/`); skip cleanly if absent.
2. Parse the stdout `BenchRun`; `BenchSpec::validate` the `required_results`.
3. Append to the Iceberg `bench_runs` / `bench_results` tables (+ `test_outcomes`),
tagged `release_id` + per-repo `git_sha` + dep-graph snapshot id. The
append-only `bench_history.jsonl` stays as the on-disk receipt + migration
source.
4. The `no_regression` gate compares the new run to the newest same-machine
baseline and fails if any metric drops > `max_regression_pct`. holger also
runs `nexus_floor` (`holger_ops_sec ≥ nexus_ops_sec`).
### When the run never reached the warehouse — `docs render --from-json`
`nornir bench run --server` executes the bench **server-side**, where
`NORNIR_MACHINE` / `NORNIR_BENCH_ONLY` / `…SKIP` / `…PREVIEW` do not exist — every
row it writes is stamped with the literal machine `"server"`, which is useless for
a host-stamped table and poisonous if merged into a trend line. The working shape
for a host-stamped pass is therefore to run each repo's `nornir-bench` example
**directly**, in the directory that owns it, and keep the emitted `BenchRun`
envelope as a file.
`nornir docs render <repo> --from-json <path>` renders the documentation from that
file with the **same renderers** the warehouse path uses. It replaces only the
*latest run*; `bench_history` sections still read the warehouse. Use it whenever
the numbers are JSON on disk — never transcribe a table into a README by hand,
because a hand-written table is a copy of the data that goes stale the moment the
next bench runs.
A harness may also **refuse to emit** a publishable run (znippy-zoomies stamps
`PREVIEW` into `machine` and declines to warehouse anything when its input is
under the heavy floor). That envelope is still renderable — see the preview gate
in `docs-generation.md`; it can only reach markers that ask for it with
`preview=yes`.
## Live CPU/mem telemetry (the in-bench sampler)
Every bench unit is wrapped by a **low-overhead 1 Hz sampler**
(`crates/nornir-bench/src/telemetry.rs`) that records the CPU/mem footprint of *that unit* and
persists it durably to the warehouse — no `/tmp` stopgap, no `perf`/`ptrace`, no
extra crate.
**How it samples (non-perturbing).** `Sampler::start()` spawns one background
thread (`bench-telemetry`) that sleeps most of the time and, ~once a second,
reads four small `/proc` files and folds the deltas into running min/avg/max
accumulators:
- `/proc/self/stat` — `utime`+`stime` (fields 14/15) and `cutime`+`cstime`
(16/17), in clock ticks, divided by `sysconf(_SC_CLK_TCK)`. Over the wall
window this gives **`proc_cores_avg` = task-clock ÷ wall**, the cores THIS
PROCESS occupied: `1.0` is one core, `12.0` is twelve. **This is the only
number that says anything about the bench, and the only one the
core-saturation gate reads.** It counts this process's threads plus children
**reaped** during the unit (`child_cpu_ms`); an arm whose work sits in a
still-running child under-reports.
- `/proc/stat` — the aggregate `cpu` line and per-core `cpuN` lines. Per
interval: **CPU% aggregate** = `100·(1 − idle_delta/total_delta)`; **cores-busy**
= how many of the N cores had per-core utilisation **> 50%**. ⚠️ **HOST-WIDE —
the whole box, every process on it.** Ambient context only. Measured on oden
(32 cores) 2026-08-03: the strictly single-threaded arm `gunnar.walk_1m`
reported `cores_busy_max` 32/32 and 99.85–99.95% CPU while the process held
**0.56 of one core**; `gunnar.clone_concurrent_64` genuinely occupied
10.99–12.11 of 32 (0.37) where the published metric claimed 0.94. A gate
reading these columns passes forever on a shared box.
- `/proc/self/status` — `VmHWM`, the kernel's resident-set high-water mark → **peak
RSS (MB)** (a high-water mark, so a 1 Hz read never misses a transient peak).
- `/proc/meminfo` — `(MemTotal − MemAvailable)/MemTotal` → **peak system mem%**.
All reads happen on the *sampler* thread, never the bench thread; each sample is a
few small synchronous reads plus integer subtractions (sub-millisecond, once a
second), so it does not skew the bench. On a host without `/proc` the thread reads
nothing and the summary is all-zero except `elapsed_ms` — it degrades gracefully
and never panics — and every `proc_*` field is `None`, "not measured", which is
deliberately distinct from a measured `0.0` so a gate can tell "no data" from
"used no cores". `Sampler::proc_cores_now()` / `cores_busy_now()` publish the
latest interval's numbers lock-free for the live `[cores C/N · host-busy B/N]`
progress suffix.
**The run → warehouse path.** When a unit returns, the harness (`run_pass` in
`crates/nornir-bench/src/api.rs`) stops the sampler and folds the [`Telemetry`] summary into the
`BenchResult.metrics` map under reserved `telem_*` keys (`telem_cpu_pct_avg`,
`telem_cores_busy_max`, `telem_n_cores`, `telem_mem_peak_mb`, …). Riding inside
the metrics map is what lets the numbers survive the bench-binary → `nornir`
**stdout-JSON hop** with zero change to a child repo's one-line
`examples/nornir-bench.rs`. On the warehouse side
(`append_bench_run_async`), those `telem_*` keys are **lifted out** of each result
and written as one row per `(run_id, bench)` into the dedicated **`bench_telemetry`**
table; they are *excluded* from `bench_results` so the load profile never competes
in the docs metric-comparison renderer.
**The `bench_telemetry` table** (one row per `(run_id, bench)`, partitioned by
`repo`):
| col | type | meaning |
|-----|------|---------|
| run_id | string | FK → `bench_runs.run_id` |
| repo | string | partition key |
| bench | string | the `BenchResult.name` (the unit) |
| n_cores | int | logical cores the sampler observed |
| cpu_pct_avg / cpu_pct_max | double | ⚠️ HOST-WIDE aggregate CPU% across cores |
| cores_busy_avg / cores_busy_max | double / int | ⚠️ HOST-WIDE cores >50% busy (avg / peak) — the whole box |
| mem_peak_mb | double | peak process RSS (MB) |
| mem_pct_max | double | peak system memory utilisation (%) |
| elapsed_ms | double | wall time of the unit |
| proc_cpu_ms | double? | CPU ms of this process's own threads |
| child_cpu_ms | double? | CPU ms of children **reaped** during the unit |
| proc_cores_avg | double? | **task-clock ÷ wall** — mean cores THIS process occupied |
| proc_cores_max | double? | peak of the same — **the saturation gate's only input** |
The four `proc_*` columns are OPTIONAL (`double?`): rows written before
2026-08-03 have no such measurement, and `NULL` there means *not measured* —
never `0.0`. Existing tables evolve forward additively on the next bench append.
**How to read it.** Query the typed rows via
`IcebergWarehouse::query_bench_telemetry(Some("<repo>"))` (or the generic
`scan_preview("bench_telemetry", n)` in the viz table browser). Join on `run_id`
to a `bench_runs` row for date/version/machine context. A bench showing
`proc_cores_max ≈ 1.0` under the multi-core (`mt`) pass is single-threaded in
practice — read that column, **not** `cores_busy_max`, which on a busy box says
N/N for a serial arm and a saturated one alike. A high `mem_pct_max` flags a
noisy or memory-pressured host that can explain a throughput dip without a real
regression; a large `child_cpu_ms` share means the number is "process tree, as
reaped" rather than "this process".
## Comparative / "VS" benches (the pattern)
The framework already does comparisons — **holger benches holger *vs* Nexus**.
The pattern is: a single `BenchResult` carries **paired metrics** (ours +
baseline), e.g. `holger_ops_sec` + `nexus_ops_sec` in one result; `nexus_floor`
then gates on the pair. The free-form metrics map is exactly what makes this
possible without a schema change.
The **znippy-zoomies decompressors** want the same shape — *how fast are we
against the old legacy single-threaded tool*:
| crate | legacy baseline | paired metrics in one result |
|-------|-----------------|------------------------------|
| `ljar` | `unzip -t` | `ljar_mbs` + `unzip_mbs` (+ derived `speedup_x`) |
| `lzip-parallel` | `unzip` | `lzip_mbs` + `unzip_mbs` |
| `lbzip2` | C `lbzip2` / `bzip2` | `lbzip2_rs_mbs` + `lbzip2_c_mbs` |
| `lgz` | `pigz -d` / `gzip -d` | `lgz_mbs` + `pigz_mbs` + `gzip_mbs` |
| `linflate` | `miniz_oxide` / `flate2` | `linflate_mbs` + `miniz_mbs` |
These numbers exist today as **hand-authored tables in the standalone `*-rs`
repos** (ljar-rs "1.8× faster than `unzip -t`", lbzip2-rs "Performance vs C
lbzip2", lgz-rs "vs pigz/gzip"). They are **not** wired through Bencher in
znippy-zoomies, so they don't historize and aren't rendered — which is why they
went stale / "disappeared" from the znippy docs.
## Rendering into READMEs
Two marker sections render bench results (source policy for both: Iceberg
`bench_runs` first, `bench_history.jsonl` fallback — `mcp_last_run` / the
`--features docs-export` last-run resolver):
- **`bench`** (`render_bench`) — the legacy flat `| name | metric | value |`
table (one row per metric). Kept for back-compat.
- **`benches`** (`render_benches`) — pivots the latest run into a table (row per
workload, column per metric) and **bolds the winning cell** per comparison
row. This is the "how much faster than legacy" view; it's what turns the
paired ours-vs-legacy metrics above into a readable table. Direction comes
from the metric unit (`unit_of`) with a `best=metric:low` marker override.
`winner=row` bolds down each column (cross-system) and `compare=a,b,c` folds
to one row per backend — together they render the crisp **mashup**: per-
capability nornir-vs-Nessie-vs-Polaris head-to-head tables. For the *tight*
top-of-README block, `mode=compact` (one synthetic best-of row) and `top=N`
compress it further — see [Compact "best-of" mashups](#compact-best-of-mashups--modecompact--topn).
Latency results
carry the full `min/mean/p50/p90/p99/p999/max` spread so best- and worst-case
show side by side. See `docs-generation.md` → *Built-in section renderers*.
- **`bench_chart`** (`render_bench_chart`) — a native-typst horizontal bar chart
for one metric across systems (the visual headline above the tables).
```text
<!-- nornir:gen:start:benches -->
<!-- nornir:gen:end:benches -->
```
### Compact "best-of" mashups — `mode=compact` / `top=N`
The full `benches` table is honest but long: many backends × many metrics. For
the **10-second-readable block at the top of a README** there are two compaction
directives, both stacked on top of `name=`/`exclude=`/`compare=` (which select +
order the rivals first):
| directive | effect | when |
|-----------|--------|------|
| `mode=compact` | Collapse **all** matched rows to a **single synthetic `best-of` row** that, per column, takes the **absurd-best value across every variant** (max for `High`-direction metrics like `ops_sec`/`rows_per_sec`, min for `Low` ones like `p50_us`). `Neutral` columns (counts/ratios — no "best") are dropped. The best cell in a column can come from a *different* rival than the next column's. | One line that says "across everything we measured, here's the peak we hit per metric." |
| `top=N` | Keep only the **N strongest rows**, ranked by `rank=METRIC` (default: the first throughput-like column — `ops_sec` or `*_per_sec` — else the first numeric column). Direction-aware: latency metrics rank ascending. | "the 3 fastest backends" without hand-picking names. |
| `label=TEXT` | Rename the synthetic `best-of` row (compact mode only). | Branding the champion line. |
Both honour `compare=` first, so the canonical top-of-README distillation is:
```text
<!-- nornir:gen:start:benches name=table_exists mode=compact compare=skade,nessie,polaris winner=row cols=ops_sec,p50_us,p99_us -->
<!-- nornir:gen:end:benches -->
```
That renders one tight read row (peak `ops_sec`, best `p50`/`p99` across skade
vs Nessie vs Polaris); a sibling `name=data_pipe mode=compact …` row does the
same for writes. Full per-backend detail stays in the long tables further down.
Implementation: `render_benches` (`src/docs/sections.rs`) builds the best-of row
by calling `bench::direction_of` per column and reducing with `f64::max`/`min`;
`top=N` sorts by the ranked metric's direction then truncates. Inject-assert
tests `benches_compact_mode_takes_best_per_column`,
`benches_compact_mixes_best_source_per_column`, and
`benches_top_n_keeps_strongest_rows` feed known rows and assert the exact
champion cells (not merely "didn't panic").
The **`tests`** renderer (`render_tests` + `docs::test_inventory`) enumerates
unit / integration / doc tests from source via `syn`, replacing hand-authored
test inventories. `docs init` now scaffolds both `benches` and `tests` markers
into a new repo's `.nornir/README.md`.
**Still open (plan.md → "Docs section renderers"):**
- `bench_history` renderer (funnel `i-002`) — N newest `bench_runs` rows, not
just the latest 1.
- A `criterion_targets` listing (the original `i-001` idea: read `[[bench]]` +
`criterion_group!` targets) — distinct from `benches`, which renders *results*.
- **Wiring** the znippy-zoomies decompressors' vs-legacy benches through Bencher
so `benches` has paired metrics to bold (see *Current state & gaps*).
## Surfaces
| surface | how |
|---------|-----|
| CLI | `nornir bench run <repo>` (measure + persist), `nornir bench history-show <repo>` |
| Release | `bench` stage in `run_pipeline`; gates `no_regression`, `nexus_floor`, `docs_fresh` |
| MCP | `bench_history`, `nexus_floor`, `no_regression`, `docs_fresh` |
| Docs | the `bench` marker section (above) |
| Raw | `cargo run --example nornir-bench` (single-package) / `cargo run -p xtask --example nornir-bench` (workspace) — what `nornir bench run` subprocesses |
| `cargo bench` shim (optional) | a `[[bench]]` target so the muscle-memory command runs the same suite — see below |
### The `cargo bench` shim (optional, opt-in)
The canonical entry point is the `examples/nornir-bench.rs` *example* — not a
`[[bench]]` — because examples may use `[dev-dependencies]` (a `bin` may not) and
nothing about nornir is allowed to leak into a published crate. nornir always
runs the example via `cargo run --example`; it never calls `cargo bench`.
If a repo wants the familiar `cargo bench` command to run the *same* suite, add a
`[[bench]]` shim. A `[[bench]]` is a separate binary, so it must link the same
`register_bench!` registrations — get that by `#[path]`-including the example
source (benches may use dev-deps, exactly like examples), so `inventory` collects
every bencher into the bench binary too:
```rust
// benches/nornir.rs
#[allow(dead_code)]
#[path = "../examples/nornir-bench.rs"]
mod suite; // links the register_bench! statics
fn main() -> anyhow::Result<()> {
nornir::bench::api::run_main_json()
}
```
```toml
# Cargo.toml
[[bench]]
name = "nornir"
harness = false
```
Then `cargo bench -p <pkg>` runs the identical suite. Caveats: it still emits
nornir's `BenchRun` **JSON**, not criterion's human output, and it is **not** a
second bench system — nornir keeps using the example. It is opt-in and intentionally
unwired by default. (Reference: znippy `xtask/benches/nornir.rs`.)
## Gotchas
- nornir is a **dev-dependency only** — never let it leak into the child's
published crate or its production build.
- `required_results` checks result **names**, not metric keys. A renamed result
fails the release loudly; a renamed *metric* silently drops from tables —
keep metric names stable (and suffixed `_mbs`/`_ms`/`_pct`).
- A bench that errors becomes a **red `TestOutcome`**, not a hard stop (unless
`NORNIR_BENCH_STOP_ON_ERROR=1`). Check `tests` in the run, not just `results`.
- VS baselines that shell out to legacy tools (`unzip`, `bzip2`, `pigz`) are
subprocess sites — keep them in the bench/example layer (dev-only), never in
library code, per the 100%-Rust law (plan.md §4).
## Current state & gaps (2026-06-05)
- **Wired:** holger (`xtask/examples/nornir-bench.rs`, incl. holger-vs-Nexus +
blake3/memcopy), znippy (`xtask/examples/nornir-bench.rs`), and **nornir-catalog**
(`bench/examples/nornir-bench.rs`, detached-bench-crate layout; embedded vs
nornir-rest vs Nessie `table_exists`, read fast paths, `data_pipe_*` ingest, and
`commit_burst_*` — the group-commit lever).
- **Warehouse-location gotcha:** nornir's default `[storage].local_path` is
hardcoded to `workspace_holger/.nornir` (`config.rs`, `Loaded::warehouse_root`).
Any **non-holger** workspace MUST set `[storage] local_path = "<workspace>/.nornir"`
in its `nornir.toml`, or `bench run` persists into holger's warehouse. (See
workspace_nornir-catalog.) A future fix: derive the default from the config
path's workspace dir instead of hardcoding holger.
- **Not wired:** the zoomies decompressors (`ljar`, `lbzip2`, `lgz`,
`lzip-parallel`, `linflate`) — no `register_bench!`, no `required_results`;
their vs-legacy numbers live only as hand-authored tables in the standalone
`*-rs` repos.
- **Landed renderers:** `benches` (bold-best, `name=`/`exclude=`/`bold=` args)
and `tests` are implemented in `src/docs/sections.rs` and wired in znippy's
`.nornir/README.md`. `docs render` selects the run via the iceberg-first
`last_run` (same as the gates), so a freshly-persisted `nornir bench run`
feeds docs — not just the legacy `bench_history.jsonl`.
- **Landed since:** the `bench_history` renderer (newest-first trend table:
`| Version | Date | Machine | metric |`, args `limit=`/`metric=`; fed from
iceberg on CLI, from the JSONL receipt on MCP/server) and a readable
`nornir bench history-show` (metrics + test pass/fail; `--json`/`--limit`).
Still missing: a `comparison` / "VS" renderer (see above).
- **`bench_runs` is now partitioned by `repo`** (iceberg partition spec) — purely
a read-side speedup (scoped reads prune to one repo's files); **invisible to
bencher authors**, the JSON-line write contract is unchanged. See
[warehouse.md](warehouse.md) §Partitioning.
- **Ingestion has two modes (landed):** local in-process append (default), or —
with `NORNIR_SERVER` set — ship over the `Bench.Submit` gRPC RPC to the server
(`submit_bench_remote` in `src/bin/nornir.rs`, the tree's first tonic client;
needs `--features server`/`mcp`). The child still emits the JSON line in both
modes (see the ⚠ note). The RPC path is lossless (results + tests in the proto)
and the server persists into iceberg `bench_runs`, same as the local path.
- **Fixed:** `last_run` is now iceberg-first (`docs_ctx_for` / the gates), so a
freshly-persisted `nornir bench run` is rendered immediately; the
`bench_history.jsonl` is only the fallback. Verified end-to-end on nornir-catalog
(bench run → `docs render` picked up the new numbers without touching the jsonl).
**To finish benchmarks + tests + vs-legacy on the znippy README:**
(1) add decompressor benches with paired ours/legacy metrics; (2) declare their
`required_results` (a `[repo.<crate>.bench]` per decompressor, plan.md → "Bench
corpora & harness"); (3) land the `bench_history` / `comparison` renderers and
fix the iceberg `last_run` surfacing; (4) the `benches`/`tests` marker blocks
already stay in sync as runs come and go.