# Benchmarking: nornir bench, not cargo bench
knut has exactly **one** benchmark suite and it follows the nornir contract
(see znippy's `.nornir/benching.md` for the original rationale):
- Each bench is a struct implementing nornir's `Bencher` trait, registered
with `register_bench!`; `inventory` collects them at link time.
- `xtask/examples/nornir-bench.rs::main()` is `nornir::bench::api::run_main_json()`
— every registered bencher runs and **one `BenchRun` JSON line** goes to
stdout, ready for the warehouse.
- `xtask/benches/nornir.rs` is a `[[bench]]` shim that `#[path]`-includes the
example, so `cargo bench -p xtask` runs the identical suite.
## Run
```bash
cargo run -p xtask --release --example nornir-bench # canonical (JSON to stdout)
cargo bench -p xtask # same suite, familiar verb
```
## Suite
| Bencher id | What it measures |
|---|---|
| `knut.format_parquet` / `_csv` / `_json` / `_orc` / `_arrow_ipc` | 100k-row chunk write+read through `graphar::io`, rows/s + file bytes |
| `knut.builder_dump` | `VerticesBuilder` → chunked parquet, rows/s |
| `knut.merge_property_groups` | two-group row-aligned merge, rows/s |
| `knut.bifrost_fabric` | 100k table rows → vertex+edge `GraphMutation`, rows/s |
| `knut.file_source_scan` | `FileSource` read of 4 parquet chunks (100k rows) from a watched dir — the AIP-82 file-listener path, rows/s |
| `knut.ingest_lifecycle` | full create→detect→ingest round (`Listener.sync_once`: FileSource → transform fabric → null sink) over 4 chunks (100k rows), rows/s |
| `knut.cypher_render` | Arrow → Cypher rendering (the FalkorDB-ingest hot path): 100k rows × 4 cols via the column-resolved `ColRenderer`/`write_row_map`, rows/s. See [`perf.md`](perf.md) §H1 |
| `knut.falkor_bulk_encode` | `GRAPH.BULK` blob encode (DB-free): `build_bulk_payload` on 100k nodes + 100k edges — Arrow → typed binary bulk wire, blobs presized from the Arrow data, fanned over gatling. rows/s + payload bytes |
| `knut.falkor_decode` | Results-back decode (DB-free): a canned 100k-row × 4-col `GRAPH.QUERY` reply (`redis::Value` RESP tree, Int/BulkString/Double/Boolean + Nils) → Arrow `RecordBatch` via `response_to_batch_auto`, rows/s |
| `knut.falkor_parallel_dispatch` | FalkorDB client-side pipelining win (DB-free): 512 simulated `UNWIND-CREATE` round-trips, the pre-fix SERIAL one-at-a-time await loop vs the shipped `buffered_dispatch` at `available_parallelism()`, reporting `speedup_x` (→ ~concurrency) + serial/parallel windows_per_s. The half of the ~1-core ingest fix that isn't the sim; live nodes/s + `cores_busy` stays the container path's |
| `knut.iceberg_pushdown_read` *(feature `skade`)* | The Iceberg `SyncMode::Full` read three ways over one warehouse (1000000 rows, 11 columns, 4 data files): the pre-fix full-width `read()`, the pipeline-derived projection (`read_columns`), and projection + predicate (`read_filtered`), plus the plan-time `files_skipped` from `plan_stats`. Reports `projection_speedup_x` / `filter_speedup_x` |
| `knut.cdc_pushdown_read` *(feature `skade`)* | The **change-feed** read four ways over one warehouse and one change window (1000000 inserts + 3 equality-deletes, 11 columns, 4 data files): the pre-fix full-width changelog read, the pipeline-derived projection (`ChangelogBatch::project_columns`), projection + `country == SE` predicate, and the fully **renderable** `(op, before, after, snapshot, ts)` encode with update pairing. Reports `projection_speedup_x` / `filter_speedup_x` and a `pushdown_kind` field naming this as post-decode/post-scan — **not** the plan-time file skipping `knut.iceberg_pushdown_read` measures |
| `knut.doput_dispatch` *(fixture-gated on FalkorDB)* | Flight SQL DoPut serial vs parallel bulk insert vs the in-process reference, 40000 vertices — `speedup_x` and `doput_vs_inprocess_x`. Skips gracefully with no FalkorDB at `KNUT_FALKORDB_URL` |
| `knut.skade_iceberg_export` *(feature `skade`)* | Graph → Iceberg export via skade: the **zero-copy** in-memory Arrow bridge + `fast_append`, 100k rows, rows/s + `warehouse_bytes`. Off by default — `cargo run -p xtask --example nornir-bench --features skade` |
| `knut.skade_iceberg_export_parquet` *(feature `skade`)* | The same export through the **parquet temp-file** bridge — the baseline the zero-copy path beats. Measured ≈**2× faster** through the in-memory hand-off (≈5.4M vs ≈2.8M rows/s, release) |
## Latest run
100k-row workloads, `--release`, `--features skade`, 32-core machine
(median of 4 runs; 2026-06-11). Each bencher is single-shot per run, so
nanosecond in-memory micros (`bifrost_fabric`, `merge_property_groups`) read as
multi-G rows/s — they bound the pure mutation/merge cost, not an IO path.
| Bencher | rows/s | Notes |
|---|--:|---|
| `bifrost_fabric` | 3.99 G | vertex+edge `GraphMutation` build (in-memory) |
| `merge_property_groups` | 5.35 G | two-group row-aligned merge (in-memory) |
| `format_arrow_ipc` | 75.2 M w / 150.6 M r | fastest round-trip; 3.03 MB |
| `format_orc` | 49.6 M w / 69.7 M r | smallest on disk; 1.79 MB |
| `format_parquet` | 7.17 M w / 37.1 M r | 3.55 MB |
| `format_json` | 7.89 M w / 1.11 M r | 4.86 MB |
| `format_csv` | 5.65 M w / 8.03 M r | 2.46 MB |
| `file_source_scan` | 32.7 M | 4-chunk watched-dir read (AIP-82) |
| `ingest_lifecycle` | 31.9 M | full `sync_once` to null sink (no DB round-trip) |
| `builder_dump` | 1.77 M | `VerticesBuilder` → 13 parquet chunks |
| **`cypher_render`** | **3.09 M** | Arrow → Cypher map literals, the FalkorDB-ingest hot path |
| **`skade_iceberg_export`** (zero-copy) | **5.37 M** | in-memory Arrow bridge + `fast_append`; 7.24 MB warehouse |
| `skade_iceberg_export_parquet` | 2.75 M | parquet temp-file bridge baseline |
The **zero-copy in-memory Arrow bridge runs ≈1.96× the parquet temp-file path**
(5.37 M vs 2.75 M rows/s) — the headroom a direct in-memory hand-off reclaims
over serializing through a parquet round-trip. (These numbers predate the
workspace unifying on **Arrow 58**, when the bridge became a plain identity
`clone()`; see [`reasoning.md`](reasoning.md).) Runs persist to the workspace
`bench_runs` Iceberg table via `nornir bench run knut` (the `--features skade`
opt-in is passed as `NORNIR_BENCH_FEATURES=skade`).
## FalkorDB boundary cost (measured live, 2026-10-19)
Live numbers against a real FalkorDB (100k rows × 3 cols, medians of 5):
`response_to_batch_auto` decodes at **7.0–7.6M rows/s (13–14 ms)** — the
client-side decode is NOT the results-back bottleneck. The RESP **wire**
(server serialize + client parse of the same reply) costs **150–162 ms**, ~11×
the decode, with server-side result production another 42–44 ms. So the honest
budget at this boundary is the wire representation, not `convert.rs`; a
column-parallel decode was prototyped, proven zero-loss and dropped — its win
ceiling was the 13 ms.
**DB-free arm numbers (dwarf, 6-core Ryzen 5 7535HS, container, nice -19,
2026-10-19; medians over interleaved runs — single-shot ~20 ms arms on this
shared box carry ±30% noise):** `falkor_bulk_encode` ~10M entities/s both
before and after the blob presizing (the presize is proven structurally — the
estimate is a tested upper bound, so the one up-front allocation is the only
one — but at 200k entities the saved reallocs sit inside the box noise);
`falkor_decode` ~4.2–4.6M rows/s (container; ~7.0–7.6M measured live on the
measurer's box); `falkor_parallel_dispatch` speedup_x ~12 on 12 logical cores,
unchanged across the buffer_unordered → `gatling::io::run` swap (same
semantics, proven by the peak-in-flight test); `cypher_render` ~1.8–2.2M
rows/s on this host (the 3.09M in the table above is the older 32-core row).
**`--compact` assessment — BUILT AND MEASURED 2026-10-20: keep it OFF.** The
flag is implemented end to end (`FalkorExecutor::with_compact(true)`:
`graphar-flight/src/compact.rs` rebuilds the exact verbose `Value` tree from
the type-tagged compact reply, id→name catalogs from
`CALL db.propertyKeys()`/`db.labels()`/`db.relationshipTypes()` refreshed on
miss, so the ONE existing decoder runs either way — zero-loss pinned live by
`tests/integration_compact.rs` on scalars, lists, maps, points, nodes and
edges, plus DB-free pins in `compact.rs`; red-first: the live parity test
fails with the raw compact reply handed to the old decoder). The `redis` crate
(0.29) parses the compact wire fine. The measured verdict is a REGRESSION on
this client stack (release, dwarf, 100k rows, medians of 5, live server):
| shape | verbose wall | compact wall | compact/verbose |
|---|--:|--:|--:|
| `RETURN n.id, n.name, n._gar_id` (scalar ×3) | 146.8 ms | 327.0 ms | 2.2× slower |
| `RETURN n` (whole nodes) | 1087.0 ms | 1410.3 ms | 1.3× slower |
Why: scalar rows carry NO repeated names in verbose RESP2, so compact's
`[type_id, value]` framing is strictly more RESP tokens (every Int cell
becomes a 2-array) — the shape the 150–162 ms was measured on cannot win by
construction (the 146.8 ms verbose arm here reproduces that number). On entity
rows the dropped name strings do shrink the wire, but the extra per-cell
arrays cost more in the `redis` `Value` parse than the names save, and the
zero-loss contract (equal `RecordBatch`es) adds the verbose-tree rebuild on
top. `with_compact` ships OFF by default; a *direct* compact→Arrow decoder
with its own entity rendering could revisit the entity shape, but that changes
the produced batches — a different contract.
Ingest lanes for scale (same live session, 200k rows): `GRAPH.BULK`
**1.3–2.6M nodes/s** vs UNWIND ~**14.5k nodes/s**; keyed edges
(`insert_edges` MATCHing both endpoints on `_gar_id`) ~**2.7k edges/s** — the
slowest lane in the pipeline by ~5×.
**Keyed-edge lane, re-measured 2026-10-20 (dwarf, container, live server —
`graphar-falkordb/tests/live_keyed_edges.rs`):**
* The 2.7k edges/s DID have the `_gar_id` index (the measuring harness created
it). **Without** it the plan is a per-row `Node By Label Scan`: **~17
edges/s** at 100k nodes, measured live. The loader now creates the endpoint
key indexes ITSELF before the first window (`insert_edges*`, serial and
`_par`; composite keys via one idempotent per-property `create_index` each —
FalkorDB coalesces them into the label's index and the composite `MATCH`
plans `Node By Index Scan`, probed live), so the 17 edges/s footgun is gone
by construction. Red-first: a fresh keyed load on a virgin graph listed NO
index before the fix, does after.
* Edge windows now ride a **`CYPHER rows=[…]` parameter** into a constant
`UNWIND $rows … MATCH … CREATE` body (same row maps, same writer —
zero-loss pinned by the literal↔param oracle `param_windows_store_identical
_graph` and the exact-triples robot suite). The body is parsed/planned once
and every later window hits the plan cache. Interleaved A/B on the same box
(100k chain edges, `_gar_id` indexed, debug harness): literal **2472 / 2532
edges/s**, param **2911 / 2699 edges/s** — a consistent but modest **+9–15%**;
the param value parse still dominates the window. (Same-code runs ranged
2.5k–11.5k edges/s across the day on this shared box, so the interleaved
pairs are the honest comparison; release-profile spot check: 3334 edges/s.)
The remaining big lever for initial loads stays `GRAPH.BULK`.
Fixtures are synthetic and tempdirs live in `/tmp` — local disk, never the T9
USB drive (external IO would skew throughput). Container-bound performance
(FalkorDB ingest, Spark reads) lives in the `#[ignore]` integration tests,
not in the release gate. The `skade` bencher is **opt-in** (`--features skade`)
so the default release gate stays free of skade's heavy arrow-58/iceberg tree.
## Host row: oden (2026-07-22)
**oden** is a NEW host row — a Threadripper PRO 3975WX, **32 physical cores, SMT
off, 499GB RAM**. It is not the `loki 12 cores` machine the numbers above were
taken on; the two must never be merged into one table.
`NORNIR_MACHINE=oden`, `--release`, `--features skade`.
### `knut.iceberg_pushdown_read` — the Iceberg full-width-scan fix
1000000 rows x 11 columns across 4 data files; the mapping reads 2 columns.
| arm | s | rows/s | vs full |
|---|--:|--:|--:|
| full (pre-fix `Table::read()`) | 0.355870 | 2810017 | 1.00 |
| projected (`read_columns`, 2 of 11) | 0.029391 | 34023725 | **12.11** |
| projected + `country == SE` (`read_filtered`) | 0.012222 | 20454872 | **29.12** |
Plan-time data skipping (`plan_stats`, metadata only — no data file opened):
`files_baseline` 4 → `files_planned` 1 (`files_skipped` 3);
`rows_baseline_planned` 1000000 → `rows_planned` 250000.
### `knut.doput_dispatch` — the Flight SQL DoPut fix
40000 vertices, FalkorDB on loopback, `concurrency` 32.
| arm | s | rows/s |
|---|--:|--:|
| serial dispatch (`put_concurrency = 1`) | 1.716508 | 23303 |
| parallel dispatch (`put_concurrency = 32`) | 1.443421 | 27712 |
| in-process `insert_vertices_par` (no Flight wire) | 1.464803 | 27307 |
`speedup_x` 1.19, `doput_vs_inprocess_x` 0.985.
The honest reading: the Flight transport costs ~nothing (DoPut is within 1.5% of
the in-process call), so the standing "DoPut is ~100x slower than in-process"
note does not reproduce. FalkorDB's per-graph write serialisation is the ceiling
(≈28000 nodes/s); an `UNWIND` window sweep from 512 to 40000 rows moved it under
10%. The client-side pipelining win is a ROUND-TRIP win and shows up when
FalkorDB is a network hop away — which is what the DB-free
`knut.falkor_parallel_dispatch` arm models (29.5x at a simulated 500us RTT on this
same host), not on loopback.
### `korp-collectors` cross-reference
Not a knut arm, but taken on the same host in the same session:
`collectors_registry/sample_sweep` — 58 collectors resolving their offline-safe
samples, 113.85 us, 224.89 MiB/s.