<!-- nornir:class: benchmarks -->
# Design: scalable benches in the nornir bench API
**Status:** proposal (for the nornir team to implement). Authored from the
skade-katalog consumer side, which has the first scalable bench (`tpch_warehouse`).
## Goal
Let a bencher declare *its own* scalable dimension (e.g. TPC-H scale factor, row
count, thread count) and let the **nornir runner** sweep it — running the bench at
several scale points and recording one result per point — so docs can plot a
scaling curve instead of a single number. The bench owns the meaning of "scale"
(it knows its workload); nornir owns the *driving* (which points, tagging,
persistence, rendering).
Key principle the consumer asked for: *"the bench impl adds a scale function since
it knows its own test; the nornir runner calls that to get the start values."*
## Non-goals
- No change to how single-point benches work (must stay 100% backward compatible —
the ~30 existing benchers implement only `id()` + `run()` and keep working).
- nornir does not interpret scale numbers; it only passes them through and tags
results. Units/semantics live in the bench.
## API (additive to `nornir::bench::api::Bencher`)
```rust
/// A scalable dimension a bench can sweep. The bench owns the meaning.
#[derive(Clone, Debug)]
pub struct ScaleAxis {
/// Short axis key, used to tag results and in config/CLI. e.g. "sf".
pub name: &'static str,
/// Human unit for docs. e.g. "TPC-H scale factor", "rows", "threads".
pub unit: &'static str,
/// The series the bench *recommends* if the runner/config gives none —
/// the "start values". e.g. vec![0.01, 0.1, 0.5].
pub default_points: Vec<f64>,
/// Hint for series generation / plotting (log vs linear x-axis).
pub log_scale: bool,
}
pub trait Bencher: Send + Sync {
fn id(&self) -> &'static str;
/// Single-point entry (unchanged; still the required method).
fn run(&self) -> Result<BenchResult>;
/// Declares a scalable dimension. `None` (default) ⇒ single-point bench:
/// the runner calls `run()` exactly once, today's behavior.
fn scale_axis(&self) -> Option<ScaleAxis> { None }
/// Run at one scale point. Default ignores the scale and calls `run()`,
/// so a bench that overrides only `scale_axis()` still works; scalable
/// benches override this. No recursion: `run()` stays the required method.
fn run_at(&self, scale: f64) -> Result<BenchResult> {
let _ = scale;
self.run()
}
}
```
Backward compatibility: existing benchers implement `run()` only → `scale_axis()`
returns `None` → runner behaves exactly as today.
## Runner logic (`run_main_json` / nornir bench run)
```text
for b in benchers:
match b.scale_axis():
None => push(b.run()) # unchanged
Some(axis) =>
points = config_override(b.id()).unwrap_or(axis.default_points)
for p in points:
let mut r = b.run_at(p)?
r.metrics.insert(axis.name, p) # e.g. "sf" = 0.5
r.name = format!("{}@{}{}", r.name, axis.name, fmt(p)) # tpch_warehouse@sf0.5
push(r)
```
- One `BenchResult` per scale point (the existing `BenchRun.results: Vec<...>` holds
the series — no struct change needed beyond adding the axis metric).
- Tagging: both a `metrics[axis.name] = p` entry (machine-readable) **and** a name
suffix `@<axis><p>` (so existing name-substring docs filters still group the
series). Pick one canonical form; suffix is friendliest to current docs filters.
- A scale-aware bench failing at point `p` records a failure for `name@<p>` only;
other points still run (same per-result failure handling as today).
## Config / CLI overrides (the "start values" come from here or the axis default)
`nornir.toml`:
```toml
[repo.<name>.bench.scales]
# override the bench's default_points; key = bencher id
tpch_warehouse = [0.1, 0.5, 1.0]
tpch_q01 = [0.01, 0.1, 0.5]
```
CLI:
```
nornir bench run <repo> --scale tpch_warehouse=0.1,0.5,1.0
```
Resolution order per bench: CLI `--scale` > `[bench.scales]` in nornir.toml >
`ScaleAxis::default_points`.
## Docs rendering (nornir docs)
When a `benches` region's results share a base name and carry the axis metric,
render a **scaling table** (one row per scale point) or a small curve, instead of a
single row:
```
| sf | build_s | query_s | total_s | ingest_rows_per_sec |
|-----|---------|---------|---------|---------------------|
| 0.1 | ... | ... | ... | ... |
| 0.5 | ... | ... | ... | ... |
| 1.0 | ... | ... | ... | ... |
```
The `<!-- nornir:gen:start:benches name=tpch_warehouse -->` marker already groups
by name-substring; it just needs to detect the `@<axis>` series and pivot on it.
## Gates / history
- `bench_runs` persistence: unchanged — each scale point is a row, with the axis
value as a column (from `metrics`), so regression gates can compare like-for-like
scale points across runs (compare `tpch_warehouse@sf0.5` to its prior `@sf0.5`).
- `required_results` in nornir.toml may reference a specific point
(`tpch_warehouse@sf0.5`) if a gate should require it.
## Reference consumer implementation (skade-katalog, once the API lands)
```rust
impl Bencher for TpchWarehouse {
fn id(&self) -> &'static str { "tpch_warehouse" }
fn scale_axis(&self) -> Option<ScaleAxis> {
Some(ScaleAxis { name: "sf", unit: "TPC-H scale factor",
default_points: vec![0.1, 0.5, 1.0], log_scale: false })
}
fn run_at(&self, sf: f64) -> Result<BenchResult> { /* build_ctx_scaled(sf, cores) + run 22 q */ }
fn run(&self) -> Result<BenchResult> { self.run_at(0.5) } // default single point
}
```
## Open questions for the nornir team
1. Name-suffix vs metric-only tagging as the canonical series key (this doc
recommends *both*, suffix canonical).
2. Should `default_points` be `Vec<f64>` (numeric, simple) or a richer
`ScalePoint { value: f64, label: String }` for non-numeric axes? Numeric is
enough for SF/rows/threads; revisit if a categorical axis appears.
3. Per-point timeout / budget so a runaway high-scale point can't wedge a sweep.