<!-- nornir:class: benchmarks -->
# Engine A vs Engine B — HEAVY benchmark
> Head-to-head of the two swappable advisory-matching strategies behind
> `engine::AdvisoryEngine`: **Engine A (linear scan)** vs **Engine B (inverted
> index)**. Same verdicts (asserted equal every run); the question is cost. The
> rationale is in `.nornir/engine-reasoning.md`; the harness is
> `benches/engine_ab.rs`.
## Method
- Harness: `benches/engine_ab.rs` (custom, `harness = false` — measures **whole-fleet
throughput at full parallelism**, not per-call latency).
- The whole synthetic fleet is scanned through each engine on **znippy-zoomies'
no-barrier fork-join gatling** (`gatling_for_each`, one worker per core — the SAME
path the shipped `AdvisoryEngine::fleet` / warehouse scan run; **no rayon**), one
shared engine instance. Best of 3 timed passes per engine.
- Reproducible SplitMix64 corpus (no `rand` dep): advisories are a mix of
exact-version / OSV-range / all-versions rows, ~⅛ withdrawn, ~1/16 KEV, across 4
ecosystems; artifacts include deliberate misses + GPL/unknown-license + unsigned
rows so license/provenance/KEV/EPSS escalation all fire.
- **Equivalence is asserted inside the bench**: the run aborts unless Engine A and
Engine B produce the *identical* total finding count. HEAVY only — no smoke variant.
- Run:
```sh
export CARGO_TARGET_DIR=/mnt/work4t/cargo-target/modgunn-engine
cargo bench --features scan --bench engine_ab
```
- **Regression guard (Loki / `nornir bench run modgunn`).** This head-to-head is the
deep `cargo bench` sweep. The nornir Bencher harness (`bench/examples/nornir-bench.rs`)
additionally ships a lighter, historized **`modgunn.fleet_scan_indexed`** arm — the
same Engine B fleet path over the shared fixture, with the one-shot index build +
bucket count recorded and a baked-in `B ≡ A` finding-count assert — so Engine B's
throughput is tracked run-to-run alongside Engine A's `modgunn.fleet_scan`, not only
in the manual heavy bench.
## Machine
| resource | value |
|---|---|
| cores (full parallelism) | **12** (the base tables) · **32** on oden · **16** on t14s |
| system memory | 61 GiB total (~42 GiB free at run) · t14s: 46 GiB |
| target dir | `/mnt/work4t` (fast PCIe4 disk, per disk-layout law) |
| build profile | `bench` (optimized) |
## Results
### Point 1 — artifact-scaled (1 M-artifact fleet, mid-size db)
`advisories=25 000 packages=3 000 artifacts=1 000 000 cores=12`
| engine | build (s) | scan (s) | verdicts/s | µs/verdict | peak RSS |
|---|--:|--:|--:|--:|--:|
| **A** linear | 0.0000 | 31.174 | 32 078 | 31.174 | 291 MiB |
| **B** indexed | 0.0053 | 0.160 | **6 260 387** | **0.160** | 291 MiB |
Identical findings: **1 667 658** (A == B). B index buckets: 10 477.
**Winner: Engine B — 195×.**
#### Point 1 · re-run on **oden** (32-core) — 2026-07-26
`advisories=25 000 packages=3 000 artifacts=1 000 000 cores=32` · host **oden**, box **uptime 12:19**
| engine | build (s) | scan (s) | verdicts/s | µs/verdict | peak RSS |
|---|--:|--:|--:|--:|--:|
| **A** linear | 0.0000 | 8.115 | 123 225 | 8.115 | 305 MiB |
| **B** indexed | 0.0037 | 0.057 | **17 525 920** | **0.057** | 305 MiB |
Identical findings: **1 667 658** (A == B — the **same count** as the 12-core run on the same
SplitMix64 corpus: a cross-machine large-scale equivalence confirmation). B index buckets: 10 477.
**Winner: Engine B — 142×.** Absolute B throughput ~2.8× the 12-core box (faster CPU + more cores); the
A/B *ratio* is lower (142× vs 195×) because Engine A's linear scan parallelizes across 32 cores better
than the already-saturated index, narrowing the gap while both get faster.
#### Point 1 · re-run on **t14s** (16-core) — 2026-07-27
oden is powered off, so Point 1 was re-run on **t14s** (16 CPUs / 46 GB). A new host is a NEW row —
never a merge into the oden or 12-core tables above, and no number here is restated from either.
`advisories=25 000 packages=3 000 artifacts=1 000 000 cores=16` · host **t14s**, box
`uptime` **0,58** before → **17,67** after (16 gatling workers on 16 cores; no foreign process
present on either side)
| engine | build (s) | scan (s) | verdicts/s | µs/verdict | peak RSS |
|---|--:|--:|--:|--:|--:|
| **A** linear | 0.0000 | 64.288 | 15 555 | 64.288 | 302 MiB |
| **B** indexed | 0.0094 | 0.177 | **5 643 647** | **0.177** | 302 MiB |
Identical findings: **1 667 658** (A == B) — byte-identical to BOTH the 12-core and the oden runs on
the same SplitMix64 corpus, a third-machine confirmation of the A/B equivalence assertion.
B index buckets: 10 477. **Winner: Engine B — 362.8×.**
**Reproducibility.** A second clean-start run (`uptime` 0,68 → 18,63) gave A 66.602 s / B 0.205 s /
325.2× — A within 3.5 %, B within 15 %, which is the expected spread for a **0.2-second** measurement.
It is cited as corroboration only, not as the row, because a foreign build appeared on the box near its
end. Engine B's sub-second scan is inherently the noise-sensitive number here; Engine A's ~65 s is the
stable one.
**Two earlier runs on this host were REJECTED, not published.** Both picked up a foreign `rustc` build
partway through (load on a 16-core box: 1,12 → 37,17 and 0,67 → 45,24) and disagreed with each other by
30 % on scan time — they measured contention, not the engine. They are recorded here as discarded so the
published row above cannot be confused with them:
| rejected run | `uptime` before → after | A scan (s) | B scan (s) | speedup |
|---|---|--:|--:|--:|
| ⛔ 1 | `1,12` → `37,17` | 61.457 | 0.505 | 121.7× |
| ⛔ 2 | `0,67` → `45,24` | 58.167 | 0.623 | 93.3× |
Note how badly contention distorts the *ratio* specifically: Engine A's 60-second linear scan barely
notices a competing build, while Engine B's 0.2-second indexed scan is stretched 3×. A shared box makes
the A/B speedup look 3–4× *worse* than it is — which is exactly why a contended run must never be
published as a host row.
### Point 2 — db-scaled (large advisory corpus, higher index memory)
`advisories=200 000 packages=80 000 artifacts=300 000 cores=12`
| engine | build (s) | scan (s) | verdicts/s | µs/verdict | peak RSS |
|---|--:|--:|--:|--:|--:|
| **A** linear | 0.0000 | 173.615 | 1 728 | 578.718 | 213 MiB |
| **B** indexed | 0.1393 | 0.054 | **5 554 512** | **0.180** | 213 MiB |
Identical findings: **359 906** (A == B). B index buckets: 148 915.
**Winner: Engine B — 3 214×.**
## Reading it
- **Engine B wins decisively, and the margin grows with the advisory-db size** — 195×
at 25 k advisories, **3 214× at 200 k**. That is the linear engine's
`Θ(artifacts × advisories)` versus the index's `Θ(artifacts × hits_per_package)`
made concrete: at Point 2 the linear scan drops to **1 728 verdicts/s** while B holds
**5.5 M/s**.
- **Engine B's build cost is real but tiny and one-shot**: 5.3 ms at 25 k advisories,
139 ms at 200 k — amortised over the whole fleet, negligible next to the seconds/
minutes it saves. It is *not* repaid for a one-artifact gate scan, which is why the
default stays **A**.
- **Memory is modest and dominated by the corpus, not the index** (213–291 MiB peak);
the `usize`-handle inverted index adds little over the advisory rows themselves.
- **Correctness is not a trade-off**: every run asserts A ≡ B on total findings, so B's
speed costs nothing in results.
## Verdict
Use **Engine A** for the lean per-artifact gate (tiny db, one artifact — the index
never pays off). Use **Engine B** for every **fleet / warehouse scan** over the real
CVE-warehouse corpus: `modgunn warehouse scan --engine b`. Above ~10⁴ advisories it is
one-to-three orders of magnitude faster for identical output.
---
# Manifest ingest — HEAVY parse-throughput benchmark
> Pure-Rust, zero-shell parse of each ecosystem's manifest/lockfile into the shared
> `sbom::SbomComponent` model — the ingest side of `manifests.rs`. The question is raw
> parse throughput (components/sec), not a match. Harness: `benches/manifest_parse.rs`.
## Method
- Harness: `benches/manifest_parse.rs` (custom, `harness = false`). One large synthetic
manifest per ecosystem (one representative format each), generated **untimed**, then
parsed through the public `parse_manifest` dispatch; best of `REPEATS` passes.
- Single-threaded per format (parsing is per-file); reports components/sec, input size,
and peak process RSS. Host-stamped plain numbers.
- Run:
```sh
export CARGO_TARGET_DIR=/mnt/work4t/cargo-target/modgunn
cargo bench --features manifests --bench manifest_parse
# tunables: MODGUNN_BENCH_DEPS (default 50 000), MODGUNN_BENCH_REPEATS (default 5)
```
## Machine
`host=loki cores=12 build profile=bench (optimized) target dir=/mnt/work4t (PCIe4)`
## Results
`deps_per_manifest=50 000 repeats=5 cores=12`
| ecosystem | format | components | best (s) | throughput (comp/s) | input |
|---|---|--:|--:|--:|--:|
| cargo | Cargo.lock | 50 000 | 0.0659 | 759 008 | 2.5 MiB |
| pip | requirements.txt | 50 000 | 0.0053 | 9 449 922 | 0.9 MiB |
| npm | package-lock.json | 50 000 | 0.0271 | 1 845 357 | 3.4 MiB |
| maven | pom.xml | 50 000 | 0.0234 | 2 134 820 | 5.7 MiB |
| go | go.sum | 50 000 | 0.0090 | 5 566 602 | 2.0 MiB |
| gem | Gemfile.lock | 50 000 | 0.0036 | 13 967 214 | 1.1 MiB |
Total: **300 000** components across six ecosystems; peak RSS **88 MiB**.
## Reading it
- **Hand-parsed text formats dominate** (gem 14.0 M, pip 9.4 M, go 5.6 M comp/s) —
line-oriented parsing with no allocation-heavy document tree.
- **Structured formats pay for their tree**: TOML (`cargo`, via the `toml` crate) is the
floor at 759 k comp/s, XML (`maven`, roxmltree) 2.1 M, JSON (`npm`, serde_json) 1.8 M —
all still fast enough that parse is never the fleet-ingest bottleneck (the advisory
match is — see Engine A/B above).
- **Memory is flat** (88 MiB for 300 k components) — the shared `SbomComponent` model is
four owned strings, no per-ecosystem bloat.