<!-- nornir:class: history -->
# Verifying a UI / gRPC / render / CLI surface — the canonical LAW
> **What the matrix IS.** A green test-matrix row is a *provable claim that the
> surface a user actually ships and touches works* — the deployed artifact, the
> real data, the whole call chain, driven through the server, swept across its
> outcome space, asserted from emitted data with no human looking at a screen. It
> is not "the code compiled," not "a native build passed," not "it rendered
> something." When the matrix is green, the thing it names is *correct and ran*;
> when that is not true, the matrix is lying, and a lying matrix is worse than no
> matrix.
>
> **The creed (one line):** *Green ⟺ the shipped surface, on real data, through the
> server, across its whole outcome space, was proven RAN-and-CORRECT from emitted
> data — and the same oracle goes RED when it breaks.*
>
> This document is the law every UI / gRPC / render / CLI surface in the
> constellation obeys — `nornir` viz (fat + thin), the CLI (fat + `--server`),
> `facett-*` components, `korp-poc`, the MCP tools, and anything added later. It is
> a chapter of the docs book (`docs/book.pdf`) and is linked from the 🧪 Test pane
> (`state_json["test"]["doctrine"]`). Read it before you touch a red row, and read
> it before you claim a new surface is done.
---
## The eight LAWS
Each law is a rule, the **real failure that birthed it** (so you know it is not
theoretical), the concrete *how* (what to drive, what to assert, how to prove the
RED), and a tiny assert sketch. They are not ranked by importance — they are ranked
by how often they were the hole the bug crawled through.
### LAW 1 — SHIPPED ≠ TESTED: the subject under test is the DEPLOYED ARTIFACT
**Rule.** The test subject is the artifact that actually *ships* — the wasm/web
bundle for a web demo, the installed binary for the server, the published crate for
a library. Native-build green proves the native build; it does **not** prove the
wasm build a user loads. Therefore: require **native↔wasm surface parity** (the same
deck/tabs/buttons/sliders exist in both), AND a **headless-wasm robot-UI** that
loads the *actual shipped bundle* and asserts every expected surface is both
**present** and **ran** (via its emitters). A surface that is green natively but
absent or non-running in the shipped wasm is **RED**.
**The failure that birthed it.** The matrix tested the *native* `egui_kittest`
build and went green: 27 facets, every button present. But the deployed wasm demo
was missing / non-rendering the Sverige showcase buttons and sliders. What we
*tested* and what we *shipped* were two different binaries. The native green was
true and worthless.
**How.**
- Enumerate the shipping deck from the *native* build's `Tab::ALL` /
`state_json["all_tabs"]` (the canonical enumerator) and from the wasm bundle's
loaded surface; assert the two sets are equal — parity is a test, not a hope.
- Drive the shipped wasm headless (load the real `*.wasm` the deploy serves) and
read back each surface's emitter. Present-but-never-ran is still RED: a button
that exists but whose handler never fires emits nothing.
- Cache-bust the deployed bundle (`?v=<sha>`) and confirm the live bytes are the
NEW build before you trust any "it's fixed on the demo" claim — browsers cache
`*.wasm` by its constant filename and will serve the stale build (see META §4).
```rust
// parity: the deployed deck must equal the tested deck
assert_eq!(wasm_surface_ids(&loaded_bundle), native_surface_ids(&state_json),
"wasm bundle is missing surfaces the native test covered");
// present AND ran: every showcase control fired its emitter in the SHIPPED build
for id in SHOWCASE_CONTROLS {
assert!(wasm_emits.contains_key(id), "shipped wasm never ran surface {id}");
}
```
### LAW 2 — RAGNARÖK: empty / zero data on a data-bearing surface SHALL NOT PASS
**Rule.** **EMPTY / ZERO DATA IS NOT A PASS — IT IS RAGNARÖK.** A test or surface
that goes GREEN on empty / zero data is a *broken test*, not a passing one. Every
data-bearing surface MUST assert REAL POPULATED data (count > 0, real values, real
edges / rows / nodes) and go **RED on zero / empty**. There is no "renders empty
cleanly" pass for a surface that is supposed to carry data — empty = ERROR, **it
shall not pass.**
**The failure that birthed it.** The 🚇 Metro map rendered empty stubs and
"passed." A blank Metro map / empty boundary landmarks counted green, which is
exactly the row that would have surfaced the missing `deep_scan` → `call_edges`
the instant it had been forced RED.
**POPULATE-DIAGNOSTIC corollary.** When a surface is empty, the test AND the agent
MUST report WHY (which scan / table / config is missing) and WHAT the user must do
to get data — e.g. *"`deep_scan = true` not set → no `call_edges` scanned"*, or
*"the warehouse has no `call_edges` for this workspace → run a populate / deep
scan"*. If the matrix cannot itself populate the data, that is **not** a reason to
pass — it is a reason to FAIL RED and name the populate gap to the user. Never
silently pass an empty surface.
**Prove the RED by sensitivity.** Pair every populated-state assertion with its
inverse: feed the surface empty input and assert the SAME emit row goes RED. A
populated-state metric that cannot go red on empty input is asserting nothing.
(`predicates_go_red_on_empty_or_errored_slices` is the matrix-baked form of this.)
```rust
let s = drive(metro, &populated_warehouse).state_json();
assert!(s["metro_edges"].as_u64().unwrap() > 0, "RAGNARÖK: empty metro is RED");
// inverse, baked into the matrix:
let e = drive(metro, &empty_warehouse).state_json();
assert_eq!(e["metro_edges"], 0, "and the SAME oracle must go red on empty");
```
### LAW 3 — ONE VALUE ≠ COVERED: sweep the SAMPLE SPACE, per-function and whole-chain
**Rule.** A surface is covered only when every function on its call chain is swept
across its **whole outcome space** (equivalence classes + boundaries → exact
expected output), AND the **composed chain** is swept across the same input space
asserting the end-to-end contract. One value inside one partition proves nothing the
next value in that partition would. (English math term: **sample space**, Ω — the
set of all possible outcomes. "utfallsrum" is the Swedish origin name; identifiers
use `sample_space` / `SampleSpaceCoverage`.)
**The failure that birthed it.** A single green value was counted as "covered." A
function that mapped one partition correctly and the next partition wrong shipped
green, because the one value the test happened to pick landed in the correct
partition.
**How — the `facett-map3d` exemplar (the canonical shape).** Sample-space coverage
of a chain `f1 ∘ f2 ∘ … ∘ fn` is BOTH of:
- **(a) per function `fi`** — `facett-map3d/tests/chain_sweep.rs` drives a giant
fixture through EACH stage `f1(view_space) → f2(clip_tri_near) → f3(project_view)`
across 360°×100zoom×elevations, asserting EVERY stage's contract across the whole
input space (not just the last stage).
- **(b) the composed chain** — `facett-map3d/tests/partstep_sweep.rs` drives the
WHOLE chain `near_clip → project → cull → shade → depth_sort → scissor →
composite` across 7200 poses, asserting the end-to-end oracle
(`scissor.clipped_escapes == 0`) at every point.
Make it first-class with the framework: declare the outcome space, record each class
with its exact output, let the gate score it. A test that hits only the final stage,
or only one input value, is **not** sample-space-covered.
```rust
use nornir_testmatrix::utfallsrum::Outcome; // public surface: sample_space
let mut o = Outcome::for_fn("clip_tri_near")
.class("behind-near", "cz < near ⇒ vertex dropped")
.class("on-near", "cz == near ⇒ vertex kept, t==0")
.class("front", "cz > near ⇒ vertex unchanged");
// sweep one case per class, record the EXACT output as evidence:
o.hit("on-near", "clip_tri_near(cz=near) keeps vertex, t==0.0");
assert!(o.summary().meets_threshold(2)); // ≥K classes or the surface stays in the gap
```
> Cross-link: the framework primitive (generic, UI-agnostic — works for a headless
> library with no viz/server/workspaces) lives in
> [`../crates/nornir-testmatrix/.nornir/testmatrix-utfallsrum.md`](../crates/nornir-testmatrix/.nornir/testmatrix-utfallsrum.md)
> + its [adoption guide](../crates/nornir-testmatrix/.nornir/testmatrix-utfallsrum-guide.md).
> nornir's surface enumerator (LAW 5) is nornir *applying* that primitive — keep the
> two layers cleanly separated (the jewel is reusable; the viz/fat-thin/workspace
> gate is nornir's dogfood harness on top of it).
### LAW 4 — END-TO-END = THROUGH THE SERVER, not the local fat path
**Rule.** nornir is built fat/thin client → server. "End to end" therefore means the
FULL round-trip, not the in-process path:
```
GUI button / CLI verb → thin client → gRPC RPC → server handler
→ warehouse (read/append) → gRPC response → back to client/GUI surface
```
The matrix MUST exercise the real server round-trip (`@thin`) for every workspace ×
every GUI-function against a running `nornir-server`, and assert the data that came
**back from the server** — not a local read. `@fat` (embedded/in-process) is the
twin and must also be covered, but the `@thin` cells are the ones that prove the
fat/thin → server construction actually works. CLI `--server` parity is total:
whatever a viz tab shows via the server, the CLI verb must return the same via the
server (assert parity as data).
**The failure that birthed it.** Surfaces were proven on the *local fat path* and
called done. The thin client → gRPC → server → warehouse construction — the thing
that actually ships in remote mode — was never driven, so a thin-mode surface could
be stub-degraded or warehouse-blind while the fat test stayed green.
**How.**
- Stand up an ephemeral `nornir-server`; point viz `--server` and CLI `--server` at
it. The surface key carries the mode: `<workspace>/<kind:id@thin>` vs `@fat`.
- Assert the round-tripped payload, and assert `fuel_source == "warehouse"` (thin)
vs `"checkout"` (fat) so a drive *proves* thin mode is genuinely warehouse-backed,
not silently degraded to a local stub.
- Assert fat↔thin parity: the same query returns the same data through both ends.
```rust
let thin = cli(["dep-graph", "--server", &srv.addr, "--json"]).run();
let fat = cli(["dep-graph", "--json"]).run();
assert_eq!(thin.json["edges"], fat.json["edges"], "thin↔fat parity through server");
assert_eq!(thin.json["fuel_source"], "warehouse"); // proven server-backed, not stubbed
```
### LAW 5 — WORKSPACE BLIND: the surface is workspace × {tab@{fat,thin}, cli, mcp} × chain, clean-slate→populate→verify
**Rule.** A green row for one workspace does not prove the surface. The completeness
surface has three dimensions; cover all three or the matrix is not full:
1. **Workspace dimension.** The surface is `workspace × {cli_command,
viz_tab@{fat,thin}, mcp_tool, grpc} × call-chain-to-end` for EVERY served
workspace — read the live set from `workspaces_list`
(`discover::SERVED_WORKSPACES`), never a hardcoded list. The same tab that works
for `nornir` can be blank for `knut`; they are DISTINCT surfaces, each keyed
`<workspace>/<kind:id@mode>` (`workspace_surface_key`). CLI is a parallel END of
each chain with TOTAL parity to the GUI — never GUI-only.
2. **clean-slate → populate → verify.** A surface counts as covered only when, from
an EMPTY warehouse — a fresh one via a throwaway `$HOME` **or** an absolute
`[storage].local_path` pinned to a temp dir in `nornir.toml` (there is **no**
`NORNIR_WAREHOUSE_ROOT`; path env-vars were removed — the warehouse is `$HOME`-derived
per-workspace) — the REAL pipeline (clone → `nornir map` → `nornir test`/`bench`/`sync`)
populates it AND the GUI
then shows REAL data read from `state_json`. A pane that renders only because the
test *injected* the rows is NOT the populate→show path — it is a seeded fixture,
and a seeded fixture cannot prove the populate pipeline works.
3. **Sample-space (LAW 3)** — every fn on the chain swept ≥K outcome classes.
4. **Atom granularity — enumerate DOWN TO THE ATOMS via the live AccessKit tree.**
A tab is not an atom. Every rendered widget — button, label, **error-text** — is
an **AccessKit node** that `egui_kittest` walks; that walk is the only thing that
sees **sub-panel atoms** (e.g. the 🧬 tab's CloneEvents "populate status" + Jobs
panels), which coarse `Tab::ALL` / per-pane `state_json` are blind to. The
verifier renders each `tab × {empty-registry, populated}` through a
`kittest::Harness`, walks the tree, and asserts **(a)** NO error atom
(`is not served` / `Failed to load` / `unavailable`) in the empty state, **(b)**
the expected populated atoms present (LAW 2 RAGNARÖK). This is the mechanical
"find each UI element" — the **markers** (`button_registry` → 🚇 metro) are the
DECLARED truth; the **atom-walk VERIFIES** them and is auto-derived back into
them. STOP enumerating at tab/Facet granularity; STOP hand-listing surfaces in
bespoke tests — one atom-walk covers all tabs × all atoms × states, emitting
`surface_coverage` rows autonom reads like any marker. (egui has the discovery
primitive — AccessKit — so "no deterministic way to find each element" is FALSE.)
See [marker-matrix-plan.md](marker-matrix-plan.md) §2b for the L0–L3 layering.
**The failure that birthed it.** The matrix proved a tab for one workspace and
treated the surface as done; other workspaces' identical tabs went untested and
could be blank. Separately, panes that rendered from seeded rows passed without ever
proving the real populate path — rendering with seeded data is not enough; it must
survive the real clean-slate→populate path.
**How.** For each `ws` in `workspaces_list`: fresh temp warehouse → run the real
populate verb → drive every `{tab@fat, tab@thin, cli, mcp}` → assert real data from
`state_json`. If a pane is empty after populate, that is a BUG to fix (LAW 2's
populate-diagnostic), not a pass.
```rust
for ws in live_workspaces() { // from workspaces_list, not hardcoded
let wh = TempWarehouse::empty();
populate_real(ws, &wh); // clone → map → test/bench/sync
let s = drive(viz_tab(ws, Mode::Thin), &wh).state_json();
assert!(s["rows"].as_array().unwrap().len() > 0, "{ws}: blank after REAL populate");
}
```
### LAW 6 — VERIFIED BY EYEBALL is not verified: every emptiable render EMITS its populated-state as DATA
**Rule.** Every viz render that CAN be empty MUST emit its populated-state as DATA
(`functional_status` counts), so correctness is asserted from the EMIT, never from
*"does it look right?"*. If a render can degrade to blank / stub, it has to publish
the numbers that distinguish populated from empty, and a test reads them back —
headless, no human.
**The failure that birthed it.** An agent kept asking the human to eyeball a 3D
viewer — the entire emit/harness machinery had been built precisely so that was
never necessary. If you find yourself asking "does it look right?", you have skipped
the doctrine.
**Worked example — the 🚇 Metro map (`src/viz/arch_tab.rs`).** A pixel/render stage
whose correctness can't be read from a return value, so an in-fn `functional_status`
emitter (gated behind `--features testmatrix`) is the right tool. It ships, per
render:
- `metro_edges` — call-path rail segments actually drawn (0 ⇒ blank);
- `metro_nodes` — total stations (functions) drawn;
- `metro_emitter_nodes` — how many stations are MARKED emitters (emit* / trace /
`functional_status` sites along the line);
- `metro_stub_lines` — lines still degraded to a click-handler-head stub *whose
handler IS resolvable*; **MUST be 0 when populated**;
- `fuel_source` — `warehouse` (thin) vs `checkout` (fat), so a drive proves thin
mode is genuinely warehouse-backed (LAW 4), not stub-degraded.
The RAGNARÖK test (`tests/viz_arch_metro.rs`,
`metro_thin_warehouse_backed_reaches_grpc_and_emits_populated`) drives the thin metro
against a populated warehouse and asserts `metro_edges > 0` AND
`metro_stub_lines == 0` AND `metro_emitter_nodes > 0`; its inverse
(`metro_thin_empty_warehouse_is_red_not_silently_green`) feeds an empty warehouse and
asserts the SAME row goes RED. That is the fix verified WITHOUT a human looking at
pixels. Build every emptiable surface this way: emit the populated-state counts,
assert them populated, prove the RED with the empty-input inverse.
### LAW 7 — PESSIMISM: green ONLY when proven ran-and-passed-and-swept; an unverified surface is presumed BROKEN
**Rule.** Distrust every "done." A surface is presumed BROKEN until the matrix shows
it RAN end-to-end, PASSED a tight oracle, and SWEPT its sample space. "✅ all tests
pass" is not evidence — the evidence is the matrix row, with its data, on the shipped
path. Optimism is the bug.
**The failure that birthed it.** "Done" claims kept being false: "✅ all tests pass"
shipped while the live demo was visibly broken, because the oracle was blind or the
tested path was not the shipped one. Hours burned trusting the claim over the matrix.
**How.** Treat a green row with an unproven/loose oracle as a RED *finding* until you
have seen it go red on the bug (LAW 8's sensitivity proof). When a fix "works" but
the symptom persists, you fixed the wrong path/scene — re-check the path (META §6),
do not re-explain the fix. The matrix is only trustworthy when it is **provably
full**: green ⟺ ran-and-passed, blank ⟺ *you haven't driven it yet* (so go drive it),
red ⟺ real failure.
### LAW 8 — WIRE ≠ HOME: the terminus is the Iceberg WAREHOUSE, via the lock-owning server
**Rule.** `functional_status` rows emit to a JSONL wire (`NORNIR_TESTMATRIX_OUT`);
that file is a *verification convenience*, NEVER the destination. **A green row in a
/tmp JSONL is NOT "done."** Done = the rows **reconciled into the Iceberg warehouse**
via `nornir test` (run **through the server that owns the redb write-lock**) and
**read back FROM the warehouse** (the viz Test pane / `test_history`). A wire-only,
never-reconciled row is RED / incomplete.
**The failure that birthed it.** ~4491 facett graphskins call-chain rows were
correctly emitted but pointed at `/tmp` to count them, and counted as done. A
green row in a temp JSONL was waved at the human — the same failure as a blank pane.
**How.**
- A local run against a server-locked warehouse hits `redb: Database already open.
Cannot acquire lock` — that error is itself the finding: the server holds the
lock, so either point the run at the server (remote) or stop the server. Generating
data locally against a server-locked warehouse is the bug, not the data.
- Because the running server uses the *installed* binary (not a worktree), landing
rows in the LIVE warehouse is a **post-merge** step: merge → reinstall server →
`nornir test all` via `NORNIR_SERVER`. That last mile belongs in the merge
checklist, not in the leaf crate's test code.
- Read the rows BACK from the warehouse (the Test pane / `test_history`) and assert
they are there. Wire-counted ≠ warehouse-landed.
```rust
// not "rows in JSONL" — rows READ BACK from the warehouse via the lock-owner
let back = test_history(&server, ws, run_id);
assert!(back.iter().any(|r| r.test_name == "metro_thin_warehouse_backed…"),
"row emitted to wire but never reconciled into the warehouse = RED");
```
---
## The false-green checklist — run this before you ever say "done"
Every "no" is a RED finding. A surface is not done until every box is "yes."
- [ ] **Shipped artifact?** Did I test the build that actually ships (wasm bundle /
installed binary / published crate), not just the native build? Is the
native↔wasm surface set proven equal, and did the shipped build's emitters
fire? *(LAW 1)*
- [ ] **Empty goes RED?** On a data-bearing surface, does empty/zero input drive the
SAME oracle RED — and did I prove it (fed empty, watched it fail)? Did I name
the populate gap when empty? *(LAW 2)*
- [ ] **Sample space swept?** Did every fn on the chain sweep ≥K outcome classes
(boundaries + partitions, exact expected output per class), AND the composed
chain across the same space — not one value, not only the last stage? *(LAW 3)*
- [ ] **Through the server?** Did I drive thin (`viz --server` + CLI `--server`) →
gRPC → server → warehouse → back and assert the round-tripped data + fat↔thin
parity + `fuel_source == warehouse`, not the local fat path alone? *(LAW 4)*
- [ ] **Every workspace, clean-slate→populate→verify?** Did I cover every
`workspace × {tab@{fat,thin}, cli, mcp}` from `workspaces_list`, from an EMPTY
warehouse through the REAL populate pipeline — not seeded fixtures? *(LAW 5)*
- [ ] **Asserted from emitted data?** Is correctness read from `state_json` /
`functional_status` counts, never from "does it look right?" *(LAW 6)*
- [ ] **Oracle can go RED + is contract-tight?** Did I prove the oracle goes red on
the broken code (stash the fix → red → restore → green), on the same path +
scene + pose the user runs, with the threshold = the exact contract bound (not
a `×4`/`×8` slack multiple)? *(LAW 8 of "oracle can go red" below; pessimism,
LAW 7)*
- [ ] **Landed in the WAREHOUSE?** Did the rows reconcile into Iceberg via the
lock-owning server, and did I read them BACK — not just count a /tmp JSONL?
*(LAW 8)*
- [ ] **Did I distrust the "done"?** Am I presuming this BROKEN until the matrix
proves otherwise, rather than trusting an "✅ all pass"? *(LAW 7)*
---
The rest of this document is the *technique* the eight laws lean on — the doctrines
and worked methods that have, in practice, been what actually catches the bug. They
are kept because each one was paid for in burned hours.
## The loop — validate as DATA, not pixels
1. **The component already emits what it renders, as readable data.** Read it, don't
look at it:
- `state_json()` — every Facet/pane reports its live state (camera pose, counts,
visible-feature count, the geometry it *intends* to draw).
- `nornir_testmatrix::functional_status(component, check, ok, detail)` — the
matrix row. Drive the component headless, assert the return value, emit one row.
Green matrix ⟺ every surface works.
- the robot-UI driver (egui_kittest / `nornir-robotui`) — *drive* real clicks /
scroll / tab-switch and read the resulting `state_json`.
So your first move is **assert on emitted data**, never on a screenshot.
2. **EMIT-DOCTRINE — emit only where the return value / `state_json` can't already
prove it.** If a function returns the answer, just assert it; don't add an emit.
The one class data can't trivially capture is *"did it render non-blank /
correct?"* — and even that you turn into data: emit the **geometry as numbers**
(bbox, vertex/triangle counts, "longest projected edge", "fraction of pixels ≠
background", "triangles spanning the whole box"). A starburst, a black pane, a
degenerate mesh — all become a number you can assert. Add *lots* of such emits;
they are cheap and they are how you "see".
3. **Only if a property genuinely cannot be reduced to data, render it and read the
image yourself.** You can read PNGs. Render headless (egui_kittest wgpu → PNG),
`Read` the file, judge it. This is the *fallback*, not the default — and if you
reach for it twice, add the emit (or the pixel scanner, below) that would have
caught it instead.
## Walk the WHOLE call chain, start → end (the BUILD-TIME mandate)
The binary-search section below finds ONE known-red bug fast. This is the opposite
and it is **primary**: when you **build or touch** a surface, you must track and
verify EVERY function in its call chain, input → pixel, so a bug can never sit
untested. "I tested the thing that was broken" is NOT "I verified every function I
wrote" — an agent once fixed the buggy component's chain and left the *other*
components' chains only partially covered; that is the failure this section prevents.
1. **Enumerate the chain.** Write it down, input → … → pixel (e.g. a 3D map:
`read → project → near-clip → frustum-cull → back-face-cull → shade → depth-sort →
scissor → rasterize → composite`). Use the call graph to be exhaustive —
`knowledge_call_path` / `dwarf_call_path` / `arch`. This list is the spec your
tests mirror: one verified checkpoint per node, **no gaps**.
2. **Verify every node.** For EACH function on the chain, a **matrix-evaluated**
check: assert the **return value / `state_json`** where it proves the function
correct (counts, finiteness, bbox, ordering, the resolved value); add a
`functional_status` **emit** only where the return can't prove it — the pixel
stages (`lit_px`, `ink_outside_rect == 0`, `max_projected_extent ≤ viewport·K`,
byte-identical-to-reference), per EMIT-DOCTRINE. **No node ships unverified.**
3. **Sweep it — the sample space, not one case (LAW 3).** Generate a parameter grid
(input sizes, geometry/topology kinds, camera poses incl. the nasty/degenerate
ones, canvas sizes, worker counts); run EVERY case through the chain, asserting at
each stage + emitting one matrix row per case. One example proves nothing about
the space; the grid does.
4. **Robot-drive the live surface.** Use the robot-UI harness (egui_kittest /
`nornir-robotui`) to click / scroll / navigate / orbit the REAL component, read
back `state_json` + the emit chain, assert correct + confined + non-blank, AND
scan the pixels (below). A wrong / escaped / blank frame = a RED matrix row.
5. **Emit the final picture as a PNG blob** so an agent (or the viz) can *see* the
rendered frame straight from the matrix — no human opening a window.
**Outcome:** the test matrix is the single source of truth for "does this work."
Green ⟺ the entire call chain, across the whole sample-space sweep, driven through
the real UI and the server, is proven — so the human reviews the **matrix**, never
the screen. Do this for EVERY function you write, not only the one that was red.
## THE ORACLE MUST BE ABLE TO GO RED — the threshold *is* the test
A green test proves NOTHING unless its oracle would turn RED on the actual bug. This
is the #1 way a day gets burned: "✅ all tests pass" ships while the live demo is
visibly broken — because the oracle was blind or its threshold was loose. (2026-06-21:
the OSM-3D map smeared green spokes across the whole frame in the deployed wasm demo
while `golden_248`, `no_spokes`, `render_proof`, and a 1000-case grid were ALL green.
The `max_extent_px` oracle existed — but its cap was `half_h·8`, i.e. it *tolerated a
triangle 8× the viewport*. A frame-filling smear is "inside an 8×-viewport cap," so it
passed. The oracle looked rigorous and was worthless.) Before you ever say "fixed",
satisfy ALL of:
1. **Define the bug as ONE numeric invariant on the LAST function that feeds pixels**
— the faces/mesh actually painted, not an intermediate. State it as a number: "no
painted triangle bbox exceeds the widget rect", "max projected edge ≤ viewport
diagonal", "fraction of fill-vs-background pixels ∈ [a,b]". NOT "non-blank", NOT
"is lit", NOT "didn't panic" — those pass on a smear.
2. **The threshold = the REAL contract bound, and you must justify the number.** A
cap that is a loose multiple of the viewport (`×4`, `×8`) passes the exact smear it
pretends to guard. If the correct answer is "no geometry escapes the viewport", the
cap is the viewport (`≈ 2·half_h`), full stop. Write the cap and a one-line reason
next to it; a magic loose constant is a bug in the test.
3. **PROVE the test goes RED on the current/broken code BEFORE you fix it.** `git
stash` the fix (or run against the pre-fix path), watch the new assertion FAIL,
restore, watch it PASS. A test you never saw fail is not evidence — it may be
asserting nothing. "Sensitivity proof or it didn't happen."
4. **Same PATH + same SCENE the user actually sees.** Render through the exact code
path the product uses (here: the CPU `egui` painter the wasm demo runs, NOT a
`--features wgpu` GPU path the demo never takes) and the exact fixture / camera
pose / data. A golden that looks fine because it rendered a different path/pose
than the demo is a classic false green.
5. **Then, and only then, look at ONE rendered frame** to confirm the invariant
actually captures "looks right" — the image is the *check on the oracle*, never
the proof itself. If the frame looks wrong but the test is green, the oracle (not
the frame) is the bug — tighten it until it goes red.
> Copy-paste prompt that forces this: *Define the bug as ONE tight numeric invariant
> on the LAST function feeding pixels. Assert it on the SAME render path + scene/pose
> the demo uses. PROVE the test goes RED on the current code (stash the fix → red →
> restore → green) before claiming a fix. The threshold must be the real bound, not a
> loose multiple — state and justify the number. Only after green, look at one frame
> to confirm the invariant means "looks right." green ⟺ correct, or the oracle is
> wrong.*
## BLACK-BOX THE CHAIN WITH KNOWN VALUES — assert EXACT expected output per stage
Driving the component via the demo (real scene in) is necessary but NOT sufficient.
You ALSO must do **black-box, known-value testing of the whole chain**: insert a KNOWN
input and assert the EXACT expected value comes out of EACH function `f1 → f2 → f3 →
f4 → … → pixels`. Compute the expected value independently (by hand / closed form / a
reference impl), not by capturing whatever the code currently emits.
- Feed a **hand-chosen fixture** with a known answer: a single triangle at known world
coords, a known camera pose ⇒ you can compute by hand the exact view-space coords
(`f1=view_space`), the exact clipped polygon (`f2=clip_tri_near`), the exact screen
pixels (`f3=project_view`), the exact post-bound face (`f4=screen_bound`). Assert
each stage's output **equals** that hand-computed value (within float ε).
- This catches a stage that is *plausible but wrong* — projects to the wrong place,
drops the wrong vertex, flips a sign — which a loose property ("within viewport")
passes. Property invariants + known-value exactness are BOTH required: the invariant
says "nothing escaped"; the known-value says "it landed exactly HERE".
- Do it for **every function on the chain**, with a small `(input → expected output)`
table per stage covering the boundaries. A stage with no known-input→known-output
row is untested no matter how many demo frames you ran.
> One-liner: **insert known values, assert exact known outputs at every f1→f2→f3→f4
> boundary, THEN drive the real demo scene end-to-end and scan the pixels. Both, or
> it's not verified.**
## SCAN THE PIXELS — the image oracle must ANALYZE the frame, not assert "non-blank"
A robot-UI / render test that claims it "sees" the frame MUST run image analysis on
the received pixels and assert the result is the EXPECTED structure. "non-blank" /
"has lit pixels" / "wrote a PNG" is NOT seeing — it passes on a smear, a starburst, a
garbage frame. (2026-06-21: `robot_ui` claimed to drive the OSM-3D to the failing pose
and check the frame "confined + non-blank" — and was GREEN while the live frame was a
green spoke smear. It never analyzed the image. That is a lie in the oracle.) If a
test renders pixels, it must compute features from them and assert them:
- **Spoke / starburst detection** (the exact bug class): a smear is thin
high-contrast lines radiating from points. Detect with a high-pass / edge filter
(Sobel/Laplacian) → count long thin connected high-gradient runs; assert ~0. A
correct lit mesh has compact filled regions, not radiating 1-px lines.
- **Frequency signature**: FFT / low-pass vs high-pass energy ratio. A smeared frame
has anomalous high-frequency energy along few orientations (the spokes); a clean
render's spectrum is broad/smooth. Assert the high-freq fraction is within the band
a known-good frame produces.
- **Coverage / distribution**: fraction of non-background pixels ∈ [a,b]; the painted
mass's bounding box and centroid sit where the geometry should be (not flung to a
corner); no single thin triangle covers a cross-frame diagonal.
- **Reference compare** when a blessed frame exists: perceptual diff (SSIM / per-tile
mean) under a TIGHT threshold — and prove the diff trips on the broken frame.
Build these as a reusable image-probe (e.g. a `facett_core` image-analysis helper) so
every component's render test calls the SAME scanner: `scan(frame) -> {spokes,
hf_ratio, coverage, centroid, …}`, then assert each feature. **If your render test
does not compute a number FROM the pixels and assert it, it is not an image oracle —
it is a lie.**
## EVERY EMITTER IS TWO AXES: input coverage × oracle strength
A check is `assert(invariant(f(input)))`. It has TWO independent ways to be
worthless; close BOTH for EVERY emitter / `functional_status` row:
**Axis A — input coverage (LAW 3: MULTIPLE values, not one).** Drive each function
with **min, max, mid, AND the boundaries/degenerate cases** — derived from the
function's OWN code:
- **Every `if` / clamp / divide is a boundary** → test just-below, AT, just-above, on
BOTH sides. (`if cz <= NEAR_PLANE` ⟹ cz = near−ε, near, near+ε. A divide ⟹ the input
that drives the denominator → 0 — *that* is the smear case.)
- **Degenerate / adversarial**: empty, 1 element, zero-area, camera inside the
geometry, NaN/Inf, huge & tiny scale, the nasty camera pose. Bugs live at edges.
- Parameterize a grid covering every equivalence class.
**Axis B — oracle strength (the threshold IS the test).** Each case asserts a
contract-tight, sensitivity-proven invariant (see "oracle must be able to go red"). A
loose cap passes the very bug it pretends to guard.
**They are ORTHOGONAL and BOTH required.** The OSM-3D `matrix_grid` swept ~1000 poses
(Axis A ✓) and was GREEN — because the oracle cap was `half_h·8` (Axis B ✗). 1000
values × blind oracle = green-on-broken.
### Concrete failure to never repeat (the smear, 2026-06-21)
`facett-map3d` shipped `SCREEN_BOUND_K = 4.0` (clips only to ±4× the viewport) and a
`max_extent_px` cap of `half_h·8` (tolerates a triangle 8× the viewport). Both are
**loose multiples of the viewport**, so near-grazing triangles survived as
frame-filling green spokes that every green test missed. **A screen/extent bound is
the VIEWPORT (≈ `2·half_h`), never a multiple of it.** Any `*_K` / cap constant that
is a multiple of the viewport is a test bug until proven otherwise.
### GENERALIZED — every function's output has a CONTRACT bound; assert THAT, not a slack multiple
EVERY function has an exact set/range/relation a correct output must satisfy — its
**contract**. The oracle asserts the contract bound; a *tolerance / slack constant*
(`×2`, `×8`, "close enough", "big enough") is the loophole the bug hides in:
| component / fn kind | input boundaries (Axis A) | exact contract bound (Axis B — NOT a multiple) |
|---|---|---|
| **render / map / graph3d** | near-grazing pose, camera inside geom, empty/1-tri mesh, huge/tiny scale | every painted vertex inside the **widget rect** (= viewport, not k×it); 0 ink outside |
| **graph layout** | 0/1/N nodes, disconnected, self-loop, complete graph | every node+edge position **inside the canvas**, finite (no NaN/Inf), degree == model |
| **table / grid** | 0 rows, 1 row, N rows, max col widths, unicode/empty cells | rendered row count **==** input rows (no silent truncation); every cell value in its column domain |
| **plot / chart** | empty series, single point, min==max, negative, huge range | every point maps **inside the axes**; tick min/max **bracket the data exactly** |
| **projection / transform** | denom→0 input, identity, extremes | output range = the transformed input range; **no divide-by-~0 escape** |
| **parser / codec / IPC** | empty, 1 byte, max size, malformed | **roundtrip identity**; byte/elem count preserved; schema honored |
| **search / index** | 0 hits, 1 hit, all-match, boundary key | result set **==** brute-force truth on the same input |
The general rule, domain-free:
1. Derive input boundaries from the function's own branches/divides/clamps + the
degenerate cases of its type. Sweep them all (Axis A).
2. Assert the EXACT contract bound the spec implies — equality, containment,
finiteness, conservation, roundtrip — never a slack multiple/tolerance (Axis B).
3. Prove the assertion goes RED on the broken code before the fix.
4. If you wrote a `_K` / cap / epsilon that is a *multiple* of the true bound, treat
it as a bug until you justify the exact number against the contract.
> One-liner: **sweep the input boundaries the code itself defines, and assert the
> exact output contract — coverage × exactness, both proven able to go red. A
> tolerance constant is where the bug lives.**
## MULTIVARIATE + METAMORPHIC stress — the bug lives in the SPACES BETWEEN functions
Single-axis sweeps (rotate-only, zoom-only) are the amateur trap. A graphics engine is
a deeply interconnected multivariate chain: **day/time, zoom, camera rotation, aspect
ratio, and the coordinate matrices all change at once**, and the math breaks
**between** functions where two moving variables interact — never visible when you
vary one at a time. Test the JOINT space.
1. **Multivariate state vectors — the CARTESIAN product, not one-var-at-a-time.**
Generate a massive diverse matrix combining every axis simultaneously:
`day ∈ 1..365 × zoom ∈ 1..100 (incl. float extremes: tiny, huge, denormals) ×
azimuth 0..360 × elevation × aspect/bbox`. Use `proptest` (shrinking finds the
minimal failing vector) OR clean nested matrix loops. The bug you cannot see is the
one where `zoom×rotation×day` coincide.
2. **Chain-of-function integrity end to end.** Projection feeds facet generation feeds
clipping feeds raster. Assert data integrity is preserved THROUGH the whole chain
on every state vector — not just at the ends.
2b. **EVERY function gets its OWN multivariate sweep — not just the last clip stage.**
`f1` (view_space), `f2` (near/frustum clip), `f3` (project), `f4` (screen-bound), …
EACH gets the full day×zoom×rotation×aspect matrix run through IT and its output
asserted (finite, topological, in-range), AND the chain. A bug in `f1` at a corner
of the space is invisible if you only sweep `f4`. (This is the per-stage half of
LAW 3 — the `chain_sweep.rs` shape.)
3. **The NaN/Inf plague — assert NO stage produces or propagates `NaN`/`Inf`.** Every
divide, normalize, matrix inversion is a source. After EACH function assert all
outputs `.is_finite()`. A single NaN silently poisons the rest of the chain and
renders as a blank or a smear. The cheapest, highest-value invariant — add it
everywhere.
4. **Topological sanity.** Transformed facets must stay geometrically valid: every
index points to a real vertex, polygons stay closed (first==last / ring intact),
triangle winding doesn't unpredictably invert (a flip = inside-out face =
back-face-cull eats it = hole/smear), array lengths stay consistent. Assert after
each transform.
5. **Metamorphic relations — relate OUTPUTS across related INPUTS** (you don't need the
exact pixel, only how it must CHANGE):
- **zoom ↑ (day, pose fixed) ⇒ total rendered facet area monotonically ↓ or ==**,
never grows, never glitches. (Catches the smear: a smear makes area JUMP UP.)
- rotate 360° back to start ⇒ frame identical (round-trip).
- day/time shift ⇒ only lighting/shade changes, geometry/coverage invariant.
- translate the whole scene ⇒ projected shape congruent (area/topology preserved).
6. **Actionable failure tracing.** On any violation, panic/emit the EXACT state vector
`{day, zoom, azimuth, elevation, aspect, matrix}` AND which function broke which
invariant — reproducible from the message alone, no re-bisect. `proptest` shrinking
+ this trace = the minimal reproducer handed to you.
> One-liner: **sweep the JOINT multivariate space, assert finiteness + topology +
> metamorphic relations through the WHOLE chain, and on failure dump the exact state
> vector + stage. Single-axis green is not green.**
## META — what ACTUALLY catches the bug (do these FIRST, every time)
This session burned hours because the agent trusted numbers over pixels and tested the
wrong path. The moves that actually worked:
1. **LOOK AT THE RENDERED IMAGE. Every time. Before claiming anything.** Every number
lied at least once (a loose extent cap passed an 84 000px smear; a spoke_score of
0.226 "passed" a frame-filling GREEN terrain smear because the detector only knew
thin spokes). The IMAGE never lied. The oracle's only job is to AUTOMATE what the
image shows — so validate the oracle against a looked-at known-good AND a looked-at
known-bad frame, or the oracle is untrusted.
2. **Find the PRODUCT'S ACTUAL PATH before you test anything (LAW 1).** Which feature
flags does the SHIPPING build use? (wasm = `--no-default-features` → CPU painter, NO
wgpu — so testing the native `--features wgpu` GPU path proves nothing about the
demo.) Which DATA? (`load()` = real embedded OSM, not `synthetic`.) Which
fixture/pose? (the demo's own `from_geometry` framing, not your tidy test grid; the
close-zoom pose, not dist=3.2.) Reproduce the EXACT user scene end-to-end first.
3. **The oracle fails TWO ways — too loose AND wrong-kind.** A threshold that is a
multiple of the true bound passes the smear it guards (loose). A thin-spoke detector
misses a filled-area smear (wrong feature). One pixel metric is never coverage — use
several orthogonal ones (spoke/edge, filled-area/coverage spikes, frequency,
centroid, frame-spanning-sliver) AND look.
4. **Verify the DEPLOYED artifact is the NEW build (LAW 1).** Browsers cache `*.wasm`
by its constant filename → a redeploy serves the STALE build and the "fixed" bug is
still there. Cache-bust (`?v=<sha>`) and confirm the live bytes changed.
5. **Enumerate and instrument EVERY path** (CPU+GPU, every component, every
data/fixture branch), regardless of refactoring. The bug hides in the ONE path you
didn't instrument. If `state_json` says `visible_triangles: 0` but the frame has
content — or vice-versa — you are reading a different path than you are rendering.
Make every path emit a uniquely-named matrix row so "every enumerated path produced
a row" is itself assertable.
6. **When a fix "works" but the symptom persists, you fixed the wrong path/scene.**
Don't re-explain the fix — re-check #2 (is this even the path the user sees?).
> If you do nothing else: **look at the image, on the exact path+data+pose the user
> runs, on the actually-deployed build.** Numbers come second.
## Debug by BINARY SEARCH, the way a human bisects
When a test is red and you don't know which function is wrong, **do not** add a
debug/emit to every function top-to-bottom. Bisect:
1. Find the **midpoint** of the suspect call chain (input → … → output).
2. Add ONE emit/assert there: *is the value still correct at the midpoint?*
3. Run the test, read the emitted value.
- **Correct at the midpoint** → the bug is **downstream**; recurse into the lower
half.
- **Wrong at the midpoint** → the bug is **upstream**; recurse into the upper half.
4. Halve again. In `O(log n)` steps you land on the exact faulty function. Fix it,
rerun, watch the row go green, then **remove the scaffolding emits** (keep the one
that is a real regression guard).
This is exactly how a person single-steps with a debugger — but guided, halving the
interval each time instead of walking every frame.
## Follow the data — drop your assumption when it's disproven
State your hypothesis ("the 3D starburst is a geometry bug"), then let an emit **try
to falsify it** ("bbox is in-box, 0 outliers, max face 58px → geometry is clean"). If
the data kills the hypothesis, **change the hypothesis** — do not keep fixing the
thing you assumed. (Two plausible geometry fixes were wrong; the emit proved the mesh
clean and pointed at the *render architecture* — painter's-algorithm vs a GPU z-buffer
— which no geometry fix could ever address.)
## Every fix earns a real matrix test
Per repo law: a fix that isn't guarded by a `#[test]` that **fails on the bug and
passes on the fix** is not done. Prove the sensitivity (`git stash` the fix, watch the
test fail, restore). Then it emits `functional_status` so `nornir test` carries it in
the matrix forever — and reconciles into the warehouse (LAW 8), not a /tmp JSONL.
## If a surface is EMPTY because nothing has been RUN — RUN IT, generate the data
A pane that emits empty *"because no run produced data yet"* is **NOT a pass** (LAW 2)
— it is a surface you simply haven't driven. When data is missing, **RUN the action
that generates it** (the test matrix, a bench, a knowledge scan, a populate/clone) and
then assert the now-populated emit — for EVERY workspace and EVERY repo (LAW 5). "Empty
until run" ⇒ **run it.**
- **Heavy runs: one at a time**, sequentially (the global heavy gate).
- **Go through the server that OWNS the warehouse (LAW 8).** A local viz that emits
`redb: Database already open. Cannot acquire lock` is itself a finding: the
nornir-server holds the lock → point the viz at the server (remote) or stop the
server. Generating data locally against a server-locked warehouse is the bug, not
the data.
- The matrix is trustworthy only when **provably full**: green ⟺ ran-and-passed, blank
⟺ *you haven't driven it yet* (so go drive it), red ⟺ real failure.
## Scope
Apply this to **every** UI/gRPC/render/CLI surface: `nornir` viz (fat + thin), the
CLI (fat + `--server`), `facett-*` components, `korp-poc`, the MCP tools, and anything
added later. The agent sees and validates through emitted data + the headless harness;
the human reviews the matrix, not the pixels.
---
## Status — the doctrine is LIVE, with reference implementations
This is not aspirational; the release surfaces shipped are the canonical worked
examples of *"drive it headless, assert the emitted state, every fix earns a matrix
row."* (`cargo test --features viz,server --test robot_viz` → 11/11, `--test
saga_roundtrip` → 3/3.)
- **The 🧬 release dashboard's gate overlay = pure emitted data (LAW 6).** Every node
badge (`⛔`/`✗`/`▼`/`🟡`), the cut edges, the scoped-repo + blast radius, and the
Release button's enabled state are read back from `state_json["nornir_dashboard"]`
— no pixels. `robot_drives_release_dashboard_and_state_reflects_gate_overlay` asserts
`dash["gate_ok"] == false`, `node_badges[…]["badge"].contains('⛔')`, and
`cut_edges[0] == {from:"nornir", to:"znippy", …}` after a REAL AccessKit click.
- **Each surface earns matrix rows via `run_scenario`** (the EMIT-DOCTRINE: the
`Scenario`/`Assert::State` rows ARE the `functional_status` rows).
`robot_release_dashboard_scenario_fills_the_matrix` and
`robot_release_wizard_scenario_fills_the_matrix` (7 rows).
- **Saga replay = a recorded session turned into matrix rows** — the cleanest loop.
`tests/saga_roundtrip.rs`: a recorded njord journey (`nav OffertForm → click
Submit`) replays as a 2-row `robot-ui` run, and a regressed app (Submit widget
deleted) yields a RED row `widget "Submit" not found` — the "fails on the bug,
passes on the fix" sensitivity proof, built in (LAW 7/8 of "oracle can go red").
- **The viz functional-status matrix** (`tests/viz_functional_matrix.rs`) enumerates
the FULL viz surface (every `Tab::ALL` + chrome) and emits one `functional_status`
row per item in BOTH modes, including the sensitivity oracle
`predicates_go_red_on_empty_or_errored_slices` (LAW 2) and the sentinel-ban
`surface_slices_carry_no_unwired_sentinel`. See
[`viz-functional-matrix.md`](viz-functional-matrix.md).
The static edition gate (`coverage`-style `static_findings_only`) and the
sample-space-scored surface gate (`covered_with_utfallsrum`, LAW 3) are the pattern
any *new* UI/gRPC/CLI surface copies.
> **Open debt (LAW 8).** The facett graphskins call-chain rows (`facett-graph` /
> `facett-syschart` / `facett-pipeline`, ~4491 functional rows from the
> `callchain_matrix` + `robot_pixels` tests) were emitted but only sinked to `/tmp` to
> count them — land them in the warehouse **post-merge** via `nornir test all` on the
> workspace (merge → reinstall server → `NORNIR_SERVER`). The wire is not the home.