<!-- nornir:class: design -->
# H5 — Agent × model bake-off **MATRIX** (`nornir bakeoff`)
> Design + guide + idiot-guide for the H5 epic: ONE standardized way to pit a
> task against a **cross-product of agents × models**, land the results in the
> warehouse, and watch which **agent+model combo won** in a viz **Leaderboard**
> pane that shows the agent × model grid. Mirrors the `nornir test`
> run→persist→view shape and the `release_events` / `test_results` warehouse-row
> pattern, and reuses the H1 MCP-exercising + C6 run→persist→live shape.
## 1. What it is
`nornir bakeoff run --task "…" --agents a,b --models x,y` runs ONE task across
the **`agents × models` cross-product**, recording one `agent_model_runs` row per
`(task, agent, model)` **cell** — its answer plus **score, wall_ms, tokens/s,
cost, mcp_tool_calls** — into the warehouse under a fresh `run_id`, and prints
the leaderboard. The **agent** is the caller/harness identity (`claude`, `kilo`,
a local-LLM driver); the **model** is what it drove. The viz **🏆 Leaderboard**
tab reads the same table and renders the agent × model grid with the winning cell
highlighted; CLI parity is total — the pane shows nothing `nornir bakeoff
leaderboard --matrix` can't print.
A model-only bake-off is the 1×N matrix under a single local-LLM agent, so the
pre-matrix usage (`--models …` with no `--agents`) still works unchanged.
```
nornir bakeoff demo # seed a mock matrix
nornir bakeoff run --task "Fix the bug" --agents claude,kilo --models opus,mistral
nornir bakeoff run --task "…" --mock # offline canned cells
nornir bakeoff run --task "…" --models a,b,c # 1-agent (model-only)
nornir bakeoff leaderboard # newest run, ranked
nornir bakeoff leaderboard --matrix # agent × model grid
nornir bakeoff leaderboard <run_id|model|agent> --json # raw rows
```
The `--prompt` / `--prompt-id` flags are kept as visible aliases of
`--task` / `--task-id` for back-compat.
```text
nornir bakeoff run --task T --agents a,b --models x,y
└─ ModelCaller
├─[--mock] MockCaller (canned ModelAnswer per (agent,model) cell) ─┐
└─[default] OllamaCaller (POST {host}/api/generate) ────────────────┤
▼
run_bakeoff_matrix: one AgentModelRunRow per (agent,model) cell (shared run_id + ts)
└─▶ append_agent_model_runs → Iceberg agent_model_runs
├─▶ matrix_grid() ranks cells → MatrixGrid
│ ├─▶ render_matrix / render_leaderboard (CLI)
│ └─▶ 🏆 Leaderboard pane grid (state_json.matrix)
└─▶ leaderboard() flat ranking
└─▶ render_matrix / render_leaderboard (CLI)
```
## 2. The bake-off backend (`src/warehouse/agent_model_runs.rs`)
The only thing that touches an actual LLM / agent harness is the [`ModelCaller`]
trait — now **agent-aware** (the matrix's first axis):
```rust
pub trait ModelCaller {
fn call(&self, agent: &str, model: &str, prompt: &str) -> Result<ModelAnswer>;
}
```
`ModelAnswer` carries the economics: `output`, `latency_ms`, `tokens_in/out`,
`tokens_per_s`, `score`, **`cost_usd`**, **`mcp_tool_calls`**. Two impls ship:
- **`OllamaCaller`** — POSTs `{model, prompt, stream:false}` to
`{host}/api/generate` (`--host`, else `$OLLAMA_HOST`, else
`http://localhost:11434`) using the already-vendored `ureq` client. It ignores
the `agent` axis (ollama has no agent concept) but the row still records which
agent column the cell was placed in. Reads `response`, `prompt_eval_count`,
`eval_count`, `eval_duration` (ns).
- **`MockCaller`** — a `(agent, model) → ModelAnswer` map (`with_cell(agent,
model, …)`, or `with(model, …)` for the AGENT_UNSET shorthand). The `--mock`
flag (and every test) drives this so a bake-off needs **no running LLM or
Claude**. A cell absent from the map returns an error, which the matrix runner
records as a red `ok=false` row — exercising the failure path without aborting.
`run_bakeoff_matrix(caller, run_id, ts, task_id, task, agents, models)` is the
pure cross-product→call→row pipeline both the CLI and the tests share. It does
NOT touch the warehouse (the caller appends the returned rows), so a test runs a
full matrix with a `MockCaller` and asserts the rows before any I/O. Row order is
the `(agent, model)` cross-product — stable + testable. `run_bakeoff(...)` is the
single-agent shim (`agents = [AGENT_UNSET]`). `tokens_per_s` is taken from the
backend when supplied, else derived from `tokens_out / (latency_ms/1000)`.
The default model set (`default_models()`) is the embedding-model registry's ids
(`src/vector/embed_registry.rs`); the default agent set (`default_agents()`) is a
single local-LLM agent. When the `vector` feature is off, the model default falls
back to the same ids as a literal.
## 3. Warehouse table `agent_model_runs`
Schema: `iceberg_schema::agent_model_runs` (field ids 1–16). One row per
`(run_id, agent, model, prompt_id)` **cell**:
| column | type | meaning |
|---|---|---|
| `run_id` | string | bake-off-run identity (groups a run's rows) — **partition key** |
| `ts_micros` | timestamptz | run time (shared across a run's rows) |
| `agent` | string? | **matrix axis 1** — caller/harness (claude/kilo/local-llm); legacy null → `-` |
| `model` | string | **matrix axis 2** — the model driven (ollama tag / registry id) |
| `prompt_id` | string | stable task handle (so two runs compare) |
| `prompt` | string | the task text |
| `output` | string | the cell's completion (`""` = none, never null) |
| `latency_ms` | double | wall-clock call time (`wall_ms`) |
| `tokens_in` | long | prompt token count |
| `tokens_out` | long | completion token count |
| `tokens_per_s` | double | decode throughput |
| `score` | double | `0.0..=1.0` quality — set by the **codegen judge** (see `.nornir/bakeoff-judge.md`) when `--judge-crate` is passed; `0.0` if unscored |
| `ok` | bool | did the cell answer without error? |
| `error` | string? | error detail when `ok=false` (null on success) |
| `cost_usd` | double? | USD price of the cell (`0.0` = free/local) |
| `mcp_tool_calls` | long? | MCP tool calls the agent made answering |
Fields 14–16 (`agent`, `cost_usd`, `mcp_tool_calls`) are **OPTIONAL** so the
schema is migration-safe: rows written before the matrix carry none, and the
reader defaults them back to `agent = "-"` (`AGENT_UNSET`), `cost_usd = 0.0`,
`mcp_tool_calls = 0`. The serde struct also `#[serde(default)]`s them, so a thin
client decoding a pre-matrix server's JSON still parses.
Partitioned by **`run_id`**: a bake-off writes every cell in one append (one
batch = one partition), and the leaderboard reads a whole run back, so `run_id`
prunes scans. Per-model / per-agent history reads do a residual filter.
Writer/reader (`src/warehouse/agent_model_runs.rs`, mirrors `test_results.rs`):
- `append_agent_model_runs(wh, rows)` — one Iceberg snapshot per run.
- `query_agent_model_runs(wh, sel)` — `sel ∈ Run | Model | Agent | All`, sorted
by `(ts_micros, run_id, agent, model, prompt_id)` (Iceberg has no scan order).
- `leaderboard(rows)` — flat ranking, **best first**: `ok` cells outrank failed
ones; within those, higher `score`, then higher `tokens_per_s`, then lower
`latency_ms`, then lower `cost_usd`, then `(agent, model)` cell name (a stable
deterministic order the CLI + viz agree on, so a test asserts the exact rank).
- `matrix_grid(rows)` → `MatrixGrid { agents, models, cells }` — the agent ×
model grid. Each `(agent, model)` coordinate aggregates its task rows (mean
score/tok/s/latency/cost, sum of mcp calls, AND of `ok`) into a `MatrixCell`
with its rank (same comparator as `leaderboard`, so the grid winner = the
leaderboard winner). `grid.winner()` is the rank-1 cell.
- `demo_matrix_rows(run_id, ts)` — a realistic 3-agent × 3-model mock matrix
(one failed `local-llm/opus` cell) for `nornir bakeoff demo`.
- `rows_to_json` / `render_leaderboard` / `render_matrix` — the JSON + human
renderers (flat ranked table + the grid).
## 4. The viz Leaderboard pane (`src/viz/leaderboard_tab.rs`)
The **🏆 Leaderboard** tab reads `agent_model_runs` read-only + lock-tolerant
locally, or over the `Viz.BakeoffResults` gRPC remotely (decoding into the same
`AgentModelRunRow`, so the grid renders byte-identically thin):
- **Run list** (left) — one row per bake-off `run_id`, newest on top, with the
task handle, `agents × cells` count, and the winning `agent/model` cell.
- **Matrix grid** (right, top) — an `egui::Grid` with agents down the rows,
models across the columns; each cell shows `score #rank`, the winner cell in
gold (🥇), failed cells in red (✗), empty coordinates `·`. Hover = full
economics.
- **Ranked cells** (right, below) — the flat `leaderboard()` ranking, each row
`score · tok/s · latency · cost · mcp` with an ok/✗ chip.
LAW #6 — the pane exposes exactly what it renders as data via
`state_json()["leaderboard"]`: the run summaries, the flat `leaderboard` (with
`agent`/`model`/`cell`/`cost_usd`/`mcp_tool_calls`), and the `matrix` grid
(`agents`, `models`, `cells` with per-cell rank, `winner`) — folded into the
app's `NORNIR_VIZ_STATE` dump and the `$NORNIR_VIZ_TRACE` stream.
## 5. Tests (LAW #1 — inject real values, assert real output)
- `warehouse::agent_model_runs::tests`
- `run_bakeoff_builds_rows_and_records_failures` — single-agent path: two
models + one missing; asserts kept/derived tok/s + the red error row.
- `leaderboard_ranks_ok_then_score_then_throughput` — exact score→throughput
order, failed cell last.
- `matrix_crosses_agents_and_models_and_records_a_failed_cell` — a 2×2 matrix:
asserts the full cross-product, the per-cell `agent`/`cost_usd`/`mcp` values,
and that the absent `local/opus` cell is the red one.
- `matrix_grid_ranks_cells_and_picks_winner` — a 3×3 matrix: asserts the sorted
axes, that `claude/opus` (highest score) is the rank-1 winner, the cell ranks,
and that the grid winner == the flat leaderboard winner == `render_matrix`'s.
- `demo_matrix_is_populated_and_round_trips` — appends `demo_matrix_rows` to a
tempdir warehouse, reads it back, asserts 9 cells (1 red), the winner, that
cost/mcp survived, and that the `Agent` selector filters by agent.
- `legacy_single_axis_rows_default_agent_unset` — a model-only run lands under
`AGENT_UNSET` with zero cost/mcp.
- `append_query_round_trip_groups_and_ranks` — full mock bake-off round-trip.
- `tests/viz_leaderboard_pane.rs::leaderboard_pane_ranks_matrix_by_score_then_throughput`
— injects a 4-cell agent × model matrix, selects the Leaderboard tab, asserts
`state_json()["leaderboard"]` exposes the winning cell, the ranked board (with
cost + mcp), AND the `matrix` grid (sorted axes + per-cell rank + winner) with
the failed cell last.
- `src/viz/remote.rs::tests::bakeoff_results_wire_roundtrip` — asserts the matrix
axis + economics survive the `Viz.BakeoffResults` JSON wire, AND that a
pre-matrix server's JSON (no agent/cost/mcp) decodes to the `AGENT_UNSET`
defaults.
None require a running LLM or Claude — the backend is mockable by construction.
## 6. CLI parity
| GUI (🏆 Leaderboard pane) | CLI twin |
|---|---|
| run list (newest first) + winning cell | `nornir bakeoff leaderboard` |
| agent × model grid for a run | `nornir bakeoff leaderboard --matrix` |
| ranked cells for a run | `nornir bakeoff leaderboard <run_id>` |
| per-model / per-agent history | `nornir bakeoff leaderboard <model\|agent>` |
| raw rows | `nornir bakeoff leaderboard --json` |
| (seed a demo matrix) | `nornir bakeoff demo` |
| (produces the data) | `nornir bakeoff run --task … [--agents …] [--models …] [--mock] [--matrix]` |