<!-- nornir:class: guide -->
# Bake-off **codegen judge** (`CodegenJudge` / `AnswerJudge`)
> Design + guide for the compile-and-test quality scorer that finally fills the
> bake-off's missing `score`. Companion to `.nornir/bakeoff.md` (the H5 agent ×
> model matrix). Also doubles as the **funnel #45** task-decomposition
> accept/reject oracle.
## 1. The problem it solves
A bake-off (`nornir bakeoff run`) records every `(agent, model)` cell's answer
plus its economics — latency, tokens/s, cost, mcp calls — into
`agent_model_runs`. But for a **real ollama run** the quality `score` was left
`0.0` (`OllamaCaller` has no judge), so real leaderboards ranked cells by *speed
alone*: the fastest gibberish beat the slow-but-correct answer. The mock/demo
paths hand-author plausible scores, which never reflect whether the code works.
The **codegen judge** replaces that `0.0` with a score earned by *actually
building and testing the answer*.
## 2. What it is
A **pure-Rust** evaluator — it does **not** call an LLM. It takes an answer that
was *already produced* and grades it:
1. **Copy** the target crate into a temp scratch dir (skipping `target/` +
`.git/` so it's fast + clean).
2. **Apply** the answer: either overwrite a file with it (the answer is a full
file body), or `git apply` it (the answer is a unified diff/patch).
3. **`cargo build`** the scratch crate.
4. Run the **acceptance command** (default `cargo test`; configurable, e.g. a
specific `--test` or a doctest run).
5. Turn the outcome into a `0.0..=1.0` score + a structured [`Verdict`].
### Scoring
| outcome | score |
|-------------------------------------------|-------------------------------|
| answer doesn't apply (bad patch / path) | `0.0` |
| applies but **doesn't compile** | `0.0` |
| compiles, some acceptance tests fail | **fraction passing** (`p/n`) |
| compiles, accept cmd fails, count unknown | `0.5` (compiles-but-fails) |
| compiles, accept cmd passes, no count | `1.0` |
| **build + all acceptance tests pass** | `1.0` |
The fraction comes from summing every `test result: ok. N passed; M failed; …`
line cargo prints (unit + integration + doc suites each emit one).
## 3. The API
```rust
// The trait — one method, never panics on a bad answer (a bad answer is a 0.0
// Verdict, not an Err; Err is reserved for the judge's own infra failing).
pub trait AnswerJudge {
fn judge(&self, answer: &str) -> anyhow::Result<Verdict>;
}
pub struct Verdict {
pub compiled: bool,
pub tests_passed: u32,
pub tests_total: u32,
pub score: f64, // 0.0..=1.0 — becomes the bake-off row's score
pub message: String, // one-line explanation
}
impl Verdict {
pub fn rejected(msg) -> Verdict; // a 0.0 "never built" verdict
pub fn accepted(&self) -> bool; // compiled && all tests passed ← funnel oracle
}
// The first impl: compile-and-test.
pub struct CodegenJudge { /* holds a CodegenSpec */ }
pub struct CodegenSpec {
pub crate_dir: PathBuf, // crate copied into scratch
pub apply: ApplyMode, // WriteFile { rel_path } | ApplyPatch
pub accept_cmd: Vec<String>,// default ["cargo","test"]
}
// constructors:
CodegenSpec::write_file(crate_dir, "src/lib.rs")
CodegenSpec::apply_patch(crate_dir)
.with_accept_cmd(vec!["cargo".into(), "test".into(), "--test".into(), "it".into()])
```
`AnswerJudge` is the seam so the funnel epic reuses the *same* evaluator;
`CodegenJudge` is the first impl, and more (lint judge, benchmark judge, …) can
slot in without touching the bake-off.
## 4. How it wires into the bake-off
`agent_model_runs::score_rows_with_judge(&mut rows, &judge)`:
- For every **OK** row, judge `row.output` and overwrite `row.score` with the
verdict's. Returns one `JudgeReport` per judged cell (for the CLI to print).
- **Failed** rows (`ok == false`, the model never answered) stay `0.0` — there's
nothing to compile.
- A judge **infrastructure** error (e.g. the crate can't be copied) is
non-fatal: the row keeps its prior score, the error rides in the report. The
bake-off never aborts because the judge tripped.
- Pure — no warehouse I/O; the caller persists the re-scored rows. Non-codegen
bake-offs (no spec) skip it entirely and keep whatever score the caller set.
```text
run_bakeoff_matrix → rows (score 0.0)
└─ --judge-crate set?
├─[no]──────────────────────────────────────────┐
└─[yes]─▶ score_rows_with_judge │
CodegenJudge.judge(output) │
└─▶ row.score = verdict.score ──────────┤
▼
persist as-is
└─▶ leaderboard ranks by REAL score, then tok/s
```
## 5. CLI
New flags on `nornir bakeoff run`:
| flag | meaning |
|----------------------------|---------------------------------------------------------------|
| `--judge-crate <PATH>` | target crate; turns on compile-and-test scoring |
| `--judge-file <REL_PATH>` | answer is a full file body that overwrites this path |
| `--judge-patch` | answer is a unified diff, `git apply`-ed (excl. `--judge-file`)|
| `--judge-cmd "<CMD>"` | acceptance command (default `cargo test`) |
```bash
# Score each model's answer by building+testing it against a fixture crate.
nornir bakeoff run --task "Implement add(a,b)" \
--agents local-llm --models llama3,mistral \
--judge-crate ./fixtures/addcrate --judge-file src/lib.rs
# Patch-style answers, gated on one integration test.
nornir bakeoff run --task "Fix the race" --mock \
--judge-crate ./fixtures/racey --judge-patch \
--judge-cmd "cargo test --test concurrency"
```
The run prints each cell's verdict, e.g.:
```
⚖ judging 2 cell(s) by compile-and-test against `./fixtures/addcrate` (cargo test)
local-llm/llama3: score 1.000 — compiled; all 2 acceptance tests passed
local-llm/mistral: score 0.000 — did not compile: error[E0308]: mismatched types
```
…and the leaderboard then ranks by **real score first**, tok/s second.
## 6. Doubling as the funnel #45 oracle
The funnel decomposes an idea into sub-tasks; #45 needs to decide whether a
sub-task's produced code is **accepted or rejected**. That is exactly
`AnswerJudge::judge(produced_code)` → `Verdict::accepted()` (compiled AND all
tests passed). The same `CodegenSpec` (crate + accept cmd) that scores a bake-off
cell is the per-sub-task acceptance gate, so the funnel and the bake-off share
one evaluator and one definition of "the code is correct".
## 7. Tests (inject-assert LAW)
`src/warehouse/codegen_judge.rs` tests feed real answers to a real scratch crate
and assert real scores:
- `known_good_answer_scores_one` — correct `add` body → build+test pass → **1.0**.
- `noncompiling_answer_scores_zero` — type-error body → won't compile → **0.0**.
- `compiling_but_failing_answer_scores_partial` — `add` off by one → 1/2 tests →
**0.5**.
- `patch_that_does_not_apply_scores_zero` — bogus diff → **0.0**.
- plus pure scoring-math + count-parsing unit tests.
`src/warehouse/agent_model_runs.rs` —
`matrix_scored_by_codegen_judge_lands_real_scores`: a `MockCaller` matrix whose
cells emit a GOOD body, a NON-COMPILING body, and one absent (red) cell; run
through `score_rows_with_judge` against a scratch crate. Asserts the rows land
**real, non-0.0** scores (good=1.0, non-compiling=0.0, red stays 0.0/red) and the
leaderboard now ranks the GOOD cell first on its earned score.