# The `kernel` build kind — guide
> **Class:** `guide` · audience: a techie declaring or driving a kernel build.
> The *why* is in `workspace_nordisk/.nornir/kernel-trim-design.md`;
> the field-by-field authority is the module doc of `crates/nornir-build-thing/src/kernel.rs`.
> Sibling: [`kernel-for-idiots.md`](kernel-for-idiots.md).
## What it is
Where an `[[build.iso]]` masters a medium *around* a kernel and a `[[build.vm]]`
*boots* one, a `[[build.kernel]]` is where that kernel **comes from**: fetch the
pinned kernel.org tarball (host-side, pure Rust), merge a committed kconfig
fragment set over `defconfig` or `tinyconfig`, optionally run a SAT pass over the
Kconfig DAG, and drive `make bzImage` inside a bounded toolchain container. Out
come three files in `<repo_root>/.kernel-out/<name>/` — `bzImage`,
`kernel.config`, and `<name>.kernel-pin.json` — plus a fourth,
`deps-<version>.config`, when the minifier ran.
The host contributes exactly two things: a container-engine socket and a
workdir. No host gcc, no host python, no host qemu.
## The four presets — what is IN the kernel
`preset` is the contents axis; `profile` (`vanilla` | `hardened`) is the
*orthogonal* hardening overlay, so every preset × profile pair composes.
| preset | starting point | contents | pick it when |
|---|---|---|---|
| `full` | `defconfig` | the generic-hardware kernel, untouched — **no constellation contract at all** | you will boot this on unknown metal, and you can load modules |
| `normal` | `defconfig` | + the MEASURED boot contract (EFI stub, 8250 serial, devtmpfs, initrd, TUN, virtio blk/pci/net, vfat+cp437, ISO-9660) | the default; what the appliances ship today |
| `slim` | `tinyconfig` | the dependency closure + the contract and nothing else | an appliance that knows its hardware |
| `nuked` | `tinyconfig` | `slim` − gfx/fb/vt/sound/input/USB/wireless/legacy storage | tunnr-class: serial console, virtio-net, TUN, virtio-blk, and that is the whole machine |
### Measured, linux 6.12.104, airgapped
podman rootless on dwarf, 12 cores, `-j8`, 8192 MiB cap, from the cached tarball.
All four presets have now been built **and probe-booted**:
| preset | base | bzImage | wall | PID 1 | console | **tun** | virtio-blk |
|---|---|---|---|---|---|---|---|
| `full` | `defconfig` | **13 693 952 B** | 246 s | alive | present | **ABSENT** | present |
| `normal` | `defconfig` | **13 714 432 B** | 235 s | alive | present | present | present |
| `slim` | `tinyconfig` | **1 811 456 B** | 80 s | alive | present | present | present |
| `nuked` | `tinyconfig` | **1 725 440 B** | 78 s | alive | present | present | present |
| `slim` + minify | `tinyconfig` | **1 389 568 B** | 148 s cold / 103 s warm | alive | present | present | present |
**`full` boots and is not an appliance kernel.** On plain `defconfig`,
`CONFIG_TUN` is not built in — it is a module, because `defconfig` sets
`MODULES=y` — while `VIRTIO_NET`, `VIRTIO_BLK` and `VFAT_FS` happen to be `=y`.
An appliance initramfs carries no modules, so a `full` kernel has no TUN device
and therefore **no userspace-WireGuard data plane**. That is the lesson
`base.config` exists for, and it is now measured rather than argued. (A second
`full` receipt came in at 230 s; the 246 s row is the first.)
The gap between `full` and `normal` is **20 480 bytes** — that is the entire
boot contract, and it buys a machine that can actually run the appliance. The
interesting comparison was never full-vs-normal; it is normal-vs-slim
(**7.6×**), which is what `tinyconfig` buys, and slim-vs-slim+minify
(**−23.3 %**), which is what the solver buys.
`slim` and `nuked` sit 86 016 bytes apart — a real difference in what the
appliance can do at all, which is why a picker cell that shows the *kind* and
hides the *preset* shows the half that decides nothing.
Every contract-carrying arm's built `.config` is asserted to hold
`VIRTIO_BLK=y VIRTIO_NET=y TUN=y VFAT_FS=y`. A small kernel that quietly dropped
virtio is the silent degrade the canary exists to prevent.
## Declaring one
```toml
[[build.kernel]]
name = "korp-kernel-hardened"
version = "6.12.104"
# REQUIRED — a kernel fetched unpinned is refused BY NAME:
tarball_sha256 = "949d166a…" # 64 lowercase hex, from kernel.org's signed sums
preset = "slim" # full | normal (default) | slim | nuked
profile = "hardened" # vanilla (default) | hardened
# base = "tinyconfig" # optional override; unset = whatever the preset implies
# source_url = "…" # unset = the kernel.org CDN derived from `version`
# kconfig_fragment = "kernel/extra.config" # repo-relative, merged LAST (it may veto)
toolchain_image = "" # empty + a digest = the LOCAL image id
toolchain_image_sha256 = "388a7816…" # 64 hex
toolchain_packages = [] # self-contained image ⇒ the build runs --network none
# MINIFY — the SAT pass over the Kconfig DAG (OFF by default):
# solver = "klocalizer" # unset = off, and the build is exactly what it was
# solver_version = "4.10" # the kmax in the image; HALF the formula cache key
# solver_timeout_secs = 1800 # the solver's OWN bound, under timeout_secs
# the compiler axis:
# toolchain = "gcc" # gcc (default) | clang (renders LLVM=1)
# lto = false # full clang LTO; needs toolchain = "clang"
# minimal = false # REFUSED on x86_64 — see "the third knife"
# timeout_secs = 7200 # the build bound; overrun is a loud refusal
# probe_timeout_secs = 120 # a boot that has not printed its marker is a HANG
# mem_limit_mb = 8192 # podman --memory
# cpus = 8 # podman --cpus, AND the `make -j` width
# min_free_disk_mb = 6144 # statvfs floor, checked BEFORE the fetch; 0 disables
# cache_dir = "/big/disk/nordisk-kernel" # tarballs AND kclause formulas live here
```
`kind = "kernel"` on a `[[build.variant]]` row is the equally-legal spelling;
`[[build.kernel]]` is permanent sugar for it, exactly like `iso` / `installer`.
### Every field, its default, its refusal
| field | default | refusal |
|---|---|---|
| `name` | — | empty name |
| `version` | — | must be a dotted release like `"6.12.104"` |
| `tarball_sha256` | — | **REQUIRED**; absent or not 64 lowercase hex → refused by name. "A malformed pin is not a pin" |
| `preset` | `normal` | unknown name refused BY NAME — a typo'd `"slin"` must not silently build the generic kernel that is the opposite of what was asked for |
| `profile` | `vanilla` | unknown name refused BY NAME — a typo'd `"hardned"` degrading to vanilla is a hardening claim the artifact does not honour |
| `base` | the preset's own | only `defconfig` / `tinyconfig` are wired; anything else refused by name |
| `arch` | `x86_64` | only `x86_64` is wired (the bzImage collect path is arch-specific) |
| `make_target` | `bzImage` | only `bzImage` is wired |
| `kconfig_fragment` | none | must be repo-relative (no absolute path, no `..`), and must exist in this checkout |
| `toolchain_image` | `docker.io/library/debian:trixie` | an image reference that resolves to **nothing at all** (empty name, empty digest) is refused — there would be nothing for an engine to run. The *nordisk* image has two extra refusals, below |
| `toolchain_image_sha256` | none | when set, must be 64 lowercase hex |
| `toolchain_packages` | gcc make flex bison bc libelf-dev libssl-dev xz-utils | non-empty on the nordisk image is refused |
| `solver` | unset (**off**) | only `klocalizer` is wired; an unwired name is refused BY NAME — a typo'd minifier that silently built the un-minified kernel is the same silent degrade every other axis refuses |
| `solver_version` | `4.10` | empty is refused (it is half the formula cache key); and it is **checked against what the container reports** — a mismatch refuses and publishes nothing |
| `solver_timeout_secs` | 1800 | `0` refused; and it must be **strictly under** `timeout_secs`, or the bound that is supposed to name the hung step never fires first |
| `toolchain` | `gcc` | only `gcc` / `clang` are wired |
| `lto` | `false` | `true` without `toolchain = "clang"` refused by name — the symbol would not survive `olddefconfig` under gcc and the canary would then refuse the build for a reason unrelated to the declaration |
| `minimal` | `false` | `true` on `x86_64` is a **NAMED REFUSAL** — see "the third knife" |
| `timeout_secs` | 7200 | `0` refused — "a kernel build without a budget is a hang waiting to be discovered" |
| `mem_limit_mb` | 8192 | `0` refused — an unbounded build container is how a 33 GB compile took a box down on 2026-08-20 |
| `cpus` | 8 | `0` refused — `make -j` is derived from this, and a build must not size itself to whichever host runs it |
| `probe_timeout_secs` | 120 | `0` refused (it accepted zero for a day while both its siblings refused it) |
| `min_free_disk_mb` | 6144 | `0` disables the check — and **nothing on the receipt says so**: the pin has no disk field, so a build that skipped the check and one that passed it are indistinguishable afterwards |
| `cache_dir` | the environment resolution | absolute is fine — this is a machine-local cache, not a repo input |
The bounds are real, not decorative: mid-compile the live run measured
**759.82 % CPU** (≈7.6 of the 8-core cap) and **2.508 GB / 8.59 GB**.
## MINIFY — the second knife
`solver = "klocalizer"` runs kmax's SAT solver in the **same** airgapped bounded
container, between the fragment merge and `olddefconfig`, and hands
`olddefconfig` a `.config` whose `depends on` / `select` chains are already
closed. It is **off by default**: without it the build is exactly what it always
was, and the canary is the only oracle.
Measured 2026-08-23, linux 6.12.104, airgapped, `-j8` / 8192 MiB:
| arm | bzImage | wall | note |
|---|---|---|---|
| `slim`, no solver | 1 811 456 B | 80 s | the hand-closed baseline |
| `slim` + `solver` | **1 389 568 B** | 148 s (formula cold), 103 s (warm) | **23.3 % smaller**, byte-identical across two runs, and it BOOTS: PID 1 alive, console + tun + virtio-blk, 2.1 s |
| `tinyconfig` + `normal` + `solver`, hand closure REMOVED | 1 369 088 B | 106 s | canary-clean and a **BRICK** — see below |
### The formula, and why it is cached
The expensive half is reading the kernel's own Kconfig: `kextract` + `kclause`
turn the tree into a z3 formula over **17 704 symbols** — **19 949 986 bytes**,
about **46 s**. That is content-addressed in the *same* shared cache the tarball
uses:
```
<cache_dir>/kclause/<tarball_sha256>.kmax-<solver_version>/
```
Both halves of that key are load-bearing. A formula is a function of the tree
*and* of the extractor that produced it, so keying on the tarball alone would let
two kmax releases share one entry — a cache that lies is worse than no cache,
because the build believes it. The host stages a hit into the workdir, and
publishes a miss back through a `.part` directory and a rename, exactly as the
tarball cache does.
### The closed world — get this wrong and the "minifier" makes it BIGGER
klocalizer's `--repair` reads a `.config` as **soft** assumptions and drops only
what an unsat core forces it to. Everything the `.config` never mentions is
*free*, and free is how drivers walk in. So the match config handed to the
solver is the merged `.config` **closed**: every symbol the tree knows and the
config never names is written down as `is not set`.
Measured, and the reason this paragraph exists: **78 added symbols closed,
against 472 open** — the open run full of mlx5, i40e and reiserfs that nothing
asked for. A naively-wired minifier is not a smaller kernel, it is a bigger one.
### The write-back — the closure is a file you commit
After `olddefconfig` and *before* the canary, whatever the solver added, minus
what the fragment already named, is written to `deps-<version>.config` beside the
bzImage and sealed as a fourth artifact. That is the automation of
`src/kconfig/deps-tinyconfig.config`, whose header describes the same loop done
by hand. Read it, and commit it if you want the closure pinned instead of solved
every time. A solver that ran but left no readable closure refuses the build:
a minification nobody can review is not a minification.
### Its own bound, its own refusal
Both klocalizer calls are wrapped in `timeout`. An overrun prints
`KERNEL-SOLVER-REFUSED` and exits **91** — deliberately not the canary's 90 — so
the orchestrator can say which step refused, name the knob that bounds it
(`solver_timeout_secs`) and the budget it blew. Proven live: a 5 s budget refused
in 19 s naming the variant, the step, the knob and the number. A timeout (`124`)
and any other non-zero are reported as **different facts**: "did not answer in
1800 s" and "answered UNSAT in four seconds" must not arrive as the same
sentence, and an UNSAT means the fragment asks for symbols that cannot all be on
together on this arch.
### The limit — a solver is a knife, not an oracle
This is the most important paragraph on this page. With the hand-written closure
**removed**, klocalizer found **13 of its 16 lines by itself** and missed
`CONFIG_BINFMT_ELF`, `CONFIG_MULTIUSER` and `CONFIG_UNIX` — precisely the three
the canary cannot catch, **for the identical reason**: nothing in the fragment
names them. That kernel was canary-clean, 1 369 088 bytes, and a brick; the
probe refused it with *"did NOT reach PID 1"*.
> **MINIFY makes a declared contract minimal. It does not discover an undeclared
> one.** Probe-boot is still the only thing that says a kernel runs.
## The third knife — measured, and mostly falsified
The design named three knives: where you start, which features exist, and dead
code inside what is left. On `x86_64` the third one **does not exist**:
* `minimal = true` is a **named refusal** on this arch.
`CONFIG_LD_DEAD_CODE_DATA_ELIMINATION` is gated on
`HAVE_LD_DEAD_CODE_DATA_ELIMINATION`, which only **arm, mips, riscv and
powerpc** select. The first live attempt was refused by the canary; adding its
`depends on EXPERT` parent got it refused again, because the parent was never
the problem. A flag that silently does nothing is worse than one that refuses —
it reads on the receipt as a knife that was used. The near-empty fragment is
kept for the day this kind wires an arch that has the option.
* `CONFIG_TRIM_UNUSED_KSYMS` is declared **inside `if MODULES`**, so a
`MODULES=n` kernel cannot have it — and every preset it would help most is
`MODULES=n`.
* clang LTO works and is the wrong tool for size. Same declaration, same
container, `slim`:
| toolchain | bzImage | wall | vs gcc |
|---|---|---|---|
| gcc | 1 811 456 B | 80 s | — |
| clang, `LTO_CLANG_FULL` | **2 016 256 B** | 84 s | **+204 800 B, +11.3 %** |
It boots cleanly (PID 1, console + tun + virtio-blk, 2.4 s). Full LTO inlines
across translation units, and on a kernel this small the inlining costs more
bytes than the dead code it removes. **LTO is a speed/codegen knife, not a size
knife.** It stays opt-in and off by default.
**On x86_64 there are two knives, not three:** `tinyconfig` and minify.
## The toolchain image
`apt-get`ing a build environment at build time needs a network, drifts, and
cannot be pinned. So the kernel kind ships the **text** of one image —
`src/containerfiles/kernel-toolchain.Containerfile` — and lets the ordinary
container kind build it. Every package in it names the step that needs it:
gcc/make/flex/bison/bc/libelf/openssl/pahole (compile), clang+lld+llvm (the LTO
axis), python3 + python3-dev + kmax 4.10 in a venv (minify), qemu-system-x86 +
cpio + xorriso (probe-boot).
Two named refusals apply to it and to no other image, because it is one **we**
publish:
* it must be pinned (`toolchain_image_sha256`), and
* it must declare `toolchain_packages = []`, so the build runs airgapped.
### ⚠ `podman image prune` DESTROYS these images, and no pull brings them back
`nordisk-kernel-toolchain` and `nordisk-cargo-toolchain` are built HERE and have
**no registry behind them**. MEASURED on oden 2026-09-05: `podman pull
nordisk-cargo-toolchain:live` resolves to `docker.io/library/…` and dies on
*"requested access to the resource is denied"*, and the kernel name resolves to
`localhost/…` and dies on *"pinging container registry localhost: connection
refused"*. Neither is fetchable; the only source is a re-mint.
And `podman image prune -f` removes **unused-but-tagged** images, not only
dangling ones: MEASURED 2026-09-05, one `podman container prune -f` followed by
one `podman image prune -f` took the image count **107 → 9** and took both
toolchain images with it. Pruning them is destructive and is recovered only by
re-minting (`toolchain_images_live.rs`, ~45 s each). The kernel image re-mints
to the SAME digest (`78eaa55a651d…`) because its recipe is pinned; the cargo
image does not — a re-mint gives a NEW digest and every `image_sha256` pin in
the constellation then names an image that no longer exists. Prune by name
(`podman image rm <name>`) or with `--filter dangling=true`.
Build it, in Rust, through the path that already exists:
```rust
use nornir_build_thing::kernel::{
kernel_toolchain_container, materialize_kernel_toolchain_containerfile,
};
// 1. write the committed text where your build context lives:
let cf = materialize_kernel_toolchain_containerfile(&repo_root.join("containerfiles"))?;
// 2. build it as an ORDINARY image cell (no second builder, no second engine):
let image = kernel_toolchain_container("containerfiles/kernel-toolchain.Containerfile");
let built = builder.build(&image, &repo_root)?; // BuiltImage carries the digest
```
### Pinning it — local vs published
Podman reads `name@sha256:<digest>` as a **registry** reference and tries to
pull it, so `localhost/nordisk-kernel-toolchain@sha256:…` dies on *"pinging
container registry localhost: connection refused"*. A locally-built image is
content-addressable only by its **image ID**. So:
* **not yet published** → leave `toolchain_image` **empty** and put the 64-hex
local image id in `toolchain_image_sha256`. That is both a legal `podman run`
argument and a genuine content address.
* **published** → the ordinary `<name>@sha256:<digest>` spelling, which becomes
correct the moment the image is in a registry.
Proven live: the `slim` arm rebuilt with `image=388a7816cde2…`, no name at all,
airgapped, off the cached tarball — 75 s, bzImage **1 811 456 bytes**,
byte-identical in size to the tag-addressed run.
## The shared cache
A kernel tarball is ~150 MB (measured: **148 400 064 B** for 6.12.104), and a
kclause formula is another **19 949 986 B**. Both live in one shared,
content-addressed cache. Resolution order:
1. `$NORDISK_KERNEL_CACHE`
2. `$XDG_CACHE_HOME/nordisk/kernel`
3. `$HOME/.cache/nordisk/kernel`
4. `<repo_root>/.nornir-cache/kernel` (a machine with no home at all, e.g. CI)
…or the declared `cache_dir`, which overrides all four — that is how you put it
on the big disk.
```
<cache>/tarballs/<sha256>.tar.xz
<cache>/kclause/<sha256>.kmax-<solver_version>/
```
Tarballs are **keyed by the pin itself**, so a stale entry is not expressible: a
file whose bytes do not hash to its own name is not a hit, it is garbage that
gets re-fetched. A download lands on a pid-suffixed `.part` inside the cache and
is **renamed** on success, so a half-download can never look like an entry to a
concurrent build; a formula is published the same way, through a `.part`
directory. Entries are hard-linked into the workdir (free), or copied where the
filesystem will not link.
## The disk floor
`min_free_disk_mb` (default 6144) is checked with `statvfs` **before the fetch**,
and refuses by name with both numbers. The floor is measured, not feared: a
`normal` build's workdir peaks at **2.5 GB**, a `slim` one at **1.9 GB**, plus
the 148 MB cached tarball. 6 GiB is that with room for a second variant.
A build that dies at 95 % because the filesystem filled has spent the whole
compile to tell you something `statvfs` knew before it started. Note the honest
half: setting the floor to `0`, or running without the `kernel-build` feature
that can call `statvfs` at all, leaves **no trace on the receipt** — a skipped
check and a passed one look identical afterwards.
## Refused BEFORE the container: a fragment that contradicts itself (lane AP, 2026-08-30)
`KERNEL-CONFIG-REFUSED` (below) is the canary INSIDE the container: it fires after the
tarball is fetched, the toolchain image pulled and `olddefconfig` run. One shape of that
refusal never needed the container at all — a `=n` that legally narrows a parent, leaving a
`=y` on its child dangling:
```
kernel 'tunnr-kernel': the merged fragment contradicts itself in 1 place(s) — refused BEFORE
the container runs:
- CONFIG_SERIAL_8250_CONSOLE=y cannot hold: it `depends on SERIAL_8250 = y`, and this same
merged fragment says # CONFIG_SERIAL_8250 is not set. `olddefconfig` will DROP it and the
build's canary will then refuse by name, after the container has run. Narrow the child
with its parent, or stop declaring it
```
This is lane AN's 2026-08-30 canary bug, and UpCloud is such a machine (`consoles = [vga]`).
The check reads the donor's own Kconfig tree, so it needs the source EXTRACTED — that means
every re-build and every ISO compose takes this door, and a cold first build still leans on
the canary. It reads the fragment as an OPEN world: a symbol the fragment does not mention
is *unknown*, never *off*, so only a contradiction the fragment states about itself is
reported. Zero false positives over the shipped fragment's 84 declarations.
## The kernel workdir now keeps one more file
`<donor>/.kernel-out/<kernel>/kernel.base.config` — the starting `.config`
(`tinyconfig`/`defconfig`) as the container saw it, before the fragment was merged. It is
the kconfig solver's `base` input: without it, "this symbol is residue from the preset" and
"something pulled it in" cannot be told apart. A kernel built before 2026-08-30 has none and
its `kernel.why.toml` rows simply stay unsolved — a description must never be able to veto
the thing it describes.
## Reading a `KERNEL-CONFIG-REFUSED` list
Inside the container, after `merge_config.sh -m` + `olddefconfig`, a canary loop
checks every line of the merged fragment: each `CONFIG_X=y` must be in the final
`.config`, and each `# CONFIG_X is not set` must not have crept back. It counts
every failure, prints them all, and exits **90** once:
```
KERNEL-CONFIG-REFUSED: CONFIG_EFI_STUB=y did not survive olddefconfig
KERNEL-CONFIG-REFUSED: CONFIG_TUN=y did not survive olddefconfig
…
KERNEL-CONFIG-REFUSED-TOTAL: 12 option(s) did not hold — the list above IS the
dependency closure this fragment still owes
```
**That list is not an error report, it is a work list.** An option "did not
survive" because its *parent* symbol is off, so `olddefconfig` dropped it. For
each name, add the parent that enables it and run again — or turn the solver on
and let it close the chain for you.
This is exactly how `src/kconfig/deps-tinyconfig.config` was written. The first
`slim` run refused in **18 s** naming `CONFIG_EFI_STUB` and nothing else (the
loop exited on the first bad line back then); the fixed loop returned all twelve
at once:
```
EFI_STUB · SERIAL_8250 · SERIAL_8250_CONSOLE · TUN · VIRTIO · VIRTIO_PCI
VIRTIO_BLK · VIRTIO_NET · VFAT_FS · ISO9660_FS · NLS_CODEPAGE_437 · NLS_ISO8859_1
```
Their parents — PCI, ACPI+EFI, TTY, NET/INET/NETDEVICES, BLOCK, VIRTIO_MENU,
FAT_FS, NLS — became that file, and the next run was green. Two turns of 18 s,
not one container run per symbol.
**The canary's limit, stated honestly:** it only checks what the fragment
*names*. `CONFIG_BINFMT_ELF` — without which the kernel cannot exec PID 1 at all
— can never be caught by it, and the solver misses it for the same reason. That
gap is what the probe exists to fill.
## The probe boot
`probe_kernel_with` boots the **collected** `bzImage` under QEMU *inside the
toolchain image*, with a PID 1 that is a static C binary compiled by that
image's own gcc (not a shell script: there is no shell in this initramfs, and
one there would test the shell instead of the kernel's ability to exec an ELF).
PID 1 prints its marker, mounts devtmpfs, NAMES each contract device `present`
or `ABSENT`, then powers the guest off. `-no-reboot` + `panic=1`, so a broken
kernel *ends* — a hang and a panic are different facts and must not arrive as
the same one.
Wall times: `normal` 4.2 s, `slim` 2.1 s, `nuked` 2.1 s, `slim`+minify 2.1 s,
clang-LTO `slim` 2.4 s. The device columns are in the matrix at the top of this
page.
`full`'s `("tun", false)` is the clearest thing the probe has produced: it came
from PID 1 calling `stat()` **inside the booted guest**, not from reading a
`.config`. A probe that printed only what it found would have reported a clean
boot for a kernel that cannot carry the constellation's data plane.
`KernelProbe` is a **value**, not a verdict: the caller decides what a missing
device means, because a preset that never declared virtio-blk is *supposed* to
have no `/dev/vda`.
The probe refuses on a non-zero container exit **before reading a single
marker**, and says so. That is not caution — the first live run reported
`booted: true` for a kernel that was never started, because `gcc` failed on the
PID 1 source and gcc's diagnostics *quote the offending line*, which contains
the marker strings. A transcript is evidence only if the step that produced it
succeeded.
## The receipt
`<name>.kernel-pin.json` carries every sha256 that identifies the build — the
tarball, the merged fragment, the `bzImage`, the `.config` — *and*, since
2026-08-23, the axes that decide which kernel it is: `preset`, the resolved
`base`, `toolchain`, `lto`, `solver` and `solver_version` beside the older
`profile`. Before that an auditor holding a pin could not tell a 13.7 MB kernel
from a 1.7 MB one. The same axes are in the content id, so two presets can no
longer share one content address; the *run* knobs (cpus, memory, disk floor,
cache dir) are deliberately kept out of it, because they change how a build is
run and not what it produces.
## Running the heavy live tests
Both are `#[ignore]`d and feature-gated: they never ride a default `cargo test`.
Missing env or a missing socket makes them **SKIP loudly**, never fake a green.
```bash
# ── the build (real download + real compile) ─────────────────────────────────
export KERNEL_LIVE_VERSION=6.12.104
export KERNEL_LIVE_SHA256=<64-hex from https://cdn.kernel.org/pub/linux/kernel/v6.x/sha256sums.asc>
export KERNEL_LIVE_PRESET=slim # full|normal|slim|nuked (default normal)
export KERNEL_LIVE_PROFILE=vanilla # vanilla|hardened (default vanilla)
export KERNEL_LIVE_BASE=tinyconfig # optional explicit override
export KERNEL_LIVE_NAME=e2e-slim # default live-vanilla
export KERNEL_LIVE_TOOLCHAIN="" # empty name + the id below = the LOCAL image
export KERNEL_LIVE_TOOLCHAIN_SHA256=388a7816…
export KERNEL_LIVE_TOOLCHAIN_KIND=clang # the COMPILER axis (gcc default)
export KERNEL_LIVE_LTO=1 # set at all = lto = true (needs clang)
export KERNEL_LIVE_MINIMAL=1 # set at all = minimal = true (refused on x86_64)
export KERNEL_LIVE_SOLVER=klocalizer # MINIFY on; unset = off
export KERNEL_LIVE_SOLVER_TIMEOUT=1800
export KERNEL_LIVE_SOLVER_VERSION=4.10
export KERNEL_LIVE_CACHE=/big/disk/nordisk-kernel # tarballs AND kclause formulas
export KERNEL_LIVE_CPUS=8
export KERNEL_LIVE_MEM_MB=8192
export KERNEL_LIVE_KEEP=/big/disk/kernel-e2e # keep the workdir instead of a tempdir
cargo test -p nornir-build-thing --features kernel-build \
--test kernel_build_live -- --ignored --nocapture
# ── the probe (boots what the build left on disk) ────────────────────────────
export KERNEL_PROBE_ROOT=/big/disk/kernel-e2e # the repo_root that holds .kernel-out/
export KERNEL_PROBE_NAME=e2e-slim
export KERNEL_PROBE_TOOLCHAIN=localhost/nordisk-kernel-toolchain:latest
export KERNEL_PROBE_TOOLCHAIN_SHA256=388a7816… # optional
cargo test -p nornir-build-thing --features kernel-build \
--test kernel_probe_live -- --ignored --nocapture
```
Naming a `KERNEL_LIVE_TOOLCHAIN` also clears `toolchain_packages`, so the run is
airgapped — the live run reports `network=false`. The tarball pin is deliberately
**not** baked into the test file: a committed hash would rot the day a mirror
re-rolls, and a test that fetches whatever the network answers is exactly what
this kind refuses by design.
The `#[ignore]` message still says "30+ min". That estimate was written from
fear; the measured wall is 235 s for `normal` on this hardware with a warm cache.
## The pane
A `[[build.kernel]]` row is reachable from dwarves: the matrix cell shows
preset · version · resolved base · profile, and a **Kernel tab** mounts a
rotating 3D feature graph over the presets' committed fragments, with a 14-step
scenario in the tab library (an unreachable pane is a defect this constellation
has already paid for once). Rotation re-projects a cached layout — the build
counter is asserted flat across spin frames and to rise by exactly one when the
preset genuinely changes.
One honest limit is written on the pane itself: its edges are the **name
hierarchy** at an underscore boundary (`VIRTIO → VIRTIO_PCI`), not the real
`depends on` / `select` DAG, which lives in the tarball. A kernel cell
deliberately opens **no run pane** — a bare `bzImage` boots nothing without a
medium or a rootfs around it.
The second limit this paragraph carried until 2026-08-27 — *"the `zero-hits`
node state needs the RUN loop, which is not built, so it is absent rather than
faked"* — **stopped being true on 2026-08-23**, with the RUN loop (below) and
`dwarves-gui/tests/robot_kernel_scene.rs::a_run_report_reaches_the_graph_and_zero_hits_are_not_absence`,
which drives the report onto the graph and holds the "absent ≠ zero" half red:
with **no** report there are no drop candidates, because absence of evidence is
never painted as evidence of absence.
## An OFFLINE kernel build — BUILT 2026-08-30 (lane AO · #14; spec by lane AH)
**Answer (b) below is shipped.** `nornir_build_thing::toolchain_archive` stages the
toolchain image as a file in the same content-addressed cache as the kernel tarball,
verifies it before a container is created, and refuses BY NAME rather than letting a
missing toolchain become a registry fetch. Jump to **"(b), as built"** for the
procedure and the refusals; the spec below is kept because its reasoning is what
decided the shape, and because **one of its claims turned out to be false** — see the
correction.
### What is ALREADY offline, measured
| half | state | why |
|---|---|---|
| the **compile** | **offline today** | the build container runs with `network = false` whenever `toolchain_packages` is empty (`KernelBuild::needs_network`), and `nordisk-kernel-toolchain` carries gcc/make/flex/bison/bc/libelf/libssl already. No package index, no DNS |
| the **tarball** | **offline once warmed** | it is content-addressed: `<cache_dir>/tarballs/<tarball_sha256>.tar.xz`, verified by hash before use, and a workdir hit skips the cache too. Drop the file in by hand and `cdn.kernel.org` is never reached |
| the **kconfig solver formula** | **offline once warmed** | same shape — `<cache_dir>/kclause/<tarball_sha256>.kmax-<solver>` |
So two of the three fetches are already solved by a content-addressed cache that a
courier can fill. **The whole of the remaining gap is the toolchain image**, plus the
OCI runtime under it.
### The gap, precisely
1. **The image is per-box.** `toolchain_image_sha256` is the MANIFEST DIGEST of an
image built *locally* from this crate's embedded `kernel-toolchain.Containerfile`,
so the committed pin is true only on the box that minted it
(see `KernelBuild::toolchain_repin_hint`). ⚠ **Retired 2026-08-30 (lane AV)** —
this was measured (lane AT: six mints, six ids) and then FIXED; the recipe is
reproducible now, so the pin is a property of it. Kept because the shape of the
offline answer below was decided while it was true, and because the archive
courier is still the right answer for a box with no route.
2. **Minting it needs the network anyway** — the Containerfile pulls `debian:trixie`
from a registry and `apt-get`s its compilers. So "build the image locally" is not
an offline answer; it is the online step, moved.
3. **podman must exist.** `draupnir`'s container backend is the only runner
`KernelContainerRunner` has.
### The three candidate answers, and the verdict on each
**(a) A vendored toolchain TARBALL** — ship the compiler set as a content-addressed
archive beside the kernel tarball, unpack it into the workdir, and run `make` against
it. *Rejected as the primary answer.* It replaces a reproducible image with a second,
unpinned way to get a compiler; it must then solve dynamic linking against whatever
libc the host has, which is the problem containers exist to avoid; and it does not
remove podman, because the build is still bounded (`mem_limit_mb`, `cpus`) and an
unbounded kernel build **took this box down at 33 GB on 2026-08-20**. Worth having
only as the `no-OCI-runtime-at-all` fallback, and then it needs its own `--bound`
mechanism (systemd scope or a cgroup written directly), which is new code.
**(b) An OCI ARCHIVE of the toolchain image** — `podman save`/`skopeo copy` the image
to a `docker-archive` or an OCI layout, content-addressed by the same 64 hex
characters, kept in the same cache as the tarball, and `podman load`ed when the
digest is absent. ***This is the honest fix, and it is what was built*** (see
below). It costs no new build path (the same
container, the same bounds, the same `network = false`), it makes the committed pin
true on every box because the *bytes* travel rather than the recipe, and it is the
same courier a `<cache_dir>/tarballs/` entry already uses. Concretely:
`<cache_dir>/images/<toolchain_image_sha256>.oci.tar`, a preflight that loads it when
`podman image exists` says no, and a refusal naming that exact path when neither the
image nor the archive is there. Publishing the image to a registry is the *same*
change seen from the other side and should land with it: a registry is the online
courier, the archive is the offline one, and both address it by the digest that is
already declared.
**(c) A pre-built kernel ARTIFACT in the ledger** — the `[[build.kernel]]` output
(`bzImage` + `kernel.config` + `kernel.why.toml` + the pin json) stored under
`DWARVES_HOLGER_DIR` and *consumed* instead of rebuilt. Half of this already exists —
`dwarves buildthing --installer` says so out loud when it is unset:
```text
[dwarves] kernel ledger: no artifact store configured (DWARVES_HOLGER_DIR unset …)
— a built kernel is read from the donor's .kernel-out/
```
*Complementary, not a substitute.* It is the right answer for a machine that must
**consume** a kernel offline (a CI runner composing an ISO, an appliance factory), and
it is strictly cheaper than (b) there. It is the wrong answer for a machine that must
**produce** one, and the ledger key must be the merged **fragment** hash and not the
preset name — the compose path already refuses on exactly that key
(*"the pin was built from preset 'full' (fragment e3b0c44…), the choice here merges to
preset 'slim' (fragment f190ed5…)"*), so a ledger keyed any other way would serve the
wrong kernel silently.
### The order to build it in
1. **(b) first** — the OCI archive plus the `podman image exists` preflight. It is the
only item that turns "cannot build at all" into "builds", and it deletes the per-box
pin problem as a side effect.
2. **(c) next** — the ledger read path keyed by fragment hash, for consumers.
3. **(a) only if** a target genuinely has no OCI runtime, and then with its own bound.
## (b), as BUILT — the toolchain courier (lane AO, 2026-08-30)
`crates/nornir-build-thing/src/toolchain_archive.rs`. Wired into
`build_kernel_with_reserved` as step **2c**, immediately before the run request is
composed, so nothing about the container changes: same bounds, same
`network = false`, same script.
### The cache gains a third drawer
```text
<cache_dir>/tarballs/<tarball_sha256>.tar.xz the kernel source (since 08-22)
<cache_dir>/kclause/<tarball_sha256>.kmax-<v>/ the solver formula (since 08-23)
<cache_dir>/images/<toolchain_image_sha256>.tar the TOOLCHAIN (08-30, this)
```
Same courier, same rule, same shape: a file whose content does not match its own
name is not a hit, it is garbage.
### ⚠ The correction to the spec — MEASURED, oden 2026-08-30
The spec said (b) *"makes the committed pin true on every box because the bytes
travel rather than the recipe"*. **That is false for the pin the format declares.**
`podman save` + `podman load` of `nordisk-kernel-toolchain:live`, into a separate
`--root` store on the same box:
| format | image id (config digest) | manifest digest (`.Digest`) |
|---|---|---|
| on the origin box | `ce66a0100cb8…` | `f573ca0a86c0…` |
| **docker-archive** round trip | `ce66a0100cb8…` **SAME** | `5e628ed94b35…` **CHANGED** |
| **oci-archive** round trip | `ac4606f05153…` **CHANGED** | `85189a0753c9…` **CHANGED** |
and asked of the receiving store directly:
```text
ce66a0100cb8… EXISTS (the bare image id)
sha256:ce66a0100cb8… EXISTS
nordisk-kernel-toolchain@sha256:f573ca0a… ABSENT <- the COMMITTED pin
nordisk-kernel-toolchain@sha256:5e628ed9… EXISTS (the RECEIVER's own digest)
```
Podman re-synthesises the manifest on load, so a **manifest digest describes the
copy, not the original**, and the committed pin names nothing on the machine the
archive was carried to. The **image id — the config digest — survives**, and is a
legal reference (`podman run <id>`). Hence two deliberate departures:
1. **`docker-archive`, not `oci-archive`.** The OCI conversion rewrites the config
too, so it preserves *neither* identity — it is the one format that makes the
archive unverifiable against anything the recipe can hold. It is also the format
draupnir's `export_image`/`import_image` already speak, so no second way to move
an image was added (LAW 5).
2. **An archive is verified against the IMAGE ID.** A declaration still pinning a
per-box manifest digest is refused against a staged archive, by name, with the id
it should carry instead. This is the third of the three fixes the row on publishing
the image already named ("publish it", "pin its inputs") — and the only one that
needs no registry. **Every `[[build.kernel]]` in the constellation is still on a
manifest digest**, so every one of them must be re-pinned to its id before it can
use an archive; the refusal spells out the value and both files it goes in.
### The procedure
On a box that has the image:
```bash
ID=$(podman image inspect --format '{{.Id}}' nordisk-kernel-toolchain)
podman save --format docker-archive -o <cache>/images/$ID.tar nordisk-kernel-toolchain
# or, through the door, so no shell is involved:
# nornir_build_thing::toolchain_archive::stage_toolchain_archive(
# &DraupnirKernelRunner::new(), "nordisk-kernel-toolchain", cache_dir)
```
Copy that one file to `<cache>/images/<ID>.tar` on the offline box, and set
`toolchain_image_sha256 = "<ID>"`. The staging door files the archive under the id it
actually exported — never under the name the caller passed, which is how an archive
ends up filed as something it is not.
### The five rows, and the three named refusals
| the box has | what happens | `toolchain_source` on the pin |
|---|---|---|
| the pinned image already | nothing is staged or fetched | `already-present` |
| an archive whose id == the pin | inspected (no daemon), loaded, run **by id** | `staged-archive` |
| an archive holding another image | `NORDISK-TOOLCHAIN-ARCHIVE-MISMATCH` — refused, **never loaded** | — |
| neither, and a pin is declared | `NORDISK-TOOLCHAIN-MISSING` — refused, naming the exact path | — |
| neither, and NO pin | proceeds; an unpinned image is an ONLINE declaration | `unpinned` |
| the runner has no image store | proceeds, checks nothing, **claims nothing** | `unchecked` |
`NORDISK-TOOLCHAIN-LOAD-UNCONFIRMED` is the third: an engine that answers "loaded"
and then answers "no such image" is two claims disagreeing, and only the second one
is the one the build needs. The confirmation goes through a **different door** than
the load.
`KernelPin` now records `toolchain_source` and `toolchain_image_id`, because
"which toolchain this build claims" and "whether that claim was ever checked" were
the same field, and `unchecked` and `staged-archive` read identically on a receipt
without it.
### How the airgap was PROVEN — and what the proof covers
`tests/kernel_offline_live.rs`, `#[ignore]`d. Not "we did not observe a fetch": the
**same build runs twice under the same denial**, and the only difference is whether
the archive is on the disk. One arm must fail or the other proves nothing.
The denial has to reach the container ENGINE and not merely the test process — a
podman socket is a filesystem path, so a test inside a network namespace still talks
to a service that may have a fine route to a registry. So:
```bash
# a podman with NO route, NO addresses, NO DNS, and an EMPTY store:
sudo unshare --net -- setpriv --reuid=$(id -u) --regid=$(id -g) --init-groups podman --root $D/freshroot --runroot $D/freshrun system service --time 0 unix://$D/sock/podman.sock &
# prove the denial before trusting it — this MUST fail:
podman --url unix://$D/sock/podman.sock pull docker.io/library/alpine:latest
# -> dial tcp: lookup registry-1.docker.io: Temporary failure in name resolution
```
(`unshare -rn` without sudo does not work on oden: `apparmor_restrict_unprivileged_userns=1`
refuses the uid map. `sudo unshare --net` + `setpriv` back to the user is the shape
that does.)
**Measured 2026-08-30, oden, linux 6.12.104 `nuked`, fresh empty store, no network:**
* **RED** — archive moved aside: `NORDISK-TOOLCHAIN-MISSING`, naming
`<cache>/images/ce66a0100cb8….tar`; no `bzImage` produced.
* **GREEN** — archive put back, byte-identical call: the 1.16 GB image loaded into the
empty store from the file, the container ran, a `bzImage` came out, and the pin says
`toolchain_source = "staged-archive"`, `toolchain_image = "sha256:ce66a0100cb8…"`.
**And through the CLI, the same two arms.** The test above drives `build_kernel_with`
directly — the same function `dwarves`' `assemble_cell` calls — so the verb itself was
driven too, against a scratch product fixture (no committed recipe was edited) whose
`toolchain_image_sha256` is the image ID:
```text
$ dwarves buildthing aokernel --variant offline-kernel # DOCKER_HOST -> the denied,
# empty store
RED offline-kernel [dwarves-buildthing-7] — could not start: kernel 'offline-kernel'
FAILED and produced nothing: compile kernel: NORDISK-TOOLCHAIN-MISSING: …
exit 1
GREEN BUILT 'offline-kernel' in 90.2s — 4 artifact(s)
bzImage 2 057 216 bytes · kernel.config · kernel.why.toml · the pin, which says
"toolchain_image": "sha256:ce66a0100cb8…"
"toolchain_source": "staged-archive"
"toolchain_image_id": "ce66a0100cb8…"
```
⚠ The GREEN run's process exit is still **1**, and that is NOT the kernel: the scratch
fixture declares no cargo binaries, so `buildgate` refuses a bin tally whose denominator
is zero (*"a gate whose denominator is zero passes vacuously, which is the shape it exists
to catch"*). The kernel cell itself reports `BUILT … 4 artifact(s)`. Said out loud here
because "the CLI exited 1" and "the kernel did not build" are exactly the confusion this
whole item exists to not have.
**The honest limits.** The denial is a network namespace plus an empty store. It does
not simulate a box with no podman at all — that is (a), deliberately not built — and
it says nothing about a declaration with `toolchain_packages`, which asks for a package
index and reports so through `needs_network()`.
### Still open after this
* **(c), the kernel ledger read path** keyed by the merged FRAGMENT hash — for boxes
that must *consume* a kernel offline rather than produce one. Unchanged by this work.
* **(a), a vendored toolchain tarball** — still only for a target with no OCI runtime,
and still needs its own bound before it may exist.
* **Re-pinning the constellation** from manifest digests to image ids. Nothing is
broken today (the `already-present` row covers the box that minted the pin), but
no declaration can use an archive until its pin is an id. **Since lane AV (below)
this is a mechanical edit rather than a decision** — the recipe is reproducible, so
the number is derivable by anyone with the recipe and a podman.
* **Publishing the image to a registry** remains the answer for the ONLINE half. The
archive is the offline courier; a registry is the online one; both address the image
by a digest that is declared. They are the same change seen from two sides. It is a
*convenience* now, not a correctness fix: a box with a route can mint the pinned
image itself and get the pinned number.
## The image is REPRODUCIBLE — MEASURED 2026-08-30 (lane AV)
The whole of the section above rests on a claim that was true when it was written and
is not any more: *"the committed pin is true on the box that minted it and false
everywhere else"*. Lane AT measured the claim (six mints, six different image ids) and
named the fix without measuring it; lane AV built and measured the fix.
```text
image id 3f632569f1980e96744eef4ab258341fc77c55cb17c2dabe79b243e00d6e0825
manifest digest 78eaa55a651d5bd76aae36db0f3307e267a92d2a547ee338a17822a103e0320e
```
**Four mints, four fresh empty `--root` stores, spread over fifteen minutes, one
number** — and the same number again from the bollard door (below). Before: six mints,
six ids.
### Two axes, and neither one alone is worth anything
| axis | what moves | closed by |
|---|---|---|
| the **world** | `FROM debian:trixie` is a TAG (base layer `102f96b3f4ce…` on 08-27, `31631a95db7c…` on 08-30); `apt-get install` takes today's mirror; `pip install kmax` takes today's PyPI | base by `@sha256:` digest · apt from `snapshot.debian.org/…/20260713T000000Z`, the snapshot the base image itself records in its own sources file · every pip requirement at an exact version |
| the **clock** | file mtimes, PEP 552 `.pyc` headers, apt/dpkg/alternatives logs, ldconfig's aux-cache, the random `/tmp/pip-install-XXXXXXXX/` directory pip compiles C extensions in | `ENV SOURCE_DATE_EPOCH` (CPython then emits *hash-based* pycs) · the logs and caches deleted **in the layer that makes them** · kmax's sdist unpacked to a fixed path and built in-tree with `-g0` and `--no-build-isolation` · the builder handed the same instant |
AT's correction stands and is worth repeating: **pinning the inputs is not enough**,
because an image id is a digest over both axes. So is the converse.
### Every knob is load-bearing, and each was removed on its own
Two mints per row, fresh empty stores, everything else held:
| removed | mint 1 | mint 2 | |
|---|---|---|---|
| nothing (control ×4) | `3f632569f198…` | `3f632569f198…` | **SAME** |
| `--timestamp` | `359d4c48b6f8…` | `8086d7ee318d…` | DIFFERENT |
| `ENV SOURCE_DATE_EPOCH` | `30af41ac7b5c…` | `6e03374daf31…` | DIFFERENT |
| the log + aux-cache scrub | `90342d2b0188…` | `d750f1a0d167…` | DIFFERENT |
| kmax's fixed build dir + `-g0` | `730bd56428b6…` | `36545c9ec46c…` | DIFFERENT |
### The half a Containerfile cannot reach
An image id is the digest of the image **config**, and the config carries a `created`
timestamp plus one per history entry. Nothing in Dockerfile syntax touches them — they
are the builder's, written at commit time. So the instant has to reach TWO places, and
exactly one of them may be the writer of it:
* **inside** the image, as `ENV SOURCE_DATE_EPOCH=1783900800`, and
* **on the builder**, as `podman build --timestamp 1783900800`.
The Containerfile is the writer; `KernelBuild::toolchain_source_date_epoch()` /
`kernel_toolchain_source_date_epoch()` parse the number back out of that very line, and
`DraupnirKernelRunner::build_toolchain_image` hands it to the builder. One fact, one
spelling (LAW 5).
### The bollard door, which nobody had measured
AT's mints ran through the `podman build` CLI. **Nothing in the shipped constellation
does** — `[[build.kernel]]` mints through `DraupnirKernelRunner::build_toolchain_image`,
which reaches podman's `/build` endpoint over bollard. And bollard's
`BuildImageOptions` is generated from **Docker's** API spec, which has no `timestamp`
parameter, so until this lane the shipped door had no way to spell the one flag the
measurement depended on. Verified against podman 5.7 that the endpoint *does* accept
`timestamp=<seconds>`; draupnir now appends it through bollard's own
`with_request_modifier` hook (`ContainerBoot::build_image_reproducible`), which is a
supported extension point rather than a second HTTP client beside the one bollard owns.
**MEASURED GREEN, both doors, two fresh empty stores: `3f632569f198…` and
`3f632569f198…`.** Gate:
`tests/toolchain_images_live.rs::the_bollard_mint_and_the_cli_mint_agree_on_the_image_id`.
Getting there took **three reds**, and each one was a default nobody had ever
compared — the point of running the gate rather than reasoning about it:
| red | CLI | bollard | the difference |
|---|---|---|---|
| 1 | `3f632569f198…` | `e255092edde0…` | manifest FORMAT — `podman build` defaults to OCI, the Docker-compat `/build` endpoint to Docker schema2. The layer diff ids were IDENTICAL, so the whole of the divergence was the config's media type |
| 2 | `3f632569f198…` | *404 Not Found* | `outputformat` is honoured only on podman's **libpod** route (the compat route accepts it and ignores it — measured), and libpod routes are registered under `/v5.x/`, not under the Docker API version bollard negotiates |
| 3 | `3f632569f198…` | `f3dc2997b0f4…` | LAYER COUNT — the CLI puts one layer per instruction; the libpod endpoint squashes the build into one (3 layers vs 2) |
The moral is not "bollard is awkward". It is that **the same recipe through two
clients is two builds**, and until this gate existed the constellation had only ever
measured the client it does not ship.
⚠ `timestamp` is a podman extension. Docker's builder ignores it *silently*, which is
the shape this constellation refuses — so both gates ask the BUILT image what instant it
claims (`{{.Created.Unix}}`) before comparing ids, through a different door than the
build.
### What it costs, and what happens when it goes away
Minting now depends on **snapshot.debian.org**, a Debian-project service with no uptime
promise. If it goes away the mint fails LOUDLY (apt cannot reach its mirror) instead of
quietly drifting; the already-minted image is unaffected; and the recipe still names the
exact instant, so any mirror of that snapshot restores it. A build that stops is
recoverable; a pin that quietly stops meaning anything is not.
Nothing here reaches the network at a moment the build did not before: this is the
**mint** of the toolchain image, which was already an online step by construction
(`FROM` pulls a base, `apt-get` installs compilers). A kernel BUILD on the minted image
still declares `toolchain_packages = []`, still runs `--network none`, and lane AO's
`NORDISK-TOOLCHAIN-MISSING` still refuses by name with no fallback to a registry —
re-proven end to end at this lane's HEAD under a verified denial.
### What follows for the pin
The pin question dissolves. `toolchain_image_sha256` stops being the identity of one
minting event and becomes a property of the recipe, so:
* **"which box mints for everyone" is no longer a decision.** Any box with a podman and
a route reproduces the number.
* re-pinning is a mechanical edit: one const in the repo's `*-build-thing` code, then
`cargo run -p <repo>-build-thing -- emit`.
* both spellings work — the image **id** `3f632569f198…` (which survives a
`podman save`/`load` courier) and the manifest **digest** `78eaa55a651d…` (which does
not, and is therefore still the wrong one to pin for an airgapped box).
The number to pin, if the constellation re-pins to a reproducible mint of THIS recipe,
is the image id **`3f632569f1980e96744eef4ab258341fc77c55cb17c2dabe79b243e00d6e0825`**.
Lane AV did not take that decision: it is a cross-repo edit beside live emitter work,
and the point of the measurement is that it can now be taken with a number in front of
the person taking it.
## Not yet — do not read this as working
| step | status |
|---|---|
| **The RUN loop** (`FUNCTION_PROFILER` → workload → `trace_stat` → obj→CONFIG map → unused report) | **BUILT 2026-08-23** (`2d3d6b3` + `f5645ec`). This row read **NOT BUILT** until 2026-08-27, four days after it shipped; corrected here rather than deleted, because the row is what a reader would have trusted. `crates/nornir-build-thing/src/kernel.rs` now carries the whole chain: `KernelRun::traceable`/`traceable_names` (lifted off a virtio disk, not the serial console — `f5645ec`'s finding), `never_executed()` (the magnitude, `None` when the guest could not report the total, because "zero of an unknown total" is not a measurement), `never_executed_names()` (the candidates themselves) and `unused_configs()` (the obj→CONFIG join: a symbol is a drop candidate only when **nothing** under it ran). Live on `2d3d6b3`: **281 functions entered of 18 760 traceable — 98.5 % of that kernel was never entered.** A kernel built without `profile_run = true` refuses out loud (`NORDISK-RUN-REFUSED`) rather than reporting an empty measurement that reads exactly like a workload that executed nothing. Test: `tests/kernel_run_live.rs::a_profiled_kernel_reports_what_the_workload_executed` |
| dead-code elimination (`minimal = true`) | refused by name on `x86_64`; the option is arch-shaped and lives elsewhere |
| publishing the toolchain image to a registry | still open, and since 2026-08-30 (lane AV) it IS merely a convenience. It used to be a correctness fix: `toolchain_image_sha256` named an image we build locally and could not reproduce, so the committed pin was true on the box that minted it and false on every other one. **The recipe is reproducible now** (four mints, four fresh empty stores, one image id `3f632569f198…` and one manifest digest `78eaa55a651d…`; gate `two_mints_of_the_kernel_toolchain_are_measured_against_each_other`), so a fresh machine with a route mints the pinned number itself, and a machine without one takes the archive courier (lane AO). What a registry still buys is not having to spend the ~50 s and 1.2 GB. A pin that does not resolve now has three causes — never minted here, the recipe moved, or an input came unpinned — and podman's own words for all three are `image not known`, nothing about pins. Since 2026-08-30 (lane AH) the three `runner.run` sites append `KernelBuild::toolchain_repin_hint()` to that error: it names the pin that is wrong, the three causes it can be wrong for, the runnable `podman image inspect --format '{{.Digest}}' <image>`, the image-id-is-not-a-manifest-digest trap, and both files the new value must land in. VERDICT on a `.gitignore`d per-box override: an honest STOPGAP, never the fix — the pin exists so two builds reporting the same provenance really used the same toolchain, and a file the build silently prefers keeps every build green while making exactly that unverifiable. The fix WAS to publish the image or pin its inputs; measured 2026-08-30, pinning the inputs alone leaves the clock and is not enough, and both together (lane AV) make the pin reproducible without a registry at all. If an override lands first it must be explicit, LOUD (one row per build saying the declared pin was not the one used) and carried into the build record. Test: `kernel::tests::a_per_box_toolchain_pin_says_exactly_how_to_re_pin_it` |
| `arm64`, non-`bzImage` targets | refused by name, on purpose |