Gunnar

nordisk/edda

.nornir/kernel-for-idiots.md

<!-- nornir:class: idiot -->
# Building your own Linux kernel — for beginners

> **Class:** `idiot` · Zero-knowledge walkthrough of the `kernel` build kind.
> The commands live in [`kernel-guide.md`](kernel-guide.md).

## The one-sentence version

You write down **which parts of Linux you want**, and a container downloads the
official Linux source code and builds exactly that kernel for you.

## What is a kernel, and what is a `bzImage`?

The **kernel** is the part of Linux that talks to the hardware. It is the first
program a machine runs. Everything else — your shell, your web server — asks the
kernel to do things.

A **`bzImage`** is that kernel as one file. Roughly 13 MB for a normal one. When
a machine boots, this file is what gets loaded.

## What is a "kconfig symbol"?

Linux is built from thousands of optional parts. Each part has a **switch**. The
switch has a name that starts with `CONFIG_`. That name is a *kconfig symbol*.

Three examples:

- `CONFIG_TUN=y` — "build the virtual-network-card driver **into** the kernel."
- `CONFIG_SOUND=n` — "leave sound out."
- `# CONFIG_DRM is not set` — the same thing as `=n`, just the other spelling
  Linux uses when it writes the file itself.

`y` means *yes, inside the kernel file*. `n` means *no, not at all*. There is a
third answer, `m`, meaning "build it as a separate file to load later" — a
**module**. Remember that word; it has a sting in it further down.

A file full of these lines is a **config**. Linux 6.12.104 has **17 704** of
these switches. Nobody sets them by hand.

### The catch: switches depend on other switches

`CONFIG_VIRTIO_BLK` (a virtual hard disk) needs `CONFIG_BLOCK` (the disk layer)
and `CONFIG_PCI` (the bus the disk hangs off). If you ask for the disk but not
the bus, Linux does not argue with you — it **quietly turns your switch back
off**. That silence is the whole problem this system is built around.

## "Make it smaller" is three different knives — but only two of them cut here

People say "trim the kernel" as if it were one job. It is three jobs, at three
different depths. None of them can do another one's work.

| knife | it decides… | in plain words |
|---|---|---|
| **1. where you start** | your starting config | Do you begin from `defconfig` (a big kernel that runs on almost any PC) or `tinyconfig` (almost nothing, and you add back what you need)? |
| **2. minify** | which features exist | Of the features you asked for, which switches must be on so they actually work — and no more? A solver works this out for you. |
| **3. dead-code removal** | dead code *inside* what is left | Some functions can never be called by anything you kept. The linker deletes them. |

Knives 1 and 2 are built and measured. **Knife 3 does not exist on our
machines**, and that was found out by trying it, not by guessing:

- The switch that does it, `CONFIG_LD_DEAD_CODE_DATA_ELIMINATION`, is only
  offered on arm, mips, riscv and powerpc chips. Our machines are x86_64. Asking
  for it now gets a clear refusal instead of doing nothing quietly — a setting
  that silently does nothing is worse, because the receipt makes it look like it
  worked.
- The other candidate, `CONFIG_TRIM_UNUSED_KSYMS`, only exists in kernels that
  allow modules. Our small kernels forbid modules. So it cannot be there either.
- The nearest substitute is a different compiler, **clang**, with a feature
  called LTO. It works, it boots — and it made the small kernel **bigger**:
  2 016 256 bytes against gcc's 1 811 456. That is 204 800 bytes more, 11.3 %.
  LTO is a **speed** tool, not a size tool.

So: **two knives, not three.**

## The four presets

A preset is a saved answer. You pick one word.

| you type | you get | size (measured) |
|---|---|---|
| `full` | a generic kernel for unknown hardware | **13 693 952 bytes**, 246 s |
| `normal` | the default — generic, plus the devices our machines need | **13 714 432 bytes**, 235 s |
| `slim` | start at nothing, add back only what was measured as needed | **1 811 456 bytes**, 80 s |
| `nuked` | `slim` minus graphics, sound, keyboard, USB, wi-fi | **1 725 440 bytes**, 78 s |
| `slim` + the solver | `slim`, with knife 2 as well | **1 389 568 bytes**, 148 s |

(All on Linux 6.12.104, on one machine, with the source already downloaded.
`slim` is 7.6 times smaller than `normal`. The solver takes another 23.3 % off
`slim`.)

### The 20 480 bytes that decide everything

Look at `full` and `normal`. They are almost the same size — 20 480 bytes apart —
and that tiny difference is the entire point of this system.

`full` is the plain generic kernel. It boots. But on a generic kernel,
`CONFIG_TUN` (the virtual network card our appliances send their encrypted
traffic through) is built as a **module** — a separate file, loaded later. Our
appliances have no separate files. So a `full` kernel starts up, works, and has
**no network tunnel at all**.

`normal` is `full` plus a short list of switches that were each written down
after something broke. It costs 20 480 bytes and buys a machine that can actually
do its job. That is what those 20 kilobytes are.

We did not read this off a config file. We booted the thing and asked it. See
below.

## Why a smaller kernel is not automatically a *working* kernel

After the build, a check called the **canary** runs. It reads your config back
and asks, line by line: *did every switch I asked for actually stay on?* If one
was quietly turned off, the build stops and prints its name.

The canary is good. It also has a hole you must understand:

> **The canary can only check what you WROTE DOWN.**

If you never mentioned a switch, the canary has nothing to compare against, so
it says nothing about it.

Here is the trap, and it is a real one. `CONFIG_BINFMT_ELF` is the switch that
lets Linux **run a program at all**. Not a driver — the ability to start
software. `tinyconfig` leaves it off. Nobody thinks to write it down, because
nobody thinks of "running programs" as a feature.

A kernel without it:

- builds fine,
- passes every canary line,
- boots,
- and then cannot start a single program. It is not a smaller kernel. It is a
  brick.

### "But surely the solver catches that?" — no. This was measured.

This is the most important thing on this page.

The solver (knife 2) reads Linux's own rules and works out which parent switches
must be on. It is genuinely clever. So the obvious idea is: throw away the
hand-written list of parent switches and let the solver find them all.

That was tried. The hand-written list had 16 lines. **The solver found 13 of them
by itself.** It missed three: `CONFIG_BINFMT_ELF`, `CONFIG_MULTIUSER` and
`CONFIG_UNIX`.

Those are exactly the three the canary cannot catch — and for exactly the same
reason. **Nothing in the request mentions them.** The solver answers "what must
be on for what you asked for to work". It cannot answer "what did you forget to
ask for".

That kernel was 1 369 088 bytes, passed every check, and could not run a single
program. The boot test refused it: *did NOT reach PID 1*.

> **A solver is a knife, not an oracle.** It makes what you declared as small as
> it can be. It does not discover what you failed to declare.

The only thing that catches this is **actually booting it**. That is the
**probe**: the built kernel is started inside a virtual machine (QEMU), and a
tiny program written in C is handed to it as the very first thing to run. If that
program prints its greeting on the serial line, then the kernel booted *and* it
can run software. Those are two separate facts, and the probe proves both.

The probe also names each device it looks for as `present` or `ABSENT`. It never
just prints what it found — a report that only lists successes reads identically
on a kernel that has nothing. That is how `full`'s missing network tunnel was
caught: the probe asked the running kernel, and it answered `tun ABSENT`.

## One complete worked example

Goal: build the smallest kernel that still boots, and prove it boots.

### 1. Write the declaration

In your repo's `.nornir/build-variants.toml`:

```toml
[[build.kernel]]
name = "my-first-kernel"
version = "6.12.104"
tarball_sha256 = "949d166a…"          # the fingerprint of the official source file
preset = "slim"
toolchain_image = ""                   # empty name + the id below = "the image on this box"
toolchain_image_sha256 = "388a7816…"   # the toolchain container's id
toolchain_packages = []                # it already has everything → no internet needed
```

The `tarball_sha256` is not optional. It is the fingerprint of the source
download. Without it you would be compiling whatever the network happened to
hand you. Leave it out and the build refuses, by name.

### 2. It checks the disk before it downloads anything

It wants 6 GiB free. A `slim` build's working directory peaks at 1.9 GB, plus a
148 MB source file. If the disk is too small it says so **now**, not after 80
seconds of compiling.

### 3. It fetches the source — once

The 148 MB file is saved in a shared cache, under a filename that *is* its own
fingerprint. Build three different presets of the same version and it downloads
once. A stale cache entry cannot exist: if the bytes do not match the name, it is
not a hit, it is garbage, and it gets fetched again.

### 4. It compiles, inside a box with walls

The compile happens in a container limited to 8 CPU cores and 8 GiB of memory.
This is not decoration. On 2026-08-20 an unlimited build reached 33 GB and took
the whole machine down. `mem_limit_mb = 0` is refused: "unlimited" is not a
limit.

### 5. The canary speaks — and this is the good part

The first time anyone ran `slim`, it stopped after 18 seconds with twelve lines:

```
KERNEL-CONFIG-REFUSED: CONFIG_EFI_STUB=y did not survive olddefconfig
KERNEL-CONFIG-REFUSED: CONFIG_TUN=y did not survive olddefconfig
…
KERNEL-CONFIG-REFUSED-TOTAL: 12 option(s) did not hold
```

**That is not a failure. That is the answer to a question.** Each line means "you
asked for this, and its parent switch is off, so it was dropped." The fix is to
turn the parents on: `CONFIG_PCI`, `CONFIG_BLOCK`, `CONFIG_NET`, and so on.

Those twelve answers became a permanent file, `deps-tinyconfig.config`. Every
line in it was written by a refusal, not by a human reading documentation. You
will not have to repeat this for `slim` or `nuked` — it is done. You *will* see
it again the day you ask for something new.

### 6. The build finishes

Three files land in `.kernel-out/my-first-kernel/`:

- `bzImage` — the kernel, 1 811 456 bytes;
- `kernel.config` — every switch, as it ended up;
- `my-first-kernel.kernel-pin.json` — the receipt: the fingerprint of the
  source, of the config, of the kernel, which preset it was, and which container
  built it.

### 7. Boot it — the step that catches the brick

Point the probe at what you just built and run it. It starts QEMU inside the
same container and reads the serial line. A good run says:

```
NORDISK-PROBE-PID1-ALIVE
NORDISK-PROBE-DEV console present
NORDISK-PROBE-DEV tun present
NORDISK-PROBE-DEV virtio-blk present
NORDISK-PROBE-DEVICES
```

Measured: `slim` reaches this in **2.1 seconds**, `nuked` in 2.1, `normal` in
4.2. `full` boots too — and says `tun ABSENT`, which is the whole lesson of this
page in one line.

Now you know your kernel works. Not because it compiled — because it ran.

### 8. Optional: turn the solver on

Add one line and rebuild:

```toml
solver = "klocalizer"
```

The container reads Linux's own rules, works out the smallest set of switches
that still satisfies what you asked for, and hands that to the build. `slim`
drops from 1 811 456 to **1 389 568 bytes**. Two runs produced byte-identical
kernels.

The first run takes longer — 148 s instead of 80 — because it must first turn
Linux's rules into a 19 949 986-byte puzzle for the solver. That puzzle is saved
in the same cache the source file lives in, so every later build on the same
Linux version gets it free (103 s).

It also writes a file, `deps-6.12.104.config`, listing every parent switch it had
to turn on. That is a list a human can read and commit — the same list that was
once closed by hand, done automatically.

Then boot it. Always boot it. See the measured brick above for why.

### A note on "closed" and "open" worlds

One detail decides whether the solver helps or hurts. Left to itself, the solver
treats every switch you did not mention as *free to turn on*, and it happily
turns on network-card and filesystem drivers nobody asked for — **472 extra
switches** in the measured run. Written the right way, with every unmentioned
switch explicitly turned off first, the same solve adds **78**.

A naively-wired minifier does not make your kernel smaller. It makes it bigger.
Ours writes the closed version; you do not have to do anything. But if you ever
read that the solver "added mlx5", this is why.

## Building with the internet unplugged

A kernel build needs three things off the network: the **source tarball**, the
**solver formula** (only if you turned the solver on), and the **toolchain** —
the box with the compiler in it. The first two were already solved: they live in
a cache, named after their own checksum, and you can simply copy the files onto
the machine.

The toolchain was the one that was not. It is a container image, and an image
that is not on the machine leaves podman with exactly one move: go and fetch it.
On a machine with no internet, that is where the build stops.

So the toolchain now travels the same way the tarball does — **as a file**.

**On a machine that HAS the toolchain:**

```bash
ID=$(podman image inspect --format '{{.Id}}' nordisk-kernel-toolchain)
mkdir -p ~/.cache/nordisk/kernel/images
podman save --format docker-archive -o ~/.cache/nordisk/kernel/images/$ID.tar \
    nordisk-kernel-toolchain
```

**Copy that one file** to the offline machine, into the same place
(`<cache>/images/<ID>.tar`), and put those same 64 characters in the
declaration:

```toml
toolchain_image_sha256 = "<ID>"
```

That is the whole procedure. The build then loads the image out of the file and
compiles with no network at all.

### The trap that cost this a day: two different 64-character numbers

Podman will tell you two long hex numbers about an image and they are **not
interchangeable**:

| what | how to read it | does it survive being copied to another machine? |
|---|---|---|
| the image **ID** | `podman image inspect --format '{{.Id}}'` | **YES** |
| the **manifest digest** | `podman image inspect --format '{{.Digest}}'` | **NO** |

Measured on oden: save an image, load it on another machine, and the ID comes
back the same while the digest comes back *different* — podman rebuilds the
manifest when it loads, so the digest describes the copy, not the original. Pin
the **ID**. If you pin the digest, the offline machine will tell you, by name,
that the archive holds a different image than the one you asked for — and it will
refuse rather than quietly use it.

### What it says when something is missing

There is no quiet fallback here, on purpose. If the toolchain is not on the
machine and no archive is either, the build stops with

```text
NORDISK-TOOLCHAIN-MISSING: kernel 'x': the toolchain image `…` is not in this
box's image store, and no staged archive is either.
```

and then tells you the exact path to put the file at and the exact command that
makes it. A build that quietly downloaded a toolchain instead would look
identical to a working offline build, which is the same as having no offline
build at all.

## You can click all this

There is a **Kernel tab** in the dwarves app with a slowly rotating 3D picture of
the switches each preset turns on and off. It is a real picture of the committed
files, not a decoration — though the lines between switches are drawn from their
names, not from Linux's real rule graph, and the tab says so on itself.

## Careful

* A green build is **not** a working kernel. Only the probe says that.
* A `KERNEL-CONFIG-REFUSED` list is a to-do list, not a stack trace. Read it as
  "here are the parents you still owe."
* The solver does not replace the boot test. It cannot see what nobody wrote
  down — measured, three switches, one brick.
* Never remove `tarball_sha256` to "make it work". It is the difference between
  building Linux and building whatever arrived over the wire.
* Pin the image **ID**, never the manifest digest. They are both 64 hex
  characters and only one of them means the same image on two machines.
* A log can lie. The probe once reported a successful boot for a kernel that was
  never started: the compiler had failed, and its error message quoted the source
  line containing the greeting text. The probe now checks that the step actually
  succeeded *before* it reads a single line of output. Take the lesson with you.

## More

[`kernel-guide.md`](kernel-guide.md) has every field, every refusal, and the
commands. The reasoning lives in `workspace_nordisk/.nornir/kernel-trim-design.md`.