<!-- nornir:class: reasoning -->
# Docs generation
Reference for how nornir generates the per-repo README / CHANGELOG, the
whole-documentation **book**, and (optionally) PDF/HTML/MD exports. Everything
rendered goes through **typst** — one engine, no JS, no shellout. This file
lives in `.nornir/` and is itself a chapter of `nornir docs book` (it documents
the pipeline *and* ships inside the rendered book).
Source code: [`src/docs/`](../src/docs/) — `mod.rs`, `layout.rs`,
`sections.rs`, `test_inventory.rs` (the `tests` section's backing), `hooks.rs`,
`warehouse.rs`, `search.rs`, `pages.rs` (Codeberg Pages publishing), and
feature-gated (`docs-export`) `export.rs`, `book.rs`, `svg.rs`,
`release_book.rs`.
## The `.nornir/` layout
Every repo nornir owns gets a `.nornir/` directory at its root. The source of
truth lives there; the top-level artifacts are regenerated. **All
nornir-managed media lives under `.nornir/assets/` (tracked)** — one namespace,
no copy at the repo root:
```text
repo/
├── .nornir/ ← nornir's namespace
│ ├── README.md ← SOURCE (you edit this)
│ ├── CHANGELOG.md ← SOURCE (you edit this)
│ ├── docs-generation.md ← this file (a book chapter)
│ ├── design.md ← design overview (a book chapter)
│ ├── assets/ ← TRACKED media: depgraph.svg, logos, figures
│ │ ├── depgraph.svg ← GENERATED — committed so links resolve
│ │ │ on git hosts (Codeberg/GitHub UI)
│ │ └── *.webp / *.png ← hand-placed figures
│ ├── warehouse/ ← DATA ONLY: the Iceberg warehouse (gitignored)
│ ├── cache/ ← transient: fetched images, the docs index
│ │ │ (gitignored), rehydrated from the warehouse
│ └── .gitignore ← ignores cache/ + warehouse/
├── README.md ← GENERATED — DO NOT EDIT (chmod -w)
├── CHANGELOG.md ← GENERATED — DO NOT EDIT (chmod -w)
├── docs/ ← GENERATED rendered documents (book.pdf/.md, …)
│ │ the live, link-target copies; tracked
└── ...
```
Two distinct destinations for rendered output: the **current** artifacts land
as plain files under `<repo>/docs/` (git-tracked, what the README links to),
while their full **history** is historized as rows in the Iceberg `doc_exports`
table (inline bytes, dedup by sha256). `.nornir/warehouse/` holds *only* the
Iceberg warehouse data — never rendered documents.
Why `.nornir/assets/` and not a top-level `assets/`? Git hosts resolve a
relative `.nornir/assets/x.svg` from a README at the repo root just as well as
`assets/x.svg`, so there is no web-UI reason to pollute the repo root. Keeping
generated + committed media beside the sources means one nornir-owned
namespace and no duplicate copies. (The typst exporter resolves these paths via
a filesystem root mounted at the repo root — see *Export pipeline* below.)
`MANAGED_DOCS = ["README.md", "CHANGELOG.md"]` in `layout.rs`. Adding a
new managed doc is a one-line change to that list — it then gets
`init` / `render` / `check` for free. Any other `.md` you drop into
`.nornir/` (like this file) is left alone by `render`, but is still picked up
as a chapter by `docs book`.
> Note: `assets/typst/` in the **nornir source tree** is unrelated — it holds
> the typst *templates* (`report.typ`, `web.typ`) embedded into the binary via
> `rust-embed` at build time. That is nornir's own build asset, not per-repo
> media.
## Lifecycle
| Step | CLI | Function | Effect |
|---------|------------------------------|-------------------------|--------|
| Init | `nornir docs init <repo>` | `layout::init_repo` | Creates `.nornir/{cache,warehouse}/`, migrates any pre-existing top-level `README.md` / `CHANGELOG.md` into `.nornir/` as the source, strips any prior GENERATED header, scaffolds `.nornir/.gitignore` with `cache/` **only when it does not already exist** (`if !gi.exists()`, layout.rs:400 — so a hand-extended ignore, like this repo's `cache/`+`warehouse/`, is never clobbered). Scaffolds a starter README if none exists. Idempotent. |
| Render | `nornir docs render <repo>` | `layout::render_all` + `layout::render_sources_in_place` (+ `book::build_book` for `pdf`) | For each managed doc: read source → expand marker sections → prepend GENERATED header → atomic write to the repo root. **Then** fills `nornir:gen:*` markers **in place** for every non-managed `.nornir/*.md` source (`design.md`, `plan.md`, …), which are published as-is and would otherwise carry empty (invisible) marker blocks — this is what makes the design-doc `depgraph` actually appear on disk. Honours and restores the read-only bit. No-op if the output is already byte-identical. **What it emits is controlled per-repo by `.nornir/docs.toml`** — see *Render outputs* below; with `outputs = ["pdf"]` it skips markdown entirely and only (re)builds `docs/book.pdf`. |
| Check | (release gate `docs_fresh`) | `layout::check_all` | In-memory dry-run. Bails non-zero if rendering *would* change the artifact, so CI fails when someone edits the artifact instead of the source. |
| Export | `nornir docs export …` | `export::export_repo` | Feature `docs-export`. Renders the assembled top-level `README.md` to PDF / HTML / MD via embedded typst. Writes the current artifact to `<repo>/docs/README.<ext>` and historizes the bytes in the Iceberg `doc_exports` table (`docs::record_doc_export`). |
| Book | `nornir docs book …` | `book::build_book` | Feature `docs-export`. Renders the **whole doc set** (`.nornir/*.md` + non-generated `<repo>/*.md`) into one typst document → `<repo>/docs/book.<ext>` + a `doc_exports` row. See *Whole-documentation book*. |
| History | `nornir docs history <repo>` | `docs::list_doc_exports`| Lists historized exports from the Iceberg `doc_exports` table (filter by doc/version/format). |
| Index | `nornir docs index <repo>` | `docs::build_docs_index`| Builds the **dedicated docs index** over the *generated* docs (repo-root managed docs + `docs/*.md`) and historizes it into `docs_index_snapshots`/`docs_index_blobs`. See *Docs index*. |
| Search | `nornir docs search <repo> q`| `docs::search_docs` | BM25 over the docs index; rehydrates from the warehouse on a cold cache (optionally pinned to a `--sha`). |
The render is **whole-file**: nornir rewrites the entire artifact, not
just the marker regions. That's why the artifact gets a loud header
warning, and why we recommend `chmod -w` on it (handled automatically
by `write_atomic`).
## Render outputs (`.nornir/docs.toml`)
`nornir docs render` defaults to the historical behaviour — render the managed
markdown (`README.md` / `CHANGELOG.md`) to the repo root. A repo can change what
`render` emits with a small per-repo config, `<repo>/.nornir/docs.toml`
(`docs::DocsRenderCfg` in `layout.rs`):
```toml
# <repo>/.nornir/docs.toml
outputs = ["pdf"] # ["markdown"] (default) | ["pdf"] | ["all"]
```
| `outputs` | render emits | writes markdown? |
|-----------|--------------|------------------|
| `["markdown"]` (or absent) | managed `README.md` / `CHANGELOG.md` | yes |
| `["pdf"]` | the book → `docs/book.pdf` only | **no** |
| `["all"]` | both | yes |
The `["pdf"]` mode is the way to add documentation rendering to a project whose
`*.md` are **authored by hand and must not be overwritten**: `render` builds the
book straight to `docs/book.pdf` and never writes a markdown file into the repo.
It reuses `book::build_book` but, unlike `docs book`, does **not** historize into
the warehouse — `render` is the lightweight "refresh the on-disk artifact" verb.
`pdf`/`all` require a nornir built with the `docs-export` feature; otherwise
render fails with a clear message pointing at `--features cli`.
> Aside (`Nornir::repo_dir_for` / `repo_dir_resolved`, `config.rs`): repo path
> resolution first honors an explicit `[repo.<name>].path` override, then
> tolerates a case mismatch between the `[repo.<name>]` key and the on-disk
> directory (e.g. `[repo.njord]` ↔ a `Njord/` checkout) — exact join first,
> then a case-insensitive sibling match.
>
> The `path` override (relative→joined under the workspace root, absolute→
> verbatim) is the only way to point a repo at a directory whose name differs
> from the key. It is what makes `docs book nornir` render nornir's **own** book
> from a configured `nornir-orch` worktree (`[repo.nornir] path = "nornir-orch"`)
> rather than silently resolving to the by-name `<workspace_root>/nornir`
> checkout. Before this fix the `path` key was dropped (the `Repo` struct had no
> `path` field), so the self-render fell through to the wrong/absent checkout and
> wrote an empty ~16 KB `book.pdf`. The docs handlers resolve via `repo_dir_for`;
> a relative override joins the workspace root and an absolute one wins verbatim.
> Regression-guarded by `config::tests::repo_dir_for_honors_path_override_*` and
> `docs::book::tests::self_repo_book_lands_at_real_docs_path_not_tmp` (real PDF at
> `<repo>/docs/book.pdf`, >1 source, dwarves the empty stub).
## Generated header
Every artifact begins with:
```text
<!-- ⚠ GENERATED by nornir from .nornir/README.md — DO NOT EDIT this file -->
```
Constants live in `layout.rs` as `GENERATED_HEADER_PREFIX` /
`GENERATED_HEADER_SUFFIX`. `init_repo` strips the header on
re-migration so re-init is idempotent.
There is also a legacy whole-file marker `<!-- nornir:generated -->`
(`docs::GENERATED_MARKER`) used by the older `assemble_and_check` flow
for repos that haven't moved to the `.nornir/` layout yet. New repos
should use the layout above.
## Marker syntax
Inside any source `.md` under `.nornir/`, paired HTML comments mark a
region that `sections::rewrite` will replace on every render:
```text
<!-- nornir:gen:start:bench -->
… anything here is overwritten …
<!-- nornir:gen:end:bench -->
```
Markers carry an optional whitespace-separated `key=value` arg list:
```text
<!-- nornir:gen:start:symbols binary=target/debug/holger limit=20 -->
<!-- nornir:gen:end:symbols -->
```
Rules enforced by `sections::parse_markers`:
- Start and end names must match. Mismatch → hard error with line numbers.
- Unclosed start → hard error.
- Markers inside fenced code blocks (```` ``` ```` or `~~~`) are
**ignored** — so this very file can show examples without nornir
trying to expand them.
- Args are re-emitted in sorted order on render, so the output is
deterministic regardless of how you typed them.
## Generation is NON-DESTRUCTIVE (absence means keep, never blank)
A render that has **no fresh data** for a section must never blank content that
is already there. `sections::render` receives the section's existing body and
applies one rule:
> If the renderer produced a *no-data placeholder* **and** the section already
> has a body, the existing body is kept verbatim.
That covers a real table (the numbers are protected) and an existing placeholder
alike — a no-data render is therefore **idempotent**, rewriting nothing. Swapping
one absence wording for another changes no fact but leaves the doc dirty after
every commit, and a perpetually-dirty generated file is one people stage blindly.
That habit is how the real tables got committed away in the first place.
The placeholders that count as absence are listed in `NO_DATA_PREFIXES`
(`_(no bench history yet…`, `_(no bench results…`, `_(no capability matrix yet…`,
`_(no \`<source>\` static-capabilities row…`, `_(no coverage recorded…`).
Classification lives in `is_no_data_placeholder`.
Preservation is scoped so it can't hide real state:
- An **empty** section still gets its placeholder — there is nothing to protect,
and a fresh doc should explain itself.
- **Fresh data always replaces** a stale table. Keeping applies only to absence.
- **Author errors are not absence.** A malformed marker
(`_(bench_hero needs \`fast=\` and \`slow=\` backend tokens)_`) stays visible so
it gets fixed.
**Why this rule exists.** The pre-commit hook runs `nornir docs render` on *every*
commit. Whenever the warehouse has no run for the repo — it was down, the repo was
never benched, a rival arm dropped out of the latest sweep — every bench renderer
returned its placeholder and the splice wrote it straight over the committed
table. This destroyed all 13 benchmark tables in `skade/.nornir/README-full.md`
(skade `61e954d`, 80 lines of measured numbers), and the same class of bug once
deleted the holger-vs-Nexus table. Benchmark numbers are expensive to produce and
are frequently *not* reproducible on demand; a docs step that can silently delete
them is a data-loss bug, not a formatting quirk.
Regression guards: `render_without_bench_data_preserves_existing_table`,
`render_with_unrelated_run_preserves_filtered_table`,
`preservation_does_not_block_placeholder_or_fresh_data`,
`no_data_render_is_idempotent_over_existing_placeholders`,
`no_data_classifier_separates_absence_from_content` (in `src/docs/sections.rs`).
## Built-in section renderers
Registered in `sections::render`:
### `bench`
Renders a flat Markdown table from the latest `BenchRun` — version, core
count, then one row per `(result, metric)`. If no bench history exists yet,
emits a placeholder italic line. Kept for back-compat; prefer `benches`.
```text
<!-- nornir:gen:start:bench -->
<!-- nornir:gen:end:bench -->
```
### `bench_history [limit=N] [metric=NAME]`
One row per past bench run (newest first) from the warehouse —
`| Version | Date | Machine | <metric> |`. Reads the pre-fetched `Ctx::history`
(the caller fetches via `query_bench_runs`; renderers never open the warehouse
themselves). This is what makes the hand-maintained "Benchmark history" tables in
the holger / znippy READMEs obsolete.
- `limit=N` — keep only the N most recent runs (default 20).
- `metric=NAME` — show this metric's value per run (matched against each result's
metric key or result name); default is the run's max numeric metric.
```text
<!-- nornir:gen:start:bench_history -->
<!-- nornir:gen:end:bench_history -->
```
### `benches [name=PAT] [best=metric:low,…]`
Pivots the latest `BenchRun` into a table — **one row per result/workload, one
column per metric** — and **bolds the winning cell** in each row. Cells that
share a unit (e.g. `ljar_mbs` + `unzip_mbs`, both `mbs`) form a comparison
group; within a group of ≥2 the best value is bolded. "Best" direction is
inferred from the metric's unit (`crate::bench::unit_of`: `_mbs`/`_ops_sec`
higher-better, `_ms`/`_s`/`seconds` lower-better, `_pct`/`_x` neutral). This is
what turns **paired ours-vs-legacy metrics into a "how much faster" table**.
- `name=PAT` keeps only results whose name contains `PAT`.
- `exclude=PAT` drops results whose name contains `PAT`.
- `best=metric:low,other:high` overrides direction for units the suffix
convention can't classify (e.g. a `_pct` where higher is better).
- `winner=row` flips the bolding axis: instead of best-cell-within-each-row
(paired same-unit metrics), it bolds the best cell **down each column** —
rows are rival systems/backends, columns are metrics, so it shows *which
system wins each metric*. Use for cross-system comparison tables.
- `compare=tokenA,tokenB,…` collapses the matched results to **exactly one row
per token** (first name-containing hit, in the given order). This is the
"mashup" lever: fold nornir's `nvme`/`ram`/`s3` variants down to a single
rival row for a crisp head-to-head vs Nessie / Polaris, e.g.
`name=table_exists winner=row compare=nornir_embedded,nessie,polaris`.
- `top=N` / `rank=METRIC` — keep only the top-N rows ranked by `METRIC`.
- `cols=a,b,c` — restrict (and order) the metric columns shown.
- `label=…` — override the first (name) column header.
- `mode=compact` — a denser table variant.
- `bold=false` — disable the winner-cell bolding.
Metric directions recognised by `unit_of` (suffix-matched): `_ops_sec`,
`_per_sec` (e.g. `rows_per_sec`, `commits_per_sec`), `_mbs`/`_gbs` higher-better;
`_s`/`_secs`/`_ms`/`_us`/`_ns` lower-better; `_pct`/`_x` neutral. Latency results
now carry the full spread — `min_us`/`mean_us`/`p50`/`p90`/`p99`/`p999`/`max_us`
plus `ops_sec` — so a single table shows best- and worst-case side by side.
The single *headline* number ("most meaningful, at the top") is **hand-curated**
prose — copy the row you care about above the table; the table keeps itself
fresh and shows the full comparison with the winner in bold.
```text
<!-- nornir:gen:start:benches -->
<!-- nornir:gen:end:benches -->
```
### `bench_status` (no arguments)
The run's **provenance + honesty header**, and the only marker that still renders
something useful when a repo has *no* numbers. Everything it prints is derived —
from the `BenchRun` in context, or (with no run) from the repo on disk. It takes
no arguments on purpose: a fact you can declare in a marker is a fact that can be
wrong.
Three states:
- **Measured** — `v… · host · cores · date`, the arm count, the test pass/fail
count (with each failure named), any **borrowed arms** (arms named `<ns>.<arm>`
whose `<ns>` is not this repo — knut's run carries 10 `nornir.*` because its
pinned runner predates `NORNIR_BENCH_ONLY`), and a `version resolution fell
back` note when the run stamped `0.0.0`.
- **Preview** — the harness declared its own run unpublishable by writing the
`PREVIEW` token into `machine`; the block leads with a refusal banner. See the
preview gate below.
- **Not benched** — no run at all. Instead of a silent gap, it probes the repo
(`nornir-bench.rs` entrypoints; criterion `[[bench]]` target counts per
`Cargo.toml`) and states *why* there are no numbers: no harness at all
(Skidbladnir), or criterion-only with nothing host-stamped (tunnr). **An absent
row reads as "fine" — this makes absence a rendered result.**
```text
<!-- nornir:gen:start:bench_status -->
<!-- nornir:gen:end:bench_status -->
```
#### The implausibility screen (`suspect=` on `benches`)
`bench_status` and `benches` share one screen — `docs::bench_status::suspect_reasons`
— so a table and its caveat can never disagree. A row is flagged when:
- a `*_per_s` rate ≥ 1e9 is derived from the row's shortest measured interval and
that interval is under 1e-4 s (below the timing floor; the rate is arithmetic,
not measurement), **or**
- a `*speedup*` metric on a row that also describes parallelism (`concurrency`,
`cores`, `parallel_*`) exceeds the cores that produced it. An **algorithmic**
speedup on a row with no parallelism is legitimately unbounded and is *not*
flagged.
`benches` takes `suspect=flag` (default — the row stays, its name gains ⚠, and the
reason renders under the table), `suspect=exclude` (the row is withheld and the
reason renders under the table), or `suspect=off`.
#### The preview gate (`preview=yes`)
A run stamped `PREVIEW` is **invisible** to `bench`, `benches`, `bench_chart`,
`bench_hero`, `mashup` and `capabilities` unless the marker opts in with
`preview=yes`. Those renderers report absence, and the non-destructive rule keeps
whatever the doc already had. This is why a 32-core under-floor preview cannot
overwrite znippy-zoomies' measured 12-core tables. `bench_status` is exempt —
stating the refusal is its whole job.
### `bench_chart metric=M name=PAT [title=…]`
Native-typst horizontal bar chart for one metric across the matching
workloads/systems — the visual counterpart to a `benches` table. Auto log-scale
when the spread exceeds ~100×, winner bar in green, value labels. Renders to an
SVG asset under `.nornir/assets/` with a markdown image link (under
`docs-export`); a unicode-bar text fallback otherwise. No CeTZ/package
dependency — drawn with typst `rect`/`text` primitives.
- `metric=M` (required) — the metric to chart; `name=PAT` filters the workloads.
- `exclude=PAT` — drop workloads whose name contains `PAT`.
- `dir=high|low` — force the winner direction (else inferred from the unit).
- `log=true|false` — force / disable log-scale (else auto above a ~100× spread).
- `file=NAME` — override the emitted SVG filename under `.nornir/assets/`.
- `best=metric:dir` — per-metric direction override, as for `benches`.
### `bench_hero name=PAT fast=TOK slow=TOK [unit=_s] [label=query]`
The "most dramatic" callout. Among the per-item metrics the two backends share
(keys like `q01_s`…`q22_s` — i.e. `q<digits><unit>`, lower-better), it finds the
item where `fast` beats `slow` by the biggest ratio and renders a one-line
headline plus a tiny 1-row table. Built for per-query TPC-H (nornir vs a
competitor) — the single "N× faster on the worst query" line that leads a README.
```text
<!-- nornir:gen:start:bench_hero -->
<!-- nornir:gen:end:bench_hero -->
```
### `mashup [self=TOK] [exclude=PAT] [metric=KEY] [top=N]`
The **zero-config** top-of-README comparison table. Where `benches` needs you to
hand-list the rivals (`compare=holger,nexus,artifactory winner=row …` plus
filters), `mashup` takes **no required args**: it AUTO-DISCOVERS the repo's own
system (*ours*) and **every** rival system from the latest `BenchRun`, then
renders the winning N-way table itself. A bare marker is the whole template:
```text
<!-- nornir:gen:start:mashup -->
<!-- nornir:gen:end:mashup -->
```
How it works (all inference, no configuration):
- **Systems** are read from each result's `name` — the leading `_`-delimited
segment is the system token (`holger_get_blob` → `holger`, `nexus_get_blob` →
`nexus`), so variants of one system (`nornir_ram`, `nornir_nvme`) collapse to a
single row.
- **Ours** is auto-detected: an explicit `self=` wins; else the system whose
token overlaps the repo's `[package] name` (Cargo.toml) / directory name; else
the most-frequent system (we bench ourselves across many workloads). If a rival
outnumbers ours *and* the repo name matches no token, pin ours with `self=`.
- **Rivals** are every other discovered system (minus `exclude=`).
- The table is rendered by reusing `benches` as `compare=<ours+rivals>
winner=row` — **meaty-first columns** (throughput, then p50→p99 latency) and
ours' winning cells **bolded** (best cell down each column).
- Above the table, a one-line **biggest-diff headline** (max-ratio, the same idea
as `bench_hero`, generalised past its `q<n>_s` items to any metric/rival) names
the single workload where ours leads a rival by the largest factor — emitted
only when a clear pairwise standout exists.
Optional overrides (never required — zero-config is the point):
- `self=TOKEN` — pin *ours* instead of guessing.
- `exclude=PAT` — drop rival systems whose token contains `PAT`.
- `metric=KEY` — restrict the table (and the headline) to one metric column.
- `top=N` — keep only the N strongest systems (ranked as in `benches`).
- `best=metric:dir` — per-metric direction overrides, passed through.
Empty / absent bench data renders a graceful placeholder, exactly like the other
`bench*` sections — never a panic or error.
### `tests [crate=PAT] [kind=unit|integration|doc]`
Enumerates the repo's tests grouped by the **Rust-standard** categories, parsed
from source with `syn` (so the list stays correct as tests come and go):
- **unit** — `#[test]` / `#[tokio::test]` (any `…::test`) fns under `src/`;
- **integration** — the same, under `tests/`;
- **doc** — runnable ```` ``` ```` fences in `///` / `//!` comments
(`ignore` / `text` fences are not counted). `#[bench]` is *not* a test.
`crate=PAT` limits to crates whose name contains `PAT`; `kind=` renders only one
category; `include_benches=true` also counts perf-benchmark files (`*_bench.rs`,
`perf_*`, `*throughput*`, `*benchmark*`) that `is_benchmark_file` excludes by
default. Replaces hand-authored test inventories. The enumeration itself lives in
[`test_inventory.rs`](../src/docs/test_inventory.rs) (`scan` / `scan_opts`,
parsed with `syn`); `render_tests` only formats the returned `TestInventory`.
```text
<!-- nornir:gen:start:tests -->
<!-- nornir:gen:end:tests -->
```
### `coverage [worst=N]`
Maven-style coverage face (feature N): an overall line / region `%` line plus a
per-crate table, from the pre-fetched latest `coverage` run (`Ctx::coverage`).
With `worst=N` it adds a worst-covered-files table. Renders a graceful
`_(no coverage recorded)_` when no run is pre-fetched, so a repo that never ran
`nornir coverage` still renders.
```text
<!-- nornir:gen:start:coverage -->
<!-- nornir:gen:end:coverage -->
```
### `depgraph`
The workspace crate-dependency graph, extracted via
`introspect::depgraph`.
- With `--features docs-export`: rendered to **static SVG** via the
embedded typst engine and written to `<repo_root>/.nornir/assets/depgraph.svg`.
The section body becomes a plain Markdown image link
(``). The SVG path
is **tracked in git** on purpose so the image resolves in the Codeberg
/ GitHub web UI without a build step, and the typst exporter resolves the
same path via the repo-root filesystem root.
- Without `docs-export`: falls back to a self-contained inline SVG (emitted
directly in Rust — no typst, no JS, no build step), so the README still
renders on any host with no diagram engine. Never Mermaid (NUKE-MERMAID).
The image link is emitted **relative to the doc being rendered**: the repo-root
`README.md` gets `.nornir/assets/depgraph.svg`; a source under `.nornir/`
(e.g. `design.md`) gets `assets/depgraph.svg`. Git hosts resolve `<img src>`
relative to the containing file, so a repo-root-relative link would 404 from
inside `.nornir/`. The doc directory is supplied via `Ctx::with_doc_dir`
(set by `assemble_file`); see `sections::doc_relative_asset`.
**Populating the marker (the I1 fix).** The `depgraph` source is `cargo
metadata` — it is *not* the `dep_graph_edges` warehouse table and needs no
`nornir republish`; running `nornir docs render <repo>` is enough. If
`cargo metadata` yields no packages (not a Cargo workspace), the section
renders a visible *"no dependency-graph data … re-run `nornir docs render`"*
note rather than an empty marker, so there are no silent gaps.
Note the two fill paths:
- `README.md` / `CHANGELOG.md` (the `MANAGED_DOCS`) fill from their
`.nornir/<doc>.md` **template** into the repo-root **output**.
- Every *other* `.nornir/*.md` source (`design.md`, `plan.md`, …) **is** the
published file, so `docs render` now fills its markers **in place** via
`layout::render_sources_in_place`. Before this, those sources' `depgraph`
blocks were only expanded transiently inside the PDF/book pipeline and stayed
empty (invisible) on disk.
```text
<!-- nornir:gen:start:depgraph -->
<!-- nornir:gen:end:depgraph -->
```
**Sibling path-deps on a headless server (the DOCS-LIVE / DR1 fix).** `cargo
metadata` for a `[workspace]` repo must load *every member's* dependency
manifests — including sibling **path-deps** like nornir's `skade = { path =
"../skade/skade" }`. On the live server each workspace member is checked out
*flat* under the workspace's `git/` dir, so `../skade/skade` doesn't exist
beside the repo and `cargo metadata` hard-fails with the opaque
`cargo_metadata::MetadataCommand::exec` error — the exact gap that used to break
`Mimir.SecurityScan`. So before docs-gen runs `cargo metadata`,
`depgraph::extract_with_path_deps(repo, scan_root)` (and the `Docs.Render` /
`Docs.Book` server handlers via `prepare_docs_path_deps`) runs
`security::prepare_path_deps` — **clone** (feature `net-scan`) **then symlink**
the siblings from the `git/` scan root, identical to the SBOM scan's prep — so
docs resolves the same siblings the security scan does.
If a sibling is *still* unresolved (airgap build with `net-scan` off, or no
checkout on disk), nornir surfaces a **clear, named error** (DR2) —
`docs render for <repo>: unresolved path-dep sibling(s) <dep> (<path>) — …` with
the fix (check the sibling out under the `git/` dir, or build with `net-scan`)
— never the opaque `Internal: cargo_metadata::MetadataCommand::exec`.
### `symbols binary=PATH [limit=N]`
Top public symbols by size, extracted from DWARF in `PATH` (resolved
relative to the repo root first, then the workspace root, then taken
absolute). `limit` defaults to 20. Closures (`::{{closure}}`) are
filtered out.
### `callgraph binary=PATH [limit=N]`
Most-called callees by inline-edge count from DWARF in `PATH`. `limit`
defaults to 15.
Both `symbols` and `callgraph` need a built binary; run `cargo build`
(or `--release`) before `docs render` if you want fresh tables.
## Adding a new section renderer
1. Add a `match` arm in `sections::render` dispatching to your new
`render_foo(ctx, args)` function.
2. Return a `String` that is the **body only** — the surrounding marker
comments are re-emitted by the dispatcher (with sorted args).
3. Document the section name and accepted args in the doc-comment at
the top of `sections.rs` and add a row to "Built-in section
renderers" above.
4. Add a unit test alongside the existing ones in
`sections.rs::tests` (use the `run_fixture` helper).
The `Ctx` struct (`repo_root`, `workspace_root`, `Option<&BenchRun>`)
is what every renderer gets. If you need more inputs, extend `Ctx`
rather than threading new function args.
## Adding a new managed doc
1. Add the filename to `MANAGED_DOCS` in `layout.rs`.
2. (Optional) extend `layout::scaffold_*` if `init_repo` should write a
starter when the file is absent everywhere.
3. That's it — `init`, `render`, and `check` pick it up automatically.
## Doc taxonomy & the subsystem-doc pattern
nornir's docs follow a deliberate **two-tier reference + planning spine +
generated face** shape. Keep new docs in it so the set stays navigable (and so
agents can slot in without rediscovering the convention).
| role | file(s) | what it is |
|------|---------|------------|
| **face** | `README.md` (generated) | the pitch + status + *measured* numbers; `chmod -w`, rendered from `.nornir/README.md` with marker sections filled from the warehouse |
| **map** | `.nornir/design.md` | one short section per subsystem; pure design, high altitude — the overview chapter |
| **territory** | `.nornir/<subsystem>.md` | one deep reference per subsystem (`docs-generation.md`, `vector.md`, `bencher.md`); self-contained *and* a book chapter |
| **spine** | `plan.md` | single source of truth for *planning* — open work, locked decisions, working order. What's *next*, vs the docs' what *is* |
Every subsystem doc (`territory`) shares one skeleton:
1. one-line **what + a `src/…` source pointer** (so the doc can be checked
against code);
2. **What it is** — plain language;
3. **Decided / Laws (locked)** — rationale + dates, *linking* `plan.md §4`
rather than restating it (one source per fact);
4. **Architecture** — `file:symbol` pointers + tables;
5. **Surfaces** — a CLI + MCP table;
6. **Measured / Gotchas / Current state & gaps**.
The **design ↔ how-to-use** axis runs through each subsystem doc: the top half
is design (what / laws / architecture), the bottom half is how-to-use (surfaces
/ CLI / gotchas / "adding a …"). `design.md` is design-only; `README.md` is
use-only; the subsystem docs carry both — which is why findings about a
subsystem belong in *its* doc, not scattered. Two disciplines keep it honest:
**one source per fact** (decisions live in `plan.md §4`; docs link to it) and
**source pointers everywhere**. And because discovery sweeps `.nornir/*.md`
(see below), *writing* a subsystem doc is the same act as *publishing* it as a
book chapter.
### Where each fact is documented (routing)
When you learn or change something, put it in **one** place by what it is:
| the thing | goes in |
|---|---|
| a locked decision / rejected alternative | `plan.md §4` (docs *link* to it) |
| open work, next steps, status | `plan.md` |
| one-paragraph "what is subsystem X" | `.nornir/design.md` (links to the subsystem doc) |
| deep detail on a subsystem (architecture, surfaces, gotchas, findings) | that subsystem's `.nornir/<x>.md` |
| **every warehouse table / what data lives where** | `.nornir/warehouse.md` |
| the bench framework + vs-legacy + renderers | `.nornir/bencher.md` |
| vector / semantic search internals | `.nornir/vector.md` |
| how docs are generated (this file) + the doc pattern | `.nornir/docs-generation.md` |
| the pitch + measured numbers (reader-facing) | `.nornir/README.md` → generated `README.md` |
Current subsystem docs (`territory`): `docs-generation.md`, `warehouse.md`,
`bencher.md`, `vector.md`. A new subsystem gets a new `.nornir/<name>.md` in the
skeleton above, a one-paragraph pointer in `design.md`, and a routing row here.
## Section classes & the three books (`docs.toml` routing)
Every subsystem doc is written in one of **six section classes**. The class is the
routing key: it decides which of the three PDFs a chapter lands in, and it enforces
*what may be written where* — the discipline that keeps the manual thin.
| class | file suffix | audience | contains | must NOT contain |
|---|---|---|---|---|
| `design` | `-design.md` | builder | architecture, laws, data-flow, `file:symbol` pointers — **only the design we chose** | the *why*, **any mention of rejected/considered alternatives** (→ `reasoning`), unbuilt ideas |
| `guide` | `-guide.md` | techie | CLI, workflow, worked examples | literal keystroke steps, rationale (→ `reasoning`) |
| `idiot` | `-for-idiots.md` | zero-knowledge | *every* step ("open a shell → type → press Enter") | assumed context |
| `reasoning` | `-reasoning.md` | "why?" | rationale, trade-offs, **rejected alternatives**, how-we-got-here | the descriptive *what* (that lives in design/guide) |
| `history` | `-history.md` | record | what changed + when (settled) | open questions |
| `release` | `-release.md` | operator | how to release, gates, status | design internals |
| `unsorted` | `-unsorted.md` | holding pen | genuinely undecided — "is this done, or being done?" notes | anything with a settled home |
**Strictness law (this is what makes the manual thin):** `design` and `guide` contain
ONLY what *is* — every claim source-backed (a `file:symbol` pointer) and current, with
**no *why***. The reasoning is **never lost**: it moves to `reasoning` (why we chose
this, what we rejected, the road here), which a descriptive section may *link* but must
not inline. Settled records go to `history`; genuinely undecided notes to `unsorted`.
A `design` doc describes the code; `reasoning` explains *why* it is that way — keeping
both, but apart, is the whole trick.
**The chosen-design rule (a pattern for *everything*, not just prose):** `design` states
*only the design that was chosen*, as if it were the only one — it must not say "we
picked A **over B**", not "B was rejected because…", not even "alternatives were
considered". Every trace of the road not taken — the forks, the discarded shapes, the
"why not B" — lives in `reasoning`. This is deliberate: a `design` section that argues
for itself is a `design` section that grew long. Chosen design → `design` (short);
rejected design + the argument → `reasoning`.
### The function axis: `f0` = the whole, `f1…fn` = the parts
A class file is organised by **function** (the capabilities from the workspace's
function list: `warehouse`, `docbook`, `release`, …) — one `##` section each. But a list
of parts is not a system, so **every class file opens with `f0`, the combination**: how
all the functions fit together *as one thing*, before the per-function sections describe
each in isolation.
| class | `f0` (the whole) | `f1…fn` (each function) |
|---|---|---|
| `design` | overall architecture — how the functions wire into one system | each function's architecture |
| `guide` | the end-to-end workflow that spans several functions | how to use each function |
| `idiot` | zero → running, the whole thing start to finish | every step, per function |
| `reasoning` | why this overall shape | why each function is the way it is |
So the real matrix is **class × {`f0`, `f1…fn`}**: `design.md` = `f0` (the system) +
`warehouse` + `docbook` + `release` + … ; `guide.md` = `f0` (a full session) +
per-function how-tos; and so on. `f0` is what keeps the manual a *manual* and not a pile
of datasheets. A function declares what it `uses` (the reuse graph — e.g.
`modgunn.uses = [warehouse]`); `f0` is where that graph is drawn.
**Each section carries its own skade SVG.** A section-scoped diagram (the same
Rust/typst engine as `depgraph` — never Mermaid, NUKE-MERMAID) sits at the top of each
class file via a marker, so the design's architecture, the guide's flow, and the
idiot-guide's click-path each get a picture.
### The three books (routing is DATA; `book.rs` is a reusable library)
`book.rs` is a **standalone renderer**, not nornir-specific: it takes a routing config
plus a set of *classed* sources and emits N PDFs, so any repo — or a non-nornir tool —
can drive it. The routing lives in `<repo>/.nornir/docs.toml`:
```toml
# [book.<name>] → one PDF each. `classes` selects which section classes it includes.
[book.manual] # the crisp core — README ⊂ manual
classes = ["readme", "design", "guide", "benchmarks"]
out = "docs/manual.pdf"
[book.idiot_guide]
classes = ["idiot"]
out = "docs/idiot_guide.pdf"
[book.unsorted]
classes = ["reasoning", "history", "unsorted"]
out = "docs/unsorted.pdf"
```
- **`manual.pdf`** — the README intro (with `<mashup>`) + every `design` + every `guide`
+ benchmarks. The 10-minute "what it is and how to use it." README is chapter 1, so
README ⊂ manual.
- **`idiot_guide.pdf`** — every `idiot` section, nothing else.
- **`unsorted.pdf`** — every `reasoning` + `history` + `unsorted` section: the *why*,
the record, and the holding pen — kept out of the manual but never thrown away.
This **replaces** the single `book.pdf`: manual + idiot_guide + unsorted together are the
whole set, split by *audience* instead of one undifferentiated lump. `book::build_book`
becomes `book::build_books(cfg)` returning one document per `[book.*]` entry; a source's
class is read from its filename suffix (above) or an explicit `class = "…"` key, so
routing is pure data and the same library renders any classed doc set — inside nornir or
out.
### The source-first audit (fat → thin)
To move an existing fat doc into this shape, walk from `src/`, never from the prose:
1. **Enumerate what exists** — the symbols, CLI verbs, and features actually in `src/`.
2. **Cross-check each doc claim:**
- in source **and** in doc → keep it in `design`/`guide`;
- in source, missing from doc → a gap; add it;
- in doc, **not** in source → check **git history**:
- implemented then removed → **deprecated** → `history` (or delete);
- never implemented → **a want, not a fact** → `unsorted` / `plan.md`.
3. **Move the reasoning, don't delete it** — "why we chose", rejected alternatives,
narrative → `reasoning`. The descriptive section keeps at most a one-line link to it.
The result: a `design`/`guide` set that is provably source-true, and a `manual.pdf`
that is much thinner because the diary moved to `unsorted.pdf`. This is the same
prune/merge sweep as `workspace_nordisk/.nornir/docs-crispification-plan.md` — that plan
lists *which files*; this section defines *the shape they land in*.
## Whole-documentation book (feature `docs-export`)
`nornir docs book <repo> [--format pdf|html|md] [--out PATH]` →
`book::build_book`. Renders the repo's **entire** doc set into one document:
1. **Discover** sources (`book::discover_sources`):
- every `*.md` under `.nornir/` (the editable sources), ordered
`README.md`, `CHANGELOG.md`, then alphabetically;
- then every other top-level `<repo>/*.md` alphabetically, **excluding**
the generated `MANAGED_DOCS` (their source already comes from `.nornir/`,
so nothing is duplicated).
2. **Expand** each file's marker sections via `sections::rewrite_str` (bench
tables, dep-graph, …). A file with no markers — or a stray marker-looking
line — falls back to its raw text rather than failing the whole book.
3. **Assemble** the chapters into one markdown string. A chapter that doesn't
already start with an `# H1` gets one synthesised from its file name
(`docs-generation` → *Docs Generation*); each H1 is a chapter / page break.
4. **Render** through the same typst pipeline as `export` (cover + TOC +
numbered chapters from `report.typ`).
The current artifact is written to `<repo>/docs/book.<ext>` (override with
`--out`), and the bytes are historized as a row in the Iceberg `doc_exports`
table under the doc name `book`.
## Codeberg Pages publishing (`docs publish-pages`, feature `docs-export`)
`nornir docs publish-pages <repo> [--remote origin] [--render]` →
`pages::stage_pages_branch` (`src/docs/pages.rs`). Publishes the rendered
`docs/book.pdf` to the repo's Codeberg Pages site at
`https://nordisk.codeberg.page/<repo>/`. It stages an **orphan** `pages` branch
holding **exactly three files** — no source, no history carry-over:
```text
.nojekyll (empty marker — tells Codeberg Pages to skip Jekyll)
book.pdf (a copy of <repo>/docs/book.pdf)
index.html (a dark, full-viewport PDF-embed viewer — pages::index_html)
```
- **Pure-Rust, no push.** The tree + orphan commit are built in-process via `gix`
(like `release::publish::commit_release`), pointing `refs/heads/pages`
(`PAGES_BRANCH`) at the new commit **without touching `HEAD` or the working
tree**. `gix` 0.84 has no send-pack, so the force-push is left to the operator /
CI — the command prints the one line to run (`pages::force_push_cmd`), exactly
like `release::publish::push`.
- `--render` (re)builds the book first if `docs/book.pdf` is missing;
`--remote` selects the push remote (default `origin`).
- The `pages` module itself is **not** feature-gated; only the `docs
publish-pages` CLI verb is (`docs-export`). No MCP tool.
## Docs index (separate from the code index)
The rendered docs get their **own** Tantivy index, physically separate from the
workspace code index, at `<repo>/.nornir/cache/docs-index/` (`src/docs/search.rs`):
- **What it indexes** — the *generated* docs only: the repo-root managed docs
(`README.md`, `CHANGELOG.md`) plus the rendered markdown under `<repo>/docs/`
(e.g. `book.md`). The `.nornir/*.md` **templates are deliberately excluded**
here — their `<!-- nornir:gen:* -->` markers are unfilled. Those templates are
instead indexed as *source* by the workspace code index (the walker stopped
hard-excluding `.nornir/`; it now prunes only `.nornir/warehouse/` and
`.nornir/cache/` — see `src/index/walker.rs`).
- **Historization** — `docs::snapshot_docs_index` reuses the snapshot/blob
machinery (`index::snapshot::snapshot_dir_to_iceberg`, parameterized by table
name) to persist the index into the `docs_index_snapshots` /
`docs_index_blobs` Iceberg tables. So the docs index time-travels exactly like
the code index: `docs search --sha <sha>` rehydrates the index for that SHA
straight from the warehouse.
- **Surface** — `nornir docs index <repo>` (build + historize) and
`nornir docs search <repo> "<query>"` (BM25, restores on a cold cache).
## Export pipeline (feature `docs-export`)
`export::export(markdown, meta, format, image_cache_dir, root)`:
1. Parse markdown with `pulldown-cmark`.
2. Translate events into typst markup (`md_to_typst_with`, parametrised by a
`Target` — `Paged` for PDF/SVG, `Html` for the HTML target). Raw spans /
code blocks are fenced with a backtick run longer than anything inside them,
so docs that *show* ``` fences round-trip cleanly.
3. Compile with `typst-as-lib`:
- **PDF / MD** use `report.typ` (cover, TOC, numbered headings, page
furniture) → `typst::layout::PagedDocument` → `typst_pdf::pdf` (MD is a raw
passthrough, no typst).
- **HTML** uses the minimal `web.typ` and typst's **experimental** HTML
target → `typst_html::HtmlDocument` → `typst_html::html`. This is the
low-priority path; MD + PDF are the primary outputs.
4. **Local images** (`.nornir/assets/…`) resolve because a typst filesystem
`root` is mounted at the repo root; local link paths are normalised to
project-root form (a leading `/`). A `<img src="…">` authored as raw HTML
(for git-host centering) is recovered into a real `#image`.
5. **Remote images** (``) are downloaded with `ureq` + rustls
into `.nornir/cache/images/` (content-hashed) and embedded as bytes. Pass
`None` for `image_cache_dir` to skip remote fetches (placeholder text
instead) — handy for offline / airgapped runs.
Exports are historized in the Iceberg `doc_exports` table
(`docs::record_doc_export`, schema in `warehouse::iceberg_schema::doc_exports`):
one row per distinct `(repo, doc_name, version, format, sha256)` with the full
bytes inline, queryable via `nornir docs history` / the `docs_history` MCP tool.
The current file copy lives under `<repo>/docs/`; the warehouse holds the
history (and can reconstruct any past version).
## Release gate
`release_gate_docs_fresh` runs `layout::check_all` against the repo and
fails the release if any managed artifact has drifted from its
`.nornir/` source. Enable it per repo under `[repo.<name>.gates]` in
`nornir.toml`.
## Auto-trigger (keeping generated docs fresh)
`docs render` is **manual**. To stop the published README/CHANGELOG silently
drifting from the warehouse/code between renders (we saw stale znippy/holger
test counts), two complementary mechanisms regenerate them automatically. They
split by *cost*: the README markdown is cheap + deterministic and refreshes on
**every commit**; the SVG charts + depgraph + book PDF are heavy and
non-deterministic and refresh only at **release**.
### (1) Pre-commit hook — instant, markdown-only
`nornir docs install-hooks <repo>` (`docs::hooks::install_hooks`, MCP
`docs_install_hooks`) writes a **tracked** POSIX-sh hook to
`<repo>/.githooks/pre-commit` and runs `git config core.hooksPath .githooks`
for that repo. `.githooks/` is tracked + shareable (unlike the untracked
per-clone `.git/hooks/`). The hook:
- runs `nornir docs render <repo> --markdown-only` — **README.md / CHANGELOG.md
only**, never the depgraph SVG / bench-chart SVGs / book PDF (see the
`--markdown-only` flag below), then `git add`s the regenerated managed docs so
the commit includes them;
- **degrades gracefully** — if `nornir` isn't on `PATH`, the warehouse is
unreachable, or render fails for *any* reason, it prints a warning and
**exits 0**. It never blocks an unrelated commit. Docs freshness is a
convenience here; the *hard* assertion is the release path below.
- is **idempotent** — re-running `install-hooks` rewrites the same bytes and
leaves `core.hooksPath` alone when already set.
The `--markdown-only` (alias `--no-assets`) flag on `docs render` sets
`Ctx::no_assets` (via `Ctx::markdown_only()`), which makes `render_depgraph` /
`render_bench_chart` emit their image links **without (re)writing** the
`.nornir/assets/*.svg` files (the committed assets are reused) and forces the
config to markdown so no `book.pdf` is built. That is what keeps the commit path
cheap and the markdown deterministic.
Source: `src/docs/hooks.rs` (`pre_commit_script`, `install_hooks`,
`set_hookspath`), the `Ctx::no_assets` field + `markdown_only()` in
`sections.rs`, and the `DocsOp::{Render(--markdown-only),InstallHooks}` wiring in
`src/bin/nornir.rs` (`run_docs_install_hooks`, `run_docs_render`).
### (2) Release render-and-stage — authoritative backstop (incl. SVG/PDF)
The release no longer just *checks* drift — it **renders + stages** first.
`release::gate::render_and_stage_docs(layout, ctx)` runs `layout::render_all`
+ `render_sources_in_place`, (re)builds the SVG/depgraph assets, builds the book
PDF via `book::build_book` (under `docs-export`), and `git add`s every changed
artifact (best-effort; a non-git checkout is reported, not fatal). It runs
**before** the `docs_fresh` check in the gate path (`run_gate_all` in
`src/bin/nornir.rs`), so a release always ships a fresh README + book + SVG, and
the subsequent `layout::check_all` is the final assertion — any drift *after* a
render is a real bug, not staleness. It is opt-in per repo under the same
`[repo.<name>.gates].docs_fresh` flag (rendering is the active half of the
freshness contract). The promote/publish path additionally does a full
`run_docs_render` + `docs book` before the bump commit, so the published crate
ships generated docs.
| mechanism | trigger | scope | blocks? | source |
|-----------|---------|-------|---------|--------|
| pre-commit hook | every `git commit` | README/CHANGELOG markdown only | never (exit 0 on any failure) | `docs::hooks` |
| release render-and-stage | release gate / promote | README + CHANGELOG + assets + book PDF | yes — `docs_fresh` check is the final assertion | `release::gate::render_and_stage_docs` |
## CLI / MCP surface
| MCP tool | CLI | Calls |
|------------------|--------------------------------|-----------------------------|
| `docs_init` | `nornir docs init <repo>` | `layout::init_repo` |
| `docs_render` | `nornir docs render <repo> [--markdown-only]` | `layout::render_all` |
| `docs_check` | `nornir docs check <repo>` | `layout::check_all` |
| `docs_install_hooks` | `nornir docs install-hooks <repo>` | `hooks::install_hooks` |
| `docs_export` | `nornir docs export <repo> …` | `export::export_repo` |
| `docs_book` | `nornir docs book <repo> …` | `book::build_book` |
| — | `nornir docs publish-pages <repo> [--remote] [--render]` | `pages::stage_pages_branch` |
| `docs_history` | `nornir docs history <repo> …` | `docs::list_doc_exports` |
| — | `nornir docs index <repo>` | `docs::build_docs_index` |
| — | `nornir docs search <repo> q` | `docs::search_docs` |
## Gotchas
- **Never edit the top-level `README.md` / `CHANGELOG.md` directly** —
your edits are overwritten on the next `docs render`. The `chmod -w`
bit (set by `guard_apply` on the artifact paths) is there to remind
you.
- The dispatcher rewrites markers with **sorted args**. Don't rely on
arg ordering in your source — the render will normalise it and your
next `docs check` will pass.
- Markers inside fenced code blocks are ignored. If you need to *show*
a marker in prose without fencing, escape the leading `<` or break
the literal across an inline span — otherwise `parse_markers` will
treat it as real.
- The depgraph SVG path (`.nornir/assets/depgraph.svg`) is **committed**, not
gitignored — losing it breaks the rendered image link in the web UI.
Only `.nornir/cache/` is gitignored; `.nornir/assets/` is tracked.
- DWARF-backed sections (`symbols`, `callgraph`) silently produce empty
tables if the binary isn't built. Build first, then render.