# The docbook pipeline — design
> **Class:** `design` · **Function axis:** `f0` (the pipeline as one system) + `f1` `.nornir/` layout · `f2` markers & section renderers · `f3` class taxonomy & book routing · `f4` export/typst · `f5` docs index + search · `f6` Codeberg Pages.
> Strict: this file states *only what IS*, source-backed. Every *why*, every rejected alternative, lives in [`docbook-reasoning.md`](docbook-reasoning.md). How to drive it is in [`docbook-guide.md`](docbook-guide.md).
## f0 — the pipeline (the whole)
`docbook` turns hand-edited `.nornir/*.md` **sources** into a repo's published
documentation set through **one typst engine** — no JS, no shellout, no Mermaid.
Source code: [`src/docs/`](../src/docs/) — `mod.rs` re-exports the whole surface.
The pipeline is four stages, left to right:
1. **Sources** (`f1`) — every `.nornir/*.md` is the source of truth; the repo-root
`README.md` / `CHANGELOG.md` are regenerated outputs, not sources.
2. **Marker expansion** (`f2`) — `sections::rewrite` (`src/docs/sections.rs:352`)
replaces each `<!-- nornir:gen:start:X -->…<!-- nornir:gen:end:X -->` region with
a freshly rendered table/graph, reading pre-fetched data off a `Ctx`.
3. **typst** (`f4`) — `export::export` (`src/docs/export.rs:106`) compiles the
expanded markdown to PDF/HTML/MD via an embedded typst engine.
4. **Outputs** (`f3`) — `book::build_books` (`src/docs/book.rs:537`) routes the
*classed* sources into the README plus three audience PDFs (`manual` /
`idiot_guide` / `unsorted`), and `search` builds a dedicated docs index over the
generated docs; `pages` publishes the book to Codeberg Pages.
**Reuse map (what docbook `uses`):** marker renderers pull from the **warehouse**
function (`Ctx::history` bench runs, `Ctx::coverage`) and from `introspect::depgraph`
/ DWARF; `book.rs` is a standalone renderer that takes a routing config plus classed
sources, so the same library renders any classed doc set. The **release** function
consumes docbook through `release::gate::render_and_stage_docs` (freshness gate). The
CLI/MCP verbs in `f0`'s surface (see [guide](docbook-guide.md)) are the only entry points.
| stage | entry symbol | file:symbol |
|---|---|---|
| render managed docs | `render_all` | `layout.rs:211` |
| fill `.nornir/*.md` in place | `render_sources_in_place` | `layout.rs:239` |
| expand one file's markers | `assemble_file` | `sections.rs:304` |
| dispatch one marker | `render` (match arm) | `sections.rs:258` |
| compile to a format | `export::export` | `export.rs:106` |
| route classed sources → N PDFs | `build_books` | `book.rs:537` |
| build the docs index | `build_docs_index` | `search.rs:63` |
| publish to Pages | `stage_pages_branch` | `pages.rs:108` |
## f1 — the `.nornir/` layout & `MANAGED_DOCS`
Every repo docbook owns has a `.nornir/` directory holding the sources; the top-level
artifacts are regenerated (`RepoLayout`, `layout.rs:119`).
```text
repo/
├── .nornir/
│ ├── README.md / CHANGELOG.md ← SOURCE (edited)
│ ├── *-{design,guide,reasoning}.md, *-for-idiots.md ← classed sources
│ ├── assets/ ← TRACKED media (depgraph.svg, bench charts, figures)
│ ├── warehouse/ ← Iceberg data (gitignored)
│ ├── cache/ ← transient: fetched images, docs index (gitignored)
│ └── docs.toml ← per-repo render + book routing config
├── README.md / CHANGELOG.md ← GENERATED, chmod -w
└── docs/ ← GENERATED rendered docs (book/manual/idiot_guide/unsorted PDFs)
```
- `MANAGED_DOCS = ["README.md", "CHANGELOG.md"]` (`layout.rs:44`). These fill from
their `.nornir/<doc>.md` template into the repo-root output. Adding one is a
one-line change to that list — it then gets `init`/`render`/`check` for free.
- Any *other* `.nornir/*.md` **is** the published file; `render_sources_in_place`
(`layout.rs:239`) fills its markers **in place**.
- Path resolution: `RepoLayout::source_of` / `output_of` / `docs_dir` / `export_path`
(`layout.rs:147`–`161`). All nornir-managed media lives under `.nornir/assets/`
(tracked, one namespace).
- Two output destinations: current artifacts land as files under `<repo>/docs/`
(tracked, link targets); their full history is historized as rows in the Iceberg
`doc_exports` table.
**Generated header.** Every managed artifact begins with
`GENERATED_HEADER_PREFIX` / `_SUFFIX` (`layout.rs:38`–`40`) — a loud
`⚠ GENERATED … DO NOT EDIT` banner. `write_atomic` sets the read-only bit; `init_repo`
(`layout.rs:393`) strips the header on re-migration so re-init is idempotent.
**Render outputs (`docs.toml`).** `DocsRenderCfg` (`layout.rs:71`) with
`outputs = ["markdown"|"pdf"|"all"]` selects what `render` emits:
| `outputs` | render emits | writes markdown? | predicate |
|---|---|---|---|
| `["markdown"]` (default) | managed `README.md`/`CHANGELOG.md` | yes | `wants_markdown` (`:103`) |
| `["pdf"]` | the book → `docs/book.pdf` only | no | `wants_pdf` (`:110`) |
| `["all"]` | both | yes | — |
`RenderReport` (`layout.rs:168`) is a no-op when the output is byte-identical.
## f2 — marker syntax & built-in section renderers
**Syntax.** Inside any `.nornir/` source, paired HTML comments mark a region that
`sections::rewrite` overwrites each render:
```text
<!-- nornir:gen:start:symbols binary=target/debug/holger limit=20 -->
<!-- nornir:gen:end:symbols -->
```
Parsed by `parse_markers` (`sections.rs:190`) into `Marker` (`sections.rs:148`).
Rules: start/end names must match; unclosed start is a hard error with line numbers;
markers inside fenced code blocks (```` ``` ````/`~~~`) are ignored; args are
re-emitted in **sorted order** so output is deterministic.
**Renderers** (dispatched in `render`, `sections.rs:258`; each returns the body only,
the dispatcher re-emits the marker comments):
| name | file:symbol | data source |
|---|---|---|
| `bench` | `render_bench` (`sections.rs:393`) | latest `BenchRun` |
| `bench_history [limit=N] [metric=NAME]` | `render_bench_history` (`:429`) | `Ctx::history` |
| `benches [name=] [best=] [winner=row] [compare=]` | `render_benches` (`:540`) | latest `BenchRun`, bolds winner cell |
| `bench_chart metric=M name=PAT` | `render_bench_chart` (`:1438`) | native-typst bar chart → SVG asset |
| `bench_hero name= fast= slow=` | `render_bench_hero` (`:807`) | biggest per-item ratio callout |
| `mashup [self=] [exclude=] [metric=] [top=]` | `render_mashup` (`:1055`) | auto-discovers ours+rivals |
| `tests [crate=] [kind=]` | `render_tests` (`:1581`) | `test_inventory::scan` (syn) |
| `coverage [worst=N]` | `render_coverage` (`:1622`) | `Ctx::coverage` |
| `depgraph` | `render_depgraph` (`:1797`) | `cargo metadata` → SVG |
| `symbols binary=PATH [limit=N]` | `render_symbols` (`:1916`) | DWARF |
| `callgraph binary=PATH [limit=N]` | `render_callgraph` (`:1955`) | DWARF inline edges |
`Ctx` (`sections.rs:62`) carries `repo_root`, `workspace_root`, `history`, `coverage`,
`doc_dir`, `no_assets`; builders `new`/`with_history`/`with_coverage`/`with_doc_dir`/
`markdown_only` (`sections.rs:103`–`141`). `tests` groups by Rust-standard categories
parsed with `syn` in `test_inventory::scan` (`test_inventory.rs`). `depgraph` under
`docs-export` renders a static SVG to `.nornir/assets/depgraph.svg` (tracked, so it
resolves on git hosts); without the feature, a self-contained inline SVG emitted in
Rust (`svg::render_graph_svg`, `svg.rs:30`) — never Mermaid. Its image link is emitted
relative to the doc being rendered via `Ctx::with_doc_dir` / `doc_relative_asset`.
## f3 — class taxonomy & the three books
Every source has a **class** — the routing key that decides which PDF a chapter lands
in. Class is read by `classify` (`book.rs:409`): an explicit `<!-- nornir:class: X -->`
override wins (`explicit_class`, `book.rs:474`), else the filename **suffix**, else the
bare stem, else `"other"` (which no book claims → omitted and logged).
| class | filename suffix | audience | book |
|---|---|---|---|
| `design` | `-design.md` | builder | manual |
| `guide` | `-guide.md` | techie | manual |
| `idiot` | `-for-idiots.md` / `-for-dummies.md` / `-idiots.md` / `-idiot-guide.md` | zero-knowledge | idiot_guide |
| `reasoning` | `-reasoning.md` | "why?" | unsorted |
| `history` | `-history.md` | record | unsorted |
| `benchmarks` | `-benchmarks.md` / `-bench.md` | measured | manual |
| `readme` / `changelog` | bare `README.md` / `CHANGELOG.md` | face | manual (README ⊂ manual) |
| `unsorted` | `-unsorted.md` | holding pen | unsorted |
Suffix table: `book.rs:420`–`432`. `canonical_class` (`book.rs:455`) maps a token to a
static class name.
**Routing is data.** `BookRoute` (`book.rs:384`: `name`, `classes`, `out`) is loaded
from `[book.<name>]` in `.nornir/docs.toml` by `load_routing` (`book.rs:488`); absent a
config, `default_routing` (`book.rs:509`) supplies the three-book default:
```toml
[book.manual] # classes = ["readme","changelog","design","guide","benchmarks"]
out = "docs/manual.pdf"
[book.idiot_guide] # classes = ["idiot"]
out = "docs/idiot_guide.pdf"
[book.unsorted] # classes = ["reasoning","history","unsorted"]
out = "docs/unsorted.pdf"
```
`build_books` (`book.rs:537`) classifies every discovered source once, warns on any
class no book claims (`book.rs:557`), then emits one `BuiltBook` (`book.rs:369`) per
`[book.*]` entry whose class set matches ≥1 source (no empty PDF). `build_book`
(`book.rs:614`) is the single-document path (whole-set `book.pdf`); `collect_chapters`
(`book.rs:81`) discovers + expands the sources into ordered `Chapter`s (`book.rs:34`),
and `assemble_markdown` (`book.rs:352`) joins them (each H1 = a chapter/page break, a
missing H1 is synthesised from the filename).
The **function axis** organises each class file: `f0` opens with the whole system,
then `f1…fn` describe each function; a function declares what it `uses` and `f0` draws
that graph.
## f4 — the export / typst pipeline (feature `docs-export`)
`export::export(markdown, meta, format, image_cache_dir, root)` (`export.rs:106`):
1. Parse markdown with `pulldown-cmark`.
2. Translate events to typst markup via `md_to_typst_with` (`export.rs:447`),
parametrised by `Target` (`export.rs:77`: `Paged` for PDF/SVG, `Html` for HTML).
Raw spans / code blocks are fenced with a backtick run longer than anything inside,
so docs that *show* fences round-trip.
3. Compile with `typst-as-lib`: PDF/MD use `report.typ` (cover, TOC, numbered headings)
→ `PagedDocument` → `typst_pdf::pdf`; HTML uses `web.typ` and typst's experimental
HTML target. `DocFormat` (`export.rs:37`) with `parse` (`:51`) / `extension` (`:60`).
4. Local images (`.nornir/assets/…`) resolve because a typst filesystem `root` is
mounted at the repo root (`ImageResolver`, `export.rs:264`); a raw-HTML `<img src>`
is recovered into a real `#image`.
5. Remote images (``) are downloaded with `ureq`+rustls into
`.nornir/cache/images/` (content-hashed) and embedded; `None` for `image_cache_dir`
skips fetches (offline).
`export_repo` (`export.rs:795`) renders the assembled top-level README. Exports are
historized in the Iceberg `doc_exports` table by `record_doc_export`
(`warehouse.rs`) — one row per `(repo, doc_name, version, format, sha256)` with bytes
inline; `list_doc_exports` (`warehouse.rs`) reads them back. `resolve_version`
(`book.rs:883`) supplies the version stamped on the cover.
## f5 — docs index & BM25 search
The rendered docs get their **own** Tantivy index, separate from the code index, at
`<repo>/.nornir/cache/docs-index/` (`docs_index_dir`, `search.rs:27`):
- **Indexes the generated docs only** — repo-root managed docs plus `<repo>/docs/*.md`
(`discover_doc_files`, `search.rs:46`). The `.nornir/*.md` templates are excluded here
(their markers are unfilled) — they are indexed as *source* by the workspace code
index instead.
- `build_docs_index` (`search.rs:63`) builds it; `snapshot_docs_index` (`search.rs:75`)
persists it into the `docs_index_snapshots` / `docs_index_blobs` Iceberg tables, so it
time-travels like the code index; `restore_docs_index` (`search.rs:96`) rehydrates on a
cold cache (optionally pinned to a `--sha`); `search_docs` (`search.rs:114`) runs BM25.
## f6 — Codeberg Pages publishing (feature `docs-export`)
`stage_pages_branch` (`pages.rs:108`) publishes `docs/book.pdf` to
`https://nordisk.codeberg.page/<repo>/`. It stages an **orphan** `pages` branch
(`PAGES_BRANCH`, `pages.rs:33`) holding exactly three files: `.nojekyll` (`NOJEKYLL`,
`pages.rs:45`), `book.pdf`, and `index.html` (a dark full-viewport PDF viewer,
`index_html`, `pages.rs:51`). The tree + orphan commit are built in-process via `gix`
pointing `refs/heads/pages` without touching HEAD or the working tree; the force-push
is left to the operator — the command is printed by `force_push_cmd` (`pages.rs:78`).
`PagesBuild` (`pages.rs:87`) reports the staged file names. The `pages` module is not
feature-gated; only the CLI verb is.
## Freshness: hooks & the release gate
- **Pre-commit hook** — `install_hooks` (`hooks.rs:102`) writes a tracked
`.githooks/pre-commit` (`pre_commit_script`, `hooks.rs:53`) and runs
`git config core.hooksPath .githooks` (`set_hookspath`, `hooks.rs:166`). The hook runs
`nornir docs render <repo> --markdown-only` (README/CHANGELOG only, no SVG/PDF; sets
`Ctx::no_assets` via `markdown_only`), `git add`s the result, and exits 0 on any
failure (never blocks a commit). `InstallReport` (`hooks.rs:37`).
- **Release render-and-stage** — `release::gate::render_and_stage_docs` runs
`render_all` + `render_sources_in_place`, rebuilds SVG/depgraph assets, builds the book
PDF, and `git add`s every changed artifact; then `check_all` (`layout.rs:302`) is the
final drift assertion. Opt-in per repo under `[repo.<name>.gates].docs_fresh`.