# knut — design (f0, the whole)
> **Class:** `design` · **Function axis:** `f0` — how the seven functions wire into
> one system. Each function has its own `*-design.md` chapter; this file is the map that
> joins them. Strict: it states *only the shape we run* — every *why* and every rejected
> alternative lives in [`reasoning.md`](reasoning.md).
knut is a *knytpunkt* (Swedish: hub/junction) over property graphs: it carries graph data
the whole way from the columnar world (Arrow, Iceberg, GraphAr files) to a live, queryable
graph (FalkorDB) and back out over Arrow Flight SQL — **one Arrow pipe, no JVM required**.
The through-line is **Arrow 58**, one tree workspace-wide (`Cargo.toml` `[workspace.dependencies]`),
so every seam is zero-copy or a plain `Arc` bump.
## The seven functions
| function | crate | one-line | chapter |
|---|---|---|---|
| **graphar** | `graphar` | the GraphAr `gar/v1` chunked-file substrate over `object_store` | [graphar-design.md](graphar-design.md) |
| **falkordb** | `graphar-falkordb` | Arrow `RecordBatch` → FalkorDB via `UNWIND … CREATE/MERGE` | [falkordb-design.md](falkordb-design.md) |
| **flight** | `graphar-flight` | Cypher over Arrow Flight SQL; graph → GraphAr / Iceberg export | [flight-design.md](flight-design.md) |
| **bifrost** | `knut-bifrost` | watch lakehouse tables, materialise them as a graph by CDC | [bifrost-design.md](bifrost-design.md) |
| **thund** | `knut-thund` | Arrow-native streaming dataflow engine, pluggable backend | [thund-design.md](thund-design.md) |
| **pipelines** | `knut-pipelines` | viewer for Spark Declarative Pipelines + Spark Connect jobs | [pipelines-design.md](pipelines-design.md) |
| **popviz** | `knut-popviz` (+`knut-popsim`) | population bulk-loader + headless-introspectable GUI | [popviz-design.md](popviz-design.md) |
## The reuse graph (how they fit)
The functions are a layered stack, not a flat list. Each declares what it `uses`; nothing
below depends on anything above it:
```text
graphar (leaf — object_store only)
├─ falkordb uses graphar (_gar_id identity)
│ ├─ flight uses falkordb, graphar (Flight SQL front + export)
│ ├─ bifrost uses falkordb, graphar (table→graph CDC fabric)
│ └─ popviz uses falkordb, graphar, popsim
pipelines (standalone Spark viewer; facett adapter optional)
└─ thund uses pipelines (IR ⊃ DataflowGraph; SparkBackend reuses the connector)
```
- **`_gar_id` is the spine identity.** graphar mints a stable internal id per vertex; edges
reference endpoints by it (`graphar/src/writer`, `graphar/src/reader`). That single key is
what makes falkordb's MERGE, flight's typed vertex/edge schemas, and bifröst's
MERGE-both-endpoints all line up. See [falkordb-design.md](falkordb-design.md).
- **The lakehouse → graph path** is bifröst: `TableSource` → transform fabric → `FalkorDbSink`
MERGE (`knut-bifrost/src/listener.rs:Listener::sync_once`). The **graph → lakehouse** path is
flight's export (`graphar-flight/src/export.rs`, feature `skade` for Iceberg).
- **The two engines** are siblings: bifröst (table→graph CDC) and Þund (batch+streaming
dataflow). Þund lowers its IR either to a native Arrow/DataFusion backend or, via
`Pipeline::to_sdp`, onto the **pipelines** function's Spark Connect connector — so a Þund run
and a Spark run render identically in the same viewer.
- **Two viewers, one introspection contract.** Both the pipeline viewer
(`knut-pipelines`) and the population viewer (`knut-popviz`) expose their entire UI state as
JSON from a `state_json()` that lives in the **default (no-`gui`) build**, written every frame
to an env-var-named file (`$KNUT_VIEWER_STATE` / `$KNUT_POPVIZ_STATE`). So both GUIs are
robot-tested headlessly. **Law:** any new visible control MUST be added to `state_json()`.
## The Arrow-58 invariant (the seams)
| seam | where | shape |
|---|---|---|
| chunk file ↔ Arrow | `graphar/src/io/mod.rs:read_chunk`/`write_chunk` | five formats, one Arrow schema |
| Arrow → graph | `graphar-falkordb/src/cypher.rs:ColRenderer` | per-column downcast, one pre-sized buffer |
| graph → Arrow | `graphar-flight/src/convert.rs:response_to_batch` | FalkorDB RESP → `RecordBatch` |
| Arrow → Iceberg | `graphar-flight/src/export.rs:bridge_56_to_57` | identity `clone()` — knut + skade share Arrow 58 |
| Iceberg → Arrow | `knut-bifrost/src/source.rs:IcebergSource` (feature `skade`) | reverse read of an Iceberg snapshot |
| Spark ↔ Arrow | `knut-pipelines/src/client.rs` | Spark Connect transports results as Arrow batches |
Because there is one Arrow major across the workspace, the skade/Iceberg boundary
(`bridge_56_to_57`) is a plain `Arc` bump, not a copy — the export and its round-trip test read
the same seam.
## Locked laws (system-wide)
- **arrow 58 / one tree** so `orc-rust` and `spark-connect` agree; `spark-connect` carries its
own private tonic, gated behind a feature, so the tonic majors never mix.
- **MERGE, keyed on `_gar_id`, both endpoints on every edge** — tables sync in arbitrary
order, so a placeholder endpoint node is enriched later by the vertex MERGE on the same id;
this is also what makes crash-replay idempotent.
- **`REFRESH TABLE` before every snapshot poll** — Spark caches Iceberg metadata per session.
- **"Redis" means Valkey 9+** — the graph store and the shared `SyncState` ledger run on
Valkey (Redis-API/RESP-conformant); `redis://` is only the wire scheme.
- **Heavy stacks are opt-in features** — `grpc`, `gui`, `skade`, `native`, `spark`, `wasm`,
`flight`, `metrics-http` are all off by default; the lean build is pure model + core.
- **Every shipping binary embeds version AND git-hash** — `--version` prints
`<bin> v<pkg-version> (<short-git-sha>)`. The SHA is stamped by each shipping crate's
`build.rs` (`knut-bifrost`, `knut-pipelines`, `knut-popviz`), which reads the repo's git
metadata **pure-fs / zero-shell** (walk up to `.git`; loose ref → `packed-refs`; detached
HEAD; linked-worktree `gitdir:`/`commondir`), emits `cargo:rustc-env=KNUT_GIT_HASH`, and the
crate lib exposes `GIT_HASH` + `VERSION_LINE` (`concat!` of `CARGO_PKG_VERSION` + the hash).
clap bins wire `version = <crate>::VERSION_LINE`; `knut-popviz` (no clap) prints it on
`--version`/`-V`. Applies to `bifrost`, `knutpipe`, `knut-spark-server`, `knut-popviz` — NOT
the `rel2graph-timing` bench fixture.
## Surfaces (system)
| binary / entry | function | verb |
|---|---|---|
| `knutpipe` | pipelines | `show` / `watch` / `gui` / `pipeline` |
| `bifrost` | bifrost | `run` / `gen-sample` (feature `cli`) |
| `knut-popviz` | popviz | GUI loader (feature `gui`) |
| Flight SQL server | flight | `run_server` / `run_server_with` |
| `python/knut` | — | thin Flight SQL client → pandas / NetworkX ([networkx-export.md](networkx-export.md)) |
| `xtask` | — | the nornir bench suite ([benching-benchmarks.md](benching-benchmarks.md)) |
## See also
- The end-to-end **workflow** across functions: [guide.md](guide.md) (f0).
- Zero → running the whole thing: [for-idiots.md](for-idiots.md) (f0).
- Why this overall shape, and every road not taken: [reasoning.md](reasoning.md) (f0).
## Which recipe reader this repo uses, and why (2026-09-20: the code is the source)
**Reader:** `nornir-build-thing` **0.1.18**, REGISTRY, from `crates/knut-build-thing`.
knut no longer READS a recipe at all. The `impl BuildThing` in that crate IS the
declaration, and `.nornir/build-variants.toml` is EMITTED from it
(`cargo run -p knut-build-thing -- emit`) — code → toml, one-directional, never
read back as authority. Until 2026-09-20 it ran the other way: the toml was
`include_str!`-ed into `crates/knut-build` and parsed back, so the file was the
build thing and the Rust was a reader of it. The crate's NAME changed with the
direction, because dwarves' forge finds a recipe as `<repo>-build-thing` (a
cdylib) and `knut-build` matched neither — knut's code road was closed by name.
The dependency is the REGISTRY and never a `../edda` path: a path dep is read at
manifest-load time, so on a box without the sibling it aborts resolution for the
WHOLE workspace (measured 2026-09-10). 0.1.18 is the first published version that
carries the emitter.
**Why:** in-constellation; recipe is schema 1. The requirement had been the floating
`"0.1"`, and knut's graph carried a registry copy at 0.1.8 beside the path copy at
0.1.9 — two readers of the same file, one repo. The registry half closes on the
release cascade (the pull comes through published transitives).
The rule itself — `[nornir-build-thing] schema = N`, a refusal that names the
version to upgrade to, and a loader that refuses to DROP a key it cannot read —
lives once, in **edda `.nornir/build-design.md`, "THE COMPATIBILITY RULE"**. The
cross-version guard that would have caught the 2026-08-30 skew is
`edda/nornir-build-thing-skew`.