# knut-popviz — design
> **Class:** `design` · **Function axis:** `f0` headless-core + thin-GUI-shell · `f1` the shared knut-popsim model · `f2` population gen to Arrow · `f3` introspectable state · `f4` the load driver · `f5` the egui app.
> Strict: this file states *only the shape we run*. The system-wide *why* lives in [`reasoning.md`](reasoning.md); this function ships design + [guide](popviz-guide.md).
>
> **Reuse:** `popviz.uses = [falkordb, graphar, popsim]` — see [falkordb](falkordb-design.md), [graphar](graphar-design.md); it drives `graphar-falkordb`'s `FalkorDbLoader` and wraps the shared `knut-popsim` model (`f1` below).
## f0 — headless core, thin GUI shell (the whole)
knut-popviz is a demonstrator + test harness for the **bulk-insert path**: it
generates the synthetic 12-label Swedish-population graph as Arrow and loads it
into FalkorDB through `graphar-falkordb`'s `FalkorDbLoader` (UNWIND-batched
Cypher — **no `falkordb_bulk_loader`**), while a desktop egui app shows a progress
bar per label and live `rows/s`. `crates/knut-popviz/src`.
The whole design is one split:
**The lib (`PopvizState`, `vertex_batches`, `run_load`) has no egui types and
lives in the default build; the egui binary (feature `gui`) is a pure rendering
of `PopvizState`.** So the load logic and the introspection are unit-tested with
no display, and the GUI is a thin shell over the same shared state
(`Arc<Mutex<PopvizState>>`) the driver mutates.
Two laws hold it:
| law | statement | enforced by |
|---|---|---|
| **same introspection contract as the pipeline viewer** | `PopvizState::state_json()` is the external-state oracle; the GUI dumps it to `$KNUT_POPVIZ_STATE` ~every 300 ms | `state_json` (`lib.rs:132`); [`pipelines-design.md#f6`](pipelines-design.md) shares the contract |
| **`--autostart` for robots** | `--autostart` / `KNUT_POPVIZ_AUTOSTART` kicks the load off on the first frame — a headless run loads + writes the JSON with no click | `main.rs:57`,`:102` |
**Rule:** any new visible field MUST be added to `state_json()`.
## f1 — the shared knut-popsim model (`knut-popsim`)
The **single source of truth** for the demo population — a pure, wasm-clean,
deterministic model, re-exported here (`pub use knut_popsim::{geo, LABELS,
EKONOMISKT_BISTAND_RATE}`, `lib.rs:20`,`:36`) and shared with the facett wasm demo.
`crates/knut-popsim/src`.
It has **no FalkorDB, Arrow, tonic, reqwest, tokio, or filesystem deps** (serde is
the only optional one), so it compiles for `wasm32-unknown-unknown`. Every
consumer converts the same `PopGraph` to its own world.
| item | signature | where |
|---|---|---|
| Labels | `pub const LABELS: [(&str, i64); 12]` — the 12 register labels A–L + scale-1.0 counts | `lib.rs:47` |
| Tie rates | `PERSON_TIE_PROB: [f64; 12]`, `PERSON_REL: [&str; 12]` | `lib.rs:70`,`:87` |
| Sample | `pub fn simulate(SimParams) -> PopGraph` | `lib.rs:243` |
| Schema | `pub fn schema() -> PopGraph` (12 label nodes, 11 hub + 4 cross edges) | `lib.rs:288` |
| Params | `pub struct SimParams { n_persons: u32, n_kommun: u16, seed: u64 }` | `lib.rs:209` |
| Graph | `pub struct PopGraph { nodes: Vec<PopNode>, edges: Vec<PopEdge> }`; `PopNode { label: u8, kommun: u16 }`, `PopEdge { src, dst, rel: &'static str }` | `lib.rs:193`,`:114`,`:135` |
| Geo | `pub mod geo` (`address_for`, `is_ekonomiskt_bistand`, `cluster_hotspots`, `CITIES`, `SWEDEN_BBOX`) | `geo.rs` |
**Laws:** *pure + wasm-clean* (the build must pass for `wasm32-unknown-unknown`,
default *and* `serde`); *determinism is the contract* (`simulate` uses a seeded LCG,
no `rand` — same `SimParams` → byte-identical graph; geo uses splitmix64 seeded by
node id, so a person always lands in the same place); *generic graph, not
backend-specific* (no facett/FalkorDB types). `serde` (optional) derives on the
model; `PopEdge`'s `&'static str rel` interns on deserialize (`intern_rel` over
`ALL_RELS`, `lib.rs:153`,`:146`).
## f2 — population gen to Arrow (`src/lib.rs`)
`vertex_batches(label, count, offset)` (`lib.rs:198`) emits one label's vertices
as Arrow `RecordBatch`es of ≤`CHUNK` (25 000) rows. Every batch carries `_gar_id`
(`Int64`) + `idx` + `name`; **geo-bearing labels** (`Person`/`Address`/`Property`,
via `label_has_geo`, `lib.rs:40`) additionally carry `lon`/`lat` (`Float64`) +
`kommun` (`Utf8`) from `knut-popsim::geo::address_for`, and `Person` carries the
`ekonomiskt_bistand` (`Boolean`) flag from `geo::is_ekonomiskt_bistand`. Those
columns flow verbatim through skade→Iceberg and FalkorDB.
## f3 — the introspectable state (`src/lib.rs`)
`PopvizState` (`lib.rs:79`) is the whole observable truth, shared
`Arc<Mutex<…>>` between the driver thread and the UI.
| item | shape | where |
|---|---|---|
| `PopvizState` | `{ phase, labels: Vec<LabelProgress>, elapsed_s, error, target }` | `lib.rs:79` |
| `Phase` | `Idle`/`Vertices`/`Indexes`/`Done`/`Failed` (`label`) | `lib.rs:56`,`:65` |
| `LabelProgress` | `{ name, loaded, total }` per label | `lib.rs:48` |
| derived | `total`/`loaded`/`fraction`/`rows_per_s` | `lib.rs:108`–`:122` |
| `state_json()` | the oracle: `phase`, `target`, `total`, `loaded`, `fraction`, `rows_per_s`, `elapsed_s`, per-label `{loaded,total}`, `error`, `placeholders` (e.g. `"not started"`) | `lib.rs:132` |
`state_json()` lives in the **default build** (no `gui`), so the data-layer matrix
asserts on it with no display. Same external-state contract as the pipeline
viewer's `$KNUT_VIEWER_STATE`.
## f4 — the load driver (`src/lib.rs`)
`run_load(params: LoadParams, state: Arc<Mutex<PopvizState>>, wake)` (`lib.rs:250`)
is the blocking driver (the caller runs it on its own thread). It builds a
multi-thread tokio runtime, computes per-label totals + `_gar_id` offsets, sets
`Phase::Vertices`, and dispatches on `LoadParams.target`:
| item | shape | where |
|---|---|---|
| `LoadParams` | `{ url, graph, scale, drop, target }` | `lib.rs:183` |
| `Target` | `Falkor` \| `Iceberg { warehouse }` (Iceberg needs feature `iceberg`) | `lib.rs:164` |
| Falkor path | `connect` → (`--drop` DETACH DELETE barrier) → drive `falkor_load_phase_order()` = **`[Vertices, Indexes, Edges]`**: **rayon** parallel batch-gen (`vertex_batches` `par_iter`) → per-label **concurrent** `insert_vertices_multi_par(…, concurrency, on_progress)` → `Phase::Indexes` **`create_index` per label (BEFORE edges)** → `Phase::Edges` parallel edge-gen → `insert_edges_keyed_par` → `Phase::Done` | `load_falkor` |
| Falkor multicore | `concurrency = available_parallelism()`; the "sim" (Arrow gen) fans across cores with rayon, the "sink" (UNWIND-CREATE) dispatches up to `concurrency` queries in flight over the multiplexed link — fixing the ~1-core serial ingest. Same produced graph (`_gar_id` in the batch column; order-independent). Progress `on_progress(rows)` folds into `buffer_unordered` completion | `load_falkor` |
| Falkor index-before-edges | the `_gar_id` index is built **after vertices, before edges** (`falkordb-design` f3b): edge endpoint `MATCH`es then hit an index (O(1)) instead of a full O(V) label scan, so the edge phase is O(edges) not O(edges×vertices) — the server-side half of the ingest speedup the client-side pipelining can't reach. Order guarded DB-free by `falkor_indexes_are_built_before_edges` | `falkor_load_phase_order`, `load_falkor` |
| Iceberg path | one table per label into the warehouse via skade — **no database, no container** | `load_iceberg` (`lib.rs:374`) |
On the happy path it ends in `Phase::Done`; any error lands in `Phase::Failed`
with `error` set. `wake` repaints the UI after each batch.
The Iceberg path is what lets the driver be robot-tested **end to end with no
external service**: `robot_iceberg_load_reaches_done_no_db`
(`tests/robot.rs:147`, `#[cfg(feature = "iceberg")]`) drives the real `run_load`
into a `tempfile` warehouse and asserts the shared `done`-state bar — a genuine
end-to-end drive, not synthetic state-poking, that still runs unattended in CI.
See [`popviz-guide.md`](popviz-guide.md#the-robot-test-matrix-testsrobotrs) for the
full tier matrix.
## f5 — the egui app (`src/main.rs` [feature `gui`])
`main.rs` is a pure rendering of `PopvizState`: a progress bar per label, overall
`rows/s`, a Start button, and the `$KNUT_POPVIZ_STATE` dump every ~300 ms. It
spawns `run_load` on a worker thread and reads the shared state each frame.
`--autostart` / `KNUT_POPVIZ_AUTOSTART` (`main.rs:57`) kicks the load on the first
frame (`main.rs:102`); `KNUT_POPVIZ_STATE` (`main.rs:59`) sets the dump path.
`--start-db` / `KNUT_FALKORDB_AUTOSTART` (`main.rs:215`) optionally boots a
FalkorDB container first for the `Falkor` target.
## Architecture map
| Area | Where |
|---|---|
| Population model | `knut-popsim` (`LABELS`, `geo`, `PopGraph`, `simulate`/`schema`) — re-exported; the canonical wasm-clean model. [`f1`](#f1--the-shared-knut-popsim-model) |
| Population gen (Arrow) | `src/lib.rs` (`vertex_batches` — `_gar_id` + cols, geo from `knut-popsim::geo`) |
| Introspectable state | `src/lib.rs` (`PopvizState`, `Phase`, `LabelProgress`, `state_json`) |
| Load driver | `src/lib.rs` (`run_load` — connect → per-label `insert_vertices` → `create_index`) |
| egui app | `src/main.rs` [`gui`] — progress bars + `--autostart` + `$KNUT_POPVIZ_STATE` dump |
## Robot + test-matrix coverage (the UI is driven in code)
`knut-popviz` is a thin egui shell over `PopvizState`; `state_json()` is the whole
UI contract, so the robot asserts JSON instead of pixels. Coverage
(`tests/robot.rs`, `tests/testmatrix_wiring.rs`) — all reuse **edda**'s
`nornir-robotui`/`nornir-testmatrix`, none hand-rolled:
| Tier | Runs | What it proves |
|---|---|---|
| synthetic | default, container-free, CI | phase progression idle→loading→indexing→done + the failure path |
| **iceberg** | `--features iceberg`, container-free, CI | a REAL `run_load` into a scratch skade/Iceberg warehouse reaches `done` (no DB, no Podman) |
| live FalkorDB | `#[ignore]`, needs `$KNUT_FALKORDB_URL` | the real bulk-loader end-to-end against FalkorDB |
| **matrix-emit** *(new)* | `--features "iceberg testmatrix"` | the real `run_load` **Ok arm emits a GREEN `knut-popviz/run_load` row** into the nornir matrix end-to-end — closes the gap between "the wrapper forwards" and "the loader calls it" |
The remote/introspection surface (`state_json` + the `functional_status` emit) is
exercised **in code, unattended** — no human, no live DB — so the matrix stays
honest on every CI run. See the load-path map: [`assets/popviz-loadpath.svg`](assets/popviz-loadpath.svg).
