<!-- nornir:class: history -->
# knut — branch & crate state (2026-07-29)
> **Class:** `history`. A tracked snapshot of knut's branches, sub-crates and what has landed,
> so the state survives agent context loss. knut's `main` is API-stable for **korp** /
> **korp-collectors** (path consumers) and a live map agent runs against it.
## TL;DR
All named feature branches (`feat/spark41-iceberg-falkor`, `enforce-valkey9`, `bench_fix`) are
**already merged into `origin/main`** — nothing was stranded. The "forks from yesterday" work
(Þund SQL authoring f1, chunked multicore `GRAPH.BULK` f9, popviz real-count gating) is all on
`origin/main`. Local `main` had merely drifted 3 commits behind and was fast-forwarded to
`26ddb45`. No new merges were needed; no korp-consumed API changed.
## Branch inventory (vs `origin/main` @ `26ddb45`)
| branch | tip | dated | ahead / behind | state | recommendation |
|---|---|---|---:|---|---|
| `main` (local) | `26ddb45` | 07-28 | 0 / 0 | fast-forwarded to origin/main this session | — |
| `feat/spark41-iceberg-falkor` | `e0c0378` | 07-23 | 0 / 11 | **MERGED** (ancestor of main; merge `5eb2fb3`) | delete — fully landed |
| `enforce-valkey9` | `f5dead2` | 07-24 | 0 / 7 | **MERGED** (ancestor of main; ff `f5dead2`) | delete — fully landed |
| `origin/bench_fix` | `bbef33e` | 07-17 | 0 / 49 | **MERGED** (ancestor of main) | delete — superseded |
| `origin/ny_dok` | `e5fb44c` | 07-10 | 2 / 96 | stale docs-restructuring experiment (moved all `.md` → `source_md/`, regenerated) | **ABANDON** — superseded by the current `.nornir/` docs; a rebase would be a massive conflict |
| `origin/salvage/2026-07-13` | `d200ff7` | 07-13 | 1 / 75 | single "untested salvage after overheat crash" Cargo.lock dep-bump | **HOLD/abandon** — nothing recoverable; left untouched |
`feat/spark41-iceberg-falkor` = Spark 4.1 Iceberg→FalkorDB: `bifrost watch --warehouse` (sip an
Iceberg warehouse into a graph, detached), optional S3 (s3a) build-arg, and the chunked multicore
`GRAPH.BULK` encode (f9). `enforce-valkey9` = default the FalkorDB image to a Valkey-9 build, never
stock redis. Both already in main.
## Sub-crate inventory (`crates/`)
| crate | role | recent work |
|---|---|---|
| `graphar` | Apache GraphAr format reader/writer (Parquet/CSV/ORC/JSON/Arrow, local or object store) | — |
| `graphar-falkordb` | Bulk-insert Arrow `RecordBatch` → FalkorDB (Redis/Valkey) | **f9**: chunk `GRAPH.BULK` per label into ≤2M-row blobs, encode in parallel via gatling (LAW #0, never rayon) — fixes 1-core serial encode + broken-pipe past `proto-max-bulk-len`. Valkey-9 image default. |
| `graphar-flight` | Arrow Flight SQL over FalkorDB — Cypher in, Arrow out | Any-pack the statement `do_get` ticket so FlightSQL queries reach `do_get_statement`; owns `i64_column` (de-twinned from korp). **korp consumes this.** |
| `knut-graphdb` | Backend-agnostic graph-DB trait; FalkorDB (Cypher) + embedded CozoDB (Datalog, no container) backends | renamed from `graphdb` (crates.io name taken). **korp consumes this.** |
| `knut-bifrost` | Bifröst — watch Iceberg tables (Spark Connect or Spark-free skade) → materialise a FalkorDB property graph by CDC | `watch --warehouse`; row-filter fix outside `SyncMode::Full`; skade projection/predicate pushdown. **korp consumes (`skade`,`spark` features).** |
| `knut-pipelines` | Viewer for Spark Declarative Pipelines (SDP) over Spark Connect | `nordisk_ingest` demo SDP pipeline + CSV generator; owns the `ViewerState` progress fixture (de-twinned from korp). **korp/korp-demo consume this** (`facett-adapter`,`testkit`,`grpc`,`server`,`web`). |
| `knut-popsim` | Pure-Rust generator for the synthetic 11M-person Swedish population graph (12 labels); no FalkorDB/Arrow/net/fs deps, compiles for wasm32 | — |
| `knut-popviz` | egui driver for the population bulk-loader — live progress + headless introspection | **gating fix**: `Done`/green now gated on REAL server-reported landed counts (was force-setting counters to totals → false-green on partial/no-op bulk loads). **korp consumes (`iceberg` feature).** |
| `knut-thund` | Þund — Arrow-native streaming dataflow engine, pluggable backend (native Arrow/DataFusion or lower-to-SDP via Spark Connect) | **f1**: SQL authoring surface (feature `sql`, `sqlparser 0.62` shared with datafusion 54) — a 4th front-end over the one IR alongside typed Rust builder / RON DSL / facett editor. `STREAM(t)` needs no grammar fork; `TBLPROPERTIES`; partition-transform gate. Covers the exact korp `from_ron → to_sdp` path. |
| `knut-tracing` | Shared `init_tracing()` bootstrap for knut binaries (REUSE) | — |
| `knut-build` | knut's build-thing (build space as data + the required-features / popviz-autostarts-FalkorDB rules) | path-deps `../../edda/crates/nornir-build-thing` — a build-matrix CONSUMER, not published. Source of the Cargo.lock churn (edda's crate gained `object`/`rustc-demangle` deps). |
Excluded from the native workspace: `crates/knut-bifrost/wasm-guest` (wasm32-only).
## Landed this session
- Fast-forwarded local `main` `b907a49` → `origin/main` `26ddb45` (the Þund SQL f1 merge + docs).
- Committed the untracked `.nornir/rel2graph-design.md` (design capture, 07-26; no code yet — a planned
`knut-rel2graph` engine surfaced as a korp Tools-menu function). Docs-only, zero API impact.
- Refreshed `Cargo.lock` to match the current `edda/nornir-build-thing` resolution (knut-build path dep).
Metadata only; does not affect korp (separate workspace) or any published API.
## Verification
- `cargo build --workspace` + `cargo test --workspace` (CI-style, no `--locked`; CI itself does not use
`--locked`) — see the session report for the green result.
## Pending / not done
- `ny_dok` and `salvage/2026-07-13` left on the remote (recommend abandon; not deleted by this agent).
- `knut-rel2graph` is design-only; build deferred until after the korp demo (per the design doc's Status).
## korp API risk
**None.** No korp-consumed crate (`knut-pipelines`, `knut-popviz`, `graphar-flight`, `knut-graphdb`,
`knut-bifrost`) had a public signature changed by anything landed here; all feature work was already on
`origin/main` that the live map agent runs against. The only new commits are a docs file and a lockfile
refresh.
## korp-swarm C/D/E landed (2026-07-29 — `feat/korp-swarm-knut-cde`, merged to main)
The swarm leg C/D/E, verified green and merged fast-forward (see `thund-design.md`,
`thund-reasoning.md`, `pipelines-design.md`, `bifrost-design.md` for the design capture):
- **C — Þund `ExecBackend`, one IR / pluggable backend.** `both_backends_consume_the_same_batch_ir`
(in `knut-thund/tests/iceberg_spark_gap.rs`) authors ONE `Pipeline` and runs it on the native
DataFusion backend (real `SUM(v)=60` asserted) AND lowers the SAME value to a valid SDP graph the
Spark backend accepts in `check`. All **4 authoring surfaces** lower to the one IR — typed Rust
builder (the IR itself), RON DSL (`dsl` feature, round-trips), SQL (`sql` feature, 13 tests,
`SQL→IR→RON→IR→SQL`), facett graph (reuses `knut-pipelines` `PipelineView`). `spark_4_2()` flips
CDC/sinks/persistent-views in the capability report.
- **D — Iceberg↔Spark streaming GAP, proven honest.** `SparkBackend::lowering_report` →
`SdpLoweringReport`/`SdpFlowDrop` names, per flow, every knob SDP-over-Connect drops
(watermark, window, non-append output mode, CDC on 4.1, SCD Type2 history, expectations). The
native backend RUNS what SDP drops: `native_runs_what_sdp_drops_watermark_and_window` (a late row
dropped by the watermark, windows close+evict) and `cdc_gap_spark_41_drops_all_42_drops_type2_history`
(native keeps both SCD2 versions). RED-when-broken both ways.
- **E — Iceberg→graph RON codegen seam.** `knut-bifrost` `MappingSpec` gains RON parity
(`from_ron`/`to_ron`/`load_ron`, YAML still accepted) plus `from_arrow_schema` → a candidate
mapping emitted as RON where **every** column is mapped or SURFACED in `unmapped` (nested types),
never silently dropped; guesses are listed for review. 6 `knut-bifrost` mapping tests.
Crate-split (teach thund an Iceberg source vs route thund→bifrost) is the deferred design decision
from the findings doc — left open for Rickard (runa vs knut-local RON dialect), not made implicitly.
- **Spark-view backend API (for korp).** `knut-pipelines::spark_view` adds neutral
`SparkRuntimeInfo`/`StreamingQueryInfo`/`StreamingQueryStatus` (always compiled, no gRPC), fed by
`grpc`-gated `PipelinesClient` methods (`runtime_info`, `list_active_streams`, `stream_status`,
`stream_recent_progress`, `stop_stream`). Read-side full + `stop`; executor/worker topology is off
the Connect wire and honestly not modelled.
**API stability:** every change is purely additive — new `MappingSpec` methods + 2 new
`BifrostError` RON variants, a new `spark_view` module + re-exports, extra `knut-thund` re-exports
(`SdpFlowDrop`, `SdpLoweringReport`). No existing korp-consumed signature changed;
`cargo build -p knut-pipelines -p knut-bifrost -p knut-popviz -p graphar-flight -p knut-graphdb`
is clean.
## Python client `to_networkx` tested + documented (2026-07-29 — `main`)
The `python/knut` client's `KnutClient.to_networkx` (FalkorDB → Arrow Flight SQL → pandas →
NetworkX) was wired with an example but **untested and barely documented**. Landed:
- **Real offline test** `python/tests/test_networkx.py` (9 tests) — exercises the *actual*
`to_networkx` via a `FixtureClient` that overrides `query` to serve canned `pyarrow.Table`s, so
the whole production path (`query_df.to_pandas` → `_build_networkx` → `nx.from_pandas_edgelist` /
`set_node_attributes`) runs for real; only the gRPC/DoGet hop is stubbed. Asserts node/edge
counts, node+edge attribute preservation, directedness (reverse edge absent), the undirected
option, custom column names, and the Alice→Dave shortest path from `social_graph.py`.
RED-when-broken (proven: breaking directedness fails 3, the fix below fails 2). Full suite
22 passed (9 new + 13 in `test_proto.py`).
- **Bug fixed (Python-side only).** `to_networkx` passed `edge_attr=True` for a src/dst-only edge
frame; `nx.from_pandas_edgelist` (networkx ≥3) raises `NetworkXError` when there are no
non-endpoint columns. Now `edge_attr = [...] or None`. The example never hit it (always has a
`since` column). No korp-consumed Rust API touched.
- **Testability seam.** Extracted the pure `KnutClient._build_networkx(vertex_df, edge_df, …)` from
`to_networkx`; the public `to_networkx` signature is unchanged (it now fetches then delegates).
- **Docs.** New design doc [`networkx-export.md`](networkx-export.md) (routes into `docs/manual.pdf`
via `<!-- nornir:class: design -->`), linked from `design.md` and `guide.md`. PDFs not
regenerated this session.