<!-- nornir:class: design -->
# knut-flight — an Arrow Flight server for the rel2graph output (N4)
> **Class:** `design`. A **plain Apache Arrow Flight** server (not Flight SQL)
> that serves knut's rel2graph output — nodes, edges, and the source relational
> tables — as ticketed Arrow streams. Sibling of the
> [Flight SQL server](flight-design.md) (`graphar-flight`, Cypher-in/Arrow-out
> over a live FalkorDB): this one has no database and no query language; it just
> streams the graph the mapping fabric built.
> **Source:** crate `crates/knut-flight` — `lib.rs` (`Catalog`,
> `GraphFlightService`), bin `src/bin/knut-flight-server.rs`; test
> `tests/roundtrip.rs`; Python client `demo/flight_networkx_demo.py`.
## Why a new crate, not `graphar-flight`
`graphar-flight` implements the **Flight SQL** trait: it takes Cypher, runs it
against FalkorDB, and returns Arrow. `knut-flight` implements the **plain Flight**
trait ([`FlightService`]) over an in-memory [`Catalog`] built from a
[`GraphMutation`](rel2graph-design.md) at startup. Different trait, no DB, no
query language — so it stays a clean single-binary service. Ports are disjoint:
FlightSQL uses `:50051`/`:50054`, this server defaults to **`:50055`**.
## The catalog
Built once at startup from a rel2graph [`GraphMutation`](rel2graph-design.md)
through the single seam `Catalog::from_mutation(mutation, source_tables)`. There
are two entrypoints onto that seam:
* **`Catalog::from_bench(scale)`** — the synthetic path: runs the engine over the
seeded 12-table ERP schema (deterministic, `DEFAULT_SEED`).
* **`Catalog::from_parquet_dir(spec, dir)`** — **the live-data path**: reads
`<table>.parquet` for every table in a mapping spec and runs the engine's
canonical `knut_bifrost::rel2graph::build_mutation` over the collected
`ArrowTables`. This serves a REAL graph read off disk — an Iceberg /
object-store export — with no bench and no live DB. It is the shape korp's
`iceberg2graph` step produces when it lands lakehouse tables as parquet and
hands the directory to this server. See [the engine seam](rel2graph-engine-seam.md).
Both funnel through `from_mutation`, so a bench catalog and a real-data catalog
are identical in shape. At scale 1 the bench path materialises 35 datasets:
| ticket | rows @1 | schema |
|---|---|---|
| `nodes` | 1106 | `(label: Utf8, id: Int64)` — aggregate node skeleton, all labels |
| `edges` | 2955 | `(src_label, rel, dst_label, src, dst)` — resolved edges only |
| `node/<Label>` | per label | the label's **native** vertex batch (typed property columns) |
| `edge/<src>_<rel>_<dst>` | per rel | the relationship's edge batch (`src`/`dst` + props) |
| `table/<erp.qualified.name>` | per table | the raw source relational table |
The **ticket is the dataset name as UTF-8 bytes** — no protobuf envelope — so any
Flight client (Rust, pyarrow, Go) round-trips it verbatim. The `nodes`/`edges`
aggregates carry a UNIFORM schema across labels: exactly what a networkx loader
wants (see below). The `node/*` and `edge/*` flights carry the full native typed
columns; `table/*` exposes the rel2graph **inputs** for provenance.
## The Flight surface
Read-only. Implemented verbs:
* **`list_flights`** — one [`FlightInfo`] per dataset: its Arrow schema, a `Cmd`
descriptor and a `Ticket` (both the name bytes), and `total_records`.
* **`get_flight_info`** — the `FlightInfo` for a descriptor (`cmd` bytes, or a
`/`-joined path).
* **`get_schema`** — a dataset's schema without pulling data.
* **`do_get`** — streams the dataset's `RecordBatch`es for a ticket, encoded with
`FlightDataEncoderBuilder`.
The mutating verbs (`do_put`, `do_exchange`, `do_action`, `handshake`,
`poll_flight_info`) return `unimplemented` — this is a read-only projection
server; `list_actions` returns an empty list.
## The binary
`knut-flight-server` — config via flags/env only, no interactive setup:
| flag | env | default |
|---|---|---|
| `--addr <host:port>` | `KNUT_FLIGHT_ADDR` | `0.0.0.0:50055` |
| `--scale <n>` | `KNUT_FLIGHT_SCALE` | `1` |
| `--parquet <dir>` | `KNUT_FLIGHT_PARQUET` | *(unset → bench data)* |
| `--mapping <file>` | `KNUT_FLIGHT_MAPPING` | *(unset → bench 12-table spec)* |
With `--parquet <dir>` the server serves a REAL graph from a parquet export (the
live-data path above); `--mapping` supplies the rel→graph YAML for that data and
defaults to the bench 12-table spec, so a `rel2graph-export … --parquet <dir>`
dump serves out of the box. Without `--parquet`, `--scale` sizes the bench.
`--version` prints `knut-flight-server v<version> (<git-hash>)` — the git hash is
stamped by `build.rs` (`KNUT_GIT_HASH`, pure-Rust file walk, no shelling), per
the [build-version law](build-variants.toml). `--help` lists the flags.
```bash
cargo run -p knut-flight --release --bin knut-flight-server -- --addr 0.0.0.0:50055 --scale 1
```
## Proving it end-to-end
Two proofs, one per audience:
* **Rust** — `tests/roundtrip.rs` stands the server up on an ephemeral port,
connects a real `FlightClient`, `list_flights` (asserts the 35 datasets incl.
`nodes`, `edges`, `edge/Employee_REPORTS_TO_Employee`, `table/erp.hr.employees`),
and `do_get`s `nodes`/`edges`/a source table, asserting 1106/2955 rows and that
every `REPORTS_TO` edge has `src != dst` (no self-loop, a7208dc).
* **Python** — `demo/flight_networkx_demo.py` connects with `pyarrow.flight`,
lists flights, pulls `nodes`+`edges` as Arrow, and rebuilds a `MultiDiGraph`
with the **same** `demo/knut_networkx.py` loader the file-export demo (N3) uses
— then asserts the Flight-path graph equals the file-path graph AND knut's
oracle. Same graph, two delivery roads.
* **Live data** — `tests/live_data.rs` writes the scale-1 source tables to
parquet (`write_source_parquet`, the export bin's `--parquet`), builds a
catalog with `from_parquet_dir` over that directory, asserts it equals the
bench catalog's shape (35 datasets, 1106 nodes, 2955 edges, 21 REPORTS_TO with
no self-loop), and pulls `nodes`/`edges`/a source table with a real
`FlightClient` — a client pull of REAL on-disk data, engine-driven end to end.
## N5: a service, two ways (DONE)
This clean single-binary service — in-memory state built at startup, env-only
config, no interactive setup, disjoint port — is now delivered as a service **two
ways**, both driven from ONE `ComponentSpec::knut_flight` in Skidbladnir (so the
Runtime targets never drift):
1. **systemd** — `SystemdInstaller` stages the binary into a TarZstd bundle, unpacks
it under the airgap layout, and drops `knut-flight.service` (`User=_knut`,
`Environment=KNUT_FLIGHT_ADDR=0.0.0.0:50055` / `KNUT_FLIGHT_SCALE=1`, TCP health on
50055), enable+start over systemd D-Bus. Driver:
`skidbladnir --example install-knut-flight`.
2. **Kubernetes** — `containers/knut-flight/Containerfile` images the version-stamped
release binary onto a slim glibc base (`knut-flight-server:0.1.3`); Skidbladnir's
`KubernetesRenderer` projects the same spec to a Deployment + Service in namespace
`knut` (`skidbladnir --example render-knut-flight-k8s`). Manifests + apply steps:
`Skidbladnir/deploy/knut-flight/`.
The image build copies the host-built binary (the knut workspace has path deps on
sibling repos, so it does not `cargo build` in isolation inside a container); the
binary needs glibc ≤ 2.39, so a trixie base runs it directly.
Future work: source the graph from a live sink or object store instead of the bench
generator (the `Catalog` builder is the single seam to swap).
## See also
- The file-export sibling that feeds the same networkx loader: [rel2graph-networkx-design.md](rel2graph-networkx-design.md).
- The Flight **SQL** server (Cypher/FalkorDB): [flight-design.md](flight-design.md) / [flight-guide.md](flight-guide.md).
- The graph seam + correctness oracle: [rel2graph-design.md](rel2graph-design.md), [benching-benchmarks.md](benching-benchmarks.md).