# knut-pipelines — reasoning
> **Class:** `reasoning`. This file holds the *why*, the trade-offs, and the
> gotchas. The descriptive shape we actually run is in
> [`pipelines-design.md`](pipelines-design.md), which by rule mentions none of
> this. If you want "why not X?" — X is here.
## Why one model, many frontends
The goal was **debugging and robot-testing a live pipeline viewer without a
display**. If the whole observable truth is one folded `ViewerState`, and the
only way to change it is to `apply` a `ViewerUpdate`, then a test can drive it
headless, dump `state_json()`, and assert — no window, no GPU, no Spark even
(feed synthetic updates). Every frontend (CLI text, egui GUI, run API) becomes a
pure render of that one state, so they cannot disagree. That single payoff is why
the `apply`/`state_json` seam exists, and everything else serves keeping it true.
## Why the CLI-parity law
The GUI is the seductive place to grow features, and a feature that lives *only*
in `eframe::run` cannot be asserted headlessly — it becomes invisible to the test
matrix. So the law is: every capability the GUI shows is reachable from the
command line (`dump_text`/`dump_json`/`dump_state`) and appears in `state_json()`.
This keeps the GUI a *rendering* of the model rather than a second model, and it
keeps the whole viewer testable from a shell. "Any new visible panel MUST be added
to `state_json()`" is the operational form of the same law.
## Why Spark *jobs*, not only pipelines
The narrow design would stream only SDP run events. But `ExecutionProgress`
snapshots ride off *any* `ExecutePlan`, including a plain `spark.sql(...)`. Rather
than throw that away, `execute_sql` exposes it: a plain SQL query streams
stage/task counts with no pipeline involved, so the same viewer doubles as a live
Spark job monitor for one extra code path. `run_updates` merging events + progress
into one `ViewerUpdate` stream is what lets both feed the identical fold.
## Why heavy stacks are off by default
gRPC (tonic + vendored proto + `protoc` codegen) and egui/winit/glow are large,
slow-compiling trees that most consumers of the *model* don't need. Gating them
behind `grpc`/`gui` (with `live = gui + grpc`) keeps the default build a fast,
pure model + CLI `show` — importable anywhere, including as a library that only
builds `DataflowGraph`s. **egui/eframe are pinned to 0.28** not by preference but
because `egui-snarl` 0.4 (the node-graph widget) requires it; bumping egui means
waiting on snarl. The facett adapter deliberately sits on the *other* side of a
feature wall because facett brings egui-0.34, which cannot coexist with the
gui-0.28 tree in one build.
## Why one adapter per engine (not per viewer)
The reusable *visual* component belongs in facett, engine-neutral, so a Flink or
Beam or skade-streaming pipeline could reuse the exact same `PipelineView`. That
forces the knowledge of "how Spark's `ViewerState` maps to neutral stages/edges"
into *one* file — the Spark/SDP adapter — instead of a Spark-specific viewer.
"Show a Spark pipeline" is one adapter; the next engine is another adapter, not a
fork of the viewer. Spark ships now; **Flink / Beam / skade-streaming are
deferred** (the `PipelineAdapter` trait is the only reserved seam).
## Why the facett dep never leaks into knut's public API
If facett's `PipelineModel` types appeared in knut's signatures, every knut
consumer would transitively pull facett (and its egui-0.34 tree), defeating the
whole point of the feature gate. So the adapter is strictly a *boundary*: it takes
knut types in and hands a `PipelineModel` out, under `optional = true` /
`dep:facett-pipeline`. The default build stays byte-for-byte unchanged with the
feature off — that back-compat is a hard law, pinned by the default build not
referencing facett at all.
## The gotchas (all pinned by live tests)
- **`pipelines.proto` is on Spark `master` (4.2-era), not the `v4.0.0` tag.** The
vendored copy uses `ResolvedIdentifier` flow ids. Re-vendor from whatever Spark
tag is pinned when it moves.
- **`EventLevel` is a heuristic.** `PipelineEvent` on the wire carries only
timestamp + message, so `Info`/`Warn`/`Error` is derived from the message until
Spark surfaces structured severity.
- **`DefineFlow.once` must be unset** for normal flows — Spark errors otherwise.
- **`StartRun` needs a `storage` URI** (`file://`/`s3a://`/…) or outputs are never
materialised; a run without it is only a dry validation. The path must be
writable by the Spark workers, not just the caller.
## The gaps
- Graph-id selection from the GUI (today you pass it on the CLI).
- Richer event taxonomy (blocked on Spark's structured severity, above).
- A web frontend — axum + SSE over the `Serialize` model — is sketched but
deferred; the `Serialize`-able `ViewerState` is the seam it would use.
- Per-*dataset* metrics are now **real**, not stand-ins: the adapter fills each
stage's `StageMetrics.rows` from `ViewerState::rows_written` (the counts parsed
from the run's "wrote N rows" events), and the pipeline `rows`/`elapsed_s` from
the real row sum + the measured first→last timestamped-event span. What's still
deferred: per-node *throughput/lag* (Spark's `ExecutionProgress` is
per-*job-stage*, so there's no per-dataset timer to derive rows/sec from).