# Spark pipelines with knut — for idiots
> **Class:** `idiot`. Every step, nothing assumed. Goal: from a **cold shell**,
> start a Spark Connect server, then **define, run and watch a real Spark
> Declarative Pipeline** with `knutpipe`. If you already know Rust and Spark, read
> [`pipelines-guide.md`](pipelines-guide.md) instead — this one spells out every
> keystroke.
## The 30-second mental model
- A **pipeline** = a *dataflow graph*: some **datasets** (tables/views you want to
produce) and some **flows** (the `SELECT`s that fill them).
- You hand that graph to **Spark Connect** (a gRPC server; address looks like
`sc://host:15002`). Spark figures out the order and runs it.
- knut streams back **events** ("flow X is RUNNING") and **job progress** (task
counts). The viewer shows both. You write the graph; Spark does the work; knut
shows what's happening.
## What you need first
- A terminal (the black window where you type commands).
- **Rust** installed. Type this and press **Enter**; if it prints a version you
have it:
```
cargo --version
```
If it says "command not found", install Rust: type this, press **Enter**, and
accept the default:
```
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
Then close and reopen the terminal.
- **podman** (runs the Spark server in a container). Check it:
```
podman --version
```
If missing, install it from your OS package manager (`sudo apt install podman`
on Ubuntu/Debian).
- The knut repo on disk. Type this and press **Enter**:
```
cd /run/media/rickard/T9/git/knut
```
## Step 1 — see a pipeline with NO server (learn the shape, zero setup)
Type this and press **Enter**:
```
cargo run -p knut-pipelines --bin knutpipe -- show
```
The first run compiles for a minute, then prints a built-in bronze → silver →
gold demo pipeline as text. Now see the same thing as JSON — type and **Enter**:
```
cargo run -p knut-pipelines --bin knutpipe -- show --json
```
**That JSON is the spec format.** A pipeline spec is a dataflow-graph JSON where
each flow that should run carries a `query`. A ready-made one ships in the repo at
`crates/knut-pipelines/examples/pipeline.json` — that's the file we'll run.
## Step 2 — build the Spark Connect server container (once)
Type this and press **Enter** (it downloads Spark + Iceberg and bakes an image;
takes a few minutes the first time):
```
podman build -t knut-spark-iceberg -f containers/spark-iceberg/Containerfile .
```
When the prompt comes back with no red errors, the image is built. You only ever
do this once.
## Step 3 — start the server
Type this and press **Enter**:
```
podman run -d --name knut-spark -p 15002:15002 knut-spark-iceberg
```
That starts Spark in the background and maps its port `15002` to your machine.
Now **wait** — Spark's JVM needs 10–25 seconds before it accepts connections.
Type this and press **Enter**:
```
sleep 25
```
## Step 4 — define, run and WATCH the pipeline (one command)
Type this whole block and press **Enter** (the `\` lets one command span lines):
```
cargo run -p knut-pipelines --features grpc --bin knutpipe -- \
pipeline sc://127.0.0.1:15002 \
--spec crates/knut-pipelines/examples/pipeline.json \
--storage file:///tmp/knut_pipeline
```
`--features grpc` builds the live client (first time compiles a bit longer). That
single command does the whole dance for you:
1. **connects** to Spark Connect,
2. **defines** the graph (`CreateDataflowGraph` → a `DefineOutput` per dataset →
a `DefineFlow` per flow),
3. **starts the run** and **streams the lifecycle** to your terminal.
**You should see** something like:
```text
# defining pipeline (2 datasets, 2 flows)
# graph 03ce891f-…; starting run
INFO Flow spark_catalog.default.knut_bronze is QUEUED.
INFO Flow spark_catalog.default.knut_bronze is RUNNING.
INFO Flow spark_catalog.default.knut_bronze has COMPLETED.
INFO Flow spark_catalog.default.knut_silver has COMPLETED.
INFO Run is COMPLETED.
=== final state ===
Dataflow graph 03ce891f-… — 2 dataset(s), 2 flow(s) [2 done · 0 running · 0 failed · 0 idle]
✓ [materialized_view] knut_bronze (done)
✓ [materialized_view] knut_silver (done)
```
`Run is COMPLETED` with both nodes `done` = you just ran a real Spark pipeline and
watched it. **That's the whole thing.**
The two mistakes that trip everyone the first time:
- **`--storage` is required.** Spark needs a place for checkpoints/metadata to
actually materialise outputs. Without it, the run only dry-validates.
- The storage path must be **writable by the Spark workers**, not just by you.
## Step 5 — want the pretty node graph instead of text?
Add `--gui` (and `--features gui`). Same run, drawn as a live node graph: nodes go
yellow while running, green when done, red on failure, with job-progress bars.
Type and **Enter**:
```
cargo run -p knut-pipelines --features grpc,gui --bin knutpipe -- \
pipeline sc://127.0.0.1:15002 \
--spec crates/knut-pipelines/examples/pipeline.json \
--storage file:///tmp/knut_pipeline --gui
```
## Step 6 — just watch a plain SQL job (no pipeline at all)
**Any** Spark query streams progress, so knut doubles as a job monitor. Type and
**Enter**:
```
cargo run -p knut-pipelines --features grpc --bin knutpipe -- \
--json-logs watch sc://127.0.0.1:15002 \
--sql "SELECT count(*) FROM range(0, 50000000, 1, 24)"
```
`--json-logs` prints machine-readable lines (`"event":"connected"`, then climbing
`"event":"job_progress"`) — the same lines a test drives.
## Step 7 — clean up
When you're done, stop and remove the server. Type these two, pressing **Enter**
after each:
```
podman rm -f knut-spark
rm -rf /tmp/knut_pipeline
```
## Cheat sheet
| I want to… | command (all prefixed `cargo run -p knut-pipelines … --bin knutpipe --`) |
|---|---|
| see a pipeline's shape, no server | `show [spec.json]` |
| define + run + watch (terminal) | `--features grpc … pipeline <sc-url> --spec s.json --storage file://…` |
| …same, as a live node graph | add `--features grpc,gui` and `--gui` |
| watch an existing run by id | `--features grpc … watch <sc-url> --graph ID` |
| watch a plain SQL job's progress | `--features grpc … watch <sc-url> --sql "…"` |
| machine-readable log for a test | add `--json-logs` before the subcommand |
## Where this goes next
- The **engine-agnostic** viewer (one adapter turns knut's Spark state into
facett's neutral model) — the "facett adapter" section of
[`pipelines-design.md`](pipelines-design.md#f7).
- The reusable **visual node-graph component** lives in facett:
[facett-pipeline for dummies](../../facett/.nornir/facett-pipeline-for-dummies.md).
- A **full app** wiring knut + facett together (connect → submit → render):
[korp's worked example](../../korp/.nornir/spark-pipeline-example.md).