# Watch a Spark pipeline turn green — for idiots
> **Class:** `idiot`. Every step, nothing assumed. Goal: from a **cold shell**,
> open a web page and **watch a pipeline's DAG animate from grey to green** — with
> **no Spark server at all** — then run a dead-simple change-data-capture example.
> If you already know Rust, read [`pipeline-run-guide.md`](pipeline-run-guide.md)
> instead; this one spells out every keystroke.
## The 10-second mental model
- A **pipeline** is boxes (**datasets** — tables you want) joined by arrows
(**flows** — the queries that fill them).
- knut can **draw those boxes in your web browser** and colour them as the pipeline
runs: grey = waiting, yellow = running, **green = done**.
- The demo needs **no server**. knut plays a pretend run so the boxes light up on
their own. You just watch.
## What you need first
- A terminal (the black window where you type).
- **Rust**. Type this, press **Enter**; a version number means you have it:
```
cargo --version
```
If it says "command not found", install it, then close and reopen the terminal:
```
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
- **A web browser** (Firefox, Chrome — whatever you already use).
- The knut repo on disk. Go into it (adjust the path to wherever you cloned it):
```
cd ~/git/knut
```
## Step 1 — start the demo (one command)
Type this whole line and press **Enter**:
```
cargo run -p knut-pipelines --features web --bin knutpipe -- web
```
The first run compiles for a minute. Then it prints one line like:
```
# knutpipe web viewer on http://127.0.0.1:8787
```
That address is your pipeline page. **Leave this terminal running** — it is the
server. (To stop it later, click the terminal and press **Ctrl+C**.)
## Step 2 — open the browser and watch it turn green
Open your web browser and type that address into the address bar, then **Enter**:
```
http://127.0.0.1:8787
```
You'll see three boxes — **bronze_events → silver_events → gold_daily** — joined by
arrows. Watch for a few seconds:
1. all boxes start **grey** (idle),
2. each turns **yellow** as it runs,
3. then **green** as it finishes,
4. the top line counts up: `… — 3 done · 0 running · 0 failed · 0 idle`,
5. the bottom line shows the last event, ending in `COMPLETED`.
**When all three boxes are green, you just watched a pipeline run.** That's the
whole thing. No Spark, no setup — knut replayed a synthetic run over the page.
Refresh the page (**F5**) to watch it animate again from grey.
## Step 3 (optional) — point it at a real Spark server
If you *do* have a Spark Connect server running (see
[`pipelines-for-idiots.md`](pipelines-for-idiots.md) for how to start one), stop the
demo (**Ctrl+C**) and run the live version instead — add `,grpc` and `--connect`:
```
cargo run -p knut-pipelines --features web,grpc --bin knutpipe -- \
web --connect sc://127.0.0.1:15002
```
Same page, but now the colours follow a **real** run. If the server needs a login,
put it right in the address, like
`--connect sc://127.0.0.1:15002/;user_id=alice;token=YOUR_TOKEN`.
## Step 4 (optional) — the tiniest change-data-capture example
"CDC" means: instead of rebuilding a table from scratch, you feed it a **list of
changes** (add this row, change that one, delete another) and knut merges them in.
Each change row carries a `_change_type` word: `INSERT` (add), `UPDATE_AFTER`
(change), or `DELETE` (remove).
**SCD type 2** means knut keeps the *history*: every version of a row is stored with
a "valid from" and "valid to" time, so you can ask "what did this row look like last
week?".
Here is the smallest possible one, in Rust. It says: keep a `customers` table,
matched by `id`, ordered by `seq`, keeping full history (`Type2`):
```rust
use knut_thund::ir::{CdcSpec, Flow, FlowKind, ScdType};
let flow = Flow {
name: "apply_customers".into(),
target: "customers".into(),
reads: vec!["customer_changes".into()], // the stream of change rows
kind: FlowKind::Cdc { cdc: CdcSpec {
keys: vec!["id".into()], // match rows by id
sequence_by: "seq".into(), // newest change wins
apply_as_deletes: None,
scd_type: ScdType::Type2, // Type2 = keep history
}},
query: Some("SELECT id, name, seq, _change_type FROM customer_changes".into()),
expectations: vec![],
};
```
Feed it changes like:
```
id=1 name="Ada" seq=1 _change_type=INSERT
id=1 name="Ada L." seq=2 _change_type=UPDATE_AFTER
id=2 name="Bo" seq=1 _change_type=INSERT
id=2 ... seq=3 _change_type=DELETE
```
and the `customers` table ends up with Ada's **two** versions (the first closed off,
the second still open) and Bo's version **closed** by the delete. Change `Type2` to
`Type1` and you'd keep only the latest version of each row (no history).
You don't have to hand-build these change rows for a real table: knut's bifröst
reads them straight out of an Iceberg table (it borrows skade's changelog reader, so
the `_change_type` words always mean the same thing).
Run the built-in tests that prove both behaviours:
```
cargo test -p knut-thund --features native \
cdc_scd_type1_upserts_and_deletes cdc_scd_type2_keeps_version_history
```
Two green tests = the merge works.
## Step 5 — stop the server
Click the first terminal (the one still running the web viewer) and press **Ctrl+C**.
Done.
## Cheat sheet
| I want to… | do this |
|---|---|
| watch the demo in a browser | `cargo run -p knut-pipelines --features web --bin knutpipe -- web`, open `http://127.0.0.1:8787` |
| watch it again | refresh the page (**F5**) |
| use a different port | add `--addr 127.0.0.1:9000` |
| watch a REAL server | add `,grpc` and `--connect sc://…` |
| log in to the server | put `;user_id=…;token=…` after the address |
| run the CDC example tests | `cargo test -p knut-thund --features native cdc_scd_type1_upserts_and_deletes` |
## Where this goes next
- Running a real Spark server and a real pipeline (not the demo):
[`pipelines-for-idiots.md`](pipelines-for-idiots.md).
- The full workflow (auth, Airflow events, CDC in Rust):
[`pipeline-run-guide.md`](pipeline-run-guide.md).
- How it all fits together under the hood:
[`pipeline-run-design.md`](pipeline-run-design.md).