nordisk/nornir

.nornir/clone-event-stream.md

<!-- nornir:class: design -->
# Clone / populate event stream — why a member didn't populate, readable remotely

> Design + guide + idiot-guide for **PLAN #6**: make per-member clone/fetch/populate
> failures **first-class, persisted, and readable from a thin viz/CLI client**
> instead of `eprintln!`-only on the server log. The durable twin of the monitor's
> `eprintln!("…fetch error…")`. Reuses the `release_events` / `airgap_events`
> event-table shape and the EPIC N-data run→persist→live pattern.

## 1. The problem (seen live)

When `nornir-server`'s poll loop (or a `Workspaces.Fetch` RPC, or `nornir workspace
add`) clones+fetches a monitored workspace's git members, a per-member failure used
to surface **only** as a line on the server's stderr:

```
nornir-monitor: nordisk::korp fetch error: clone-fetch https://… Couldn't obtain Username
```

A thin viz/CLI client (`nornir … --server`, `nornir viz` against `oden`) has **no
access to that log**. It just sees the member's data missing and **no reason why** —
was it an auth failure (https→ssh), a vanished/renamed repo, a private repo, a
network blip? Invisible. The populate outcome was log-only, not data.

## 2. The fix — one row per member attempt, persisted to the warehouse

New append-only Iceberg table **`clone_events`** (LAW: persist to the warehouse,
not `/tmp`; on the fast warehouse disk), one row per fetched member attempt:

| col | type | meaning |
|---|---|---|
| `ts_micros` | timestamptz | when the attempt ran (UTC) |
| `workspace` | string | the workspace being fetched |
| `member` | string | the member (or `*` for a workspace-level `republish`) |
| `remote` | string | the git remote that was cloned/fetched |
| `op` | string | `clone-fetch` \| `republish` |
| `status` | string | `ok` \| `error` |
| `detail` | string | the error chain (`{e:#}`) on error; the resolved SHA / snapshot id on ok |
| `elapsed_ms` | long | wall-clock duration of the op |

Scalar columns only (no `depends_on` DAG edges — a fetch has no inter-member
ordering), so SQL readers (DuckDB / PyIceberg) query it directly. Schema:
`src/warehouse/iceberg_schema.rs::clone_events`. Writer/reader:
`src/warehouse/clone_events.rs` (mirrors `release_events.rs` / `airgap_events.rs`).

### Where the monitor writes failures

`src/monitor.rs::fetch_workspace` now records a `MemberOutcome { member, remote, op,
status, detail, elapsed_ms }` for **every** fetched member (ok OR error) onto
`FetchReport::outcomes`. Because `nornir-server` holds the single-writer redb lock,
the monitor itself does not open the warehouse — the **call sites that own a
handle** persist the report via `clone_events::record_fetch_report(&wh, &report)`
(best-effort + non-fatal — a logging-row failure never aborts a fetch/republish):

| Call site | file:symbol |
|---|---|
| Poll-loop sweep | `src/bin/nornir-server.rs::server_sweep` (served handle) |
| `Workspaces.Fetch` RPC | `src/bin/nornir-server.rs` `WorkspacesSvc::fetch` (served handle) |
| Eager `workspace add` | `src/bin/nornir-server.rs` eager-register spawn_blocking |
| Fat CLI `workspace fetch` / `populate` | `src/bin/nornir.rs::persist_clone_events` (own handle) |

## 3. Expose it over gRPC — `Viz.CloneEvents`

`proto/nornir.proto`: `rpc CloneEvents (VizTimelineRequest) returns (JsonResponse)`
on `service Viz`. Handler in `src/bin/nornir-server.rs` (`VizSvc::clone_events`)
resolves the served workspace via `AppState::ws(metadata)` and returns
`query_clone_events(&wh, CloneSelector::Workspace(name))` as a JSON array of
`CloneEventRow` — the SAME rows the embedded read returns (mirrors
`Viz.ReleaseEvents` / `Viz.TestResults` / `Viz.BenchTelemetry`). Client decoder:
`src/viz/remote.rs::fetch_clone_events`.

## 4. Show it in the viz — 🧬 `nornir` root pane

`src/viz/nornir_tab.rs` gains a **🩺 populate status** panel for the active
workspace: a per-member grid (member · op · ok/failed · detail) with a green/red
summary (`✓ all N ok` / `✗ K member(s) failed to populate`). It loads on a ⟳
button and auto-loads on a workspace switch:

- **local mode** reads the workspace warehouse with `IcebergWarehouse::open_read_only`
  (lock-tolerant — never contends the server's single-writer lock);
- **remote mode** reads `Viz.CloneEvents`.

LAW #6 (components emit what they render): `state_json()["populate_status"]` lists
`{ workspace, total, failed, members:[{member, op, status, detail, remote, ts,
elapsed_ms}] }`, and the button is discoverable as
`{ id:"populate_status", rpc:"Viz.CloneEvents" }` so the headless matrix mechanically
covers the surface.

## 5. CLI parity — `nornir workspace events <name>`

```
nornir workspace events nordisk          # human view, newest-first
nornir workspace events nordisk --json    # raw CloneEventRow array
nornir --server oden:7878 workspace events nordisk   # thin path over Viz.CloneEvents
```

Local path reads `<root>/<name>/builds` via `open_read_only`; thin path calls
`Viz.CloneEvents` with the `nornir-workspace` header set to the target. No
viz-only capability — the CLI prints the same event list (LAW: CLI parity).

## 6. Tests (inject-assert, not "didn't panic")

- `warehouse::clone_events::fetch_report_failure_persists_and_reads_back` — feed a
  `FetchReport` with one OK + one FAILED member → assert exactly two rows persist
  with the right `(member, op, status, detail, remote, elapsed_ms)`, read back
  workspace-scoped, the failed member's error chain is intact, a different
  workspace's scope is empty, the human render names the failure, and JSON
  round-trips.
- `warehouse::clone_events::republish_outcome_records` — a workspace-level
  `republish` row records (member `*`, snapshot id in detail).
- `viz::nornir_tab::state_json_surfaces_failed_member_populate_status` — inject
  rows → assert `state_json["populate_status"]` surfaces the failed member, its
  error detail (RFC3339 ts), the failed-count, and the `Viz.CloneEvents` button.
- `viz::remote::clone_events_wire_roundtrip` — the server's `serde_json::to_string`
  ↔ the client's `from_str` round-trip is lossless (failed member + error chain
  survive).

## 7. Place in the constellation

This is the EPIC N-data read+write loop for the **fetch/populate** surface: the
producer (the monitor) streams outcomes into the warehouse; the thin client reads
them back over `Viz.CloneEvents` or `nornir workspace events`. It complements
`release_events` (release op-DAG) and `airgap_events` (bring-up DAG) — the same
event-table shape, the same "see why it stalled" goal, now for the clone step that
feeds every other table.