<!-- nornir:class: design -->
# build-info — the second half of a build (schema + emitter)
> A build produces **two things at once**: the *thing* (artifact bytes, hashed by
> `BuildProvenance` and — a later slice — stored in holger's znippy CAS) and the
> *build info* (the columnar record ABOUT the build). This note wires the second
> half in `nornir-build-thing`: the §f3 Arrow schema, a flat dep-light row, and a
> pluggable `BuildInfoSink` with an in-process default. The full constellation
> design (Flight seam, at-rest seal, jera DAG) is `workspace_nordisk/.nornir/
> build-info-flight-stream-design.md`; this is the build-thing-local slice 2.
## The schema (§f3) — one row, six kinds
Build info is inherently record-shaped, so it IS columnar. One row shape
(`build_info::BuildInfoRow`), discriminated by the `kind` column, carries the
whole stream — the Artifactory-parity **manifest rows** (`provenance` / `dep` /
`artifact`, derived from `BuildProvenance`) AND the **live feed** (`log` /
`metric` / `step`). Columns, in declared order:
`build_id` (Utf8) · `step_id` (Utf8) · `ts_micros` (Int64) · `kind` (Utf8) ·
`level` (Utf8) · `name` (Utf8) · `value` (Utf8) · `content_id` (Utf8) ·
`blob` (Binary, **the only nullable column**).
## Dep-light by construction
The rows + the sink are pure `std` + `serde` — a leaf repo's xtask gets the
emitter for free. The Arrow RecordBatch binding (`build_info_schema()` /
`build_info_batch()`) is gated behind the **`build-info-arrow`** feature, so the
default crate keeps its no-heavy-deps promise. The feature reuses the arrow
builder plumbing (`arrow-array` / `arrow-schema`, the crates + version znippy
pins) rather than re-rolling arrow; znippy's own `build_metadata_batch` builds a
*different* schema (blob-index), so only the plumbing is shared, not the shape.
## The sink seam
`BuildInfoSink` is transport-agnostic (`emit(&[BuildInfoRow])`):
- `InMemBuildInfoSink` — the **in-process default** (a shared, cloneable `Vec`):
single-box localhost, zero transport tax. Hand a clone to the build, read the
rows back afterwards.
- `NullBuildInfoSink` — the no-op default so existing `build_variant()` callers
pay nothing.
- Flight-backed sink — a **later slice** (the `edda/nornir-flight` seam is the
transport); not wired here. Same schema + sink API both ways.
## The emitter
`build_variant_with_info(set, name, repo_root, sink)` runs the same build as
`build_variant` (which now just routes to a `NullBuildInfoSink`, so callers are
unchanged) and emits: the **live feed** (`step`/`log`/`metric` for the
`cargo-build` step) as it finishes, then the **manifest rows** from the captured
`BuildProvenance`. Emission is best-effort — a sink error never fails the build.
## Tested (RED-when-broken)
- `arrow_schema_has_f3_columns_in_order` — exact column order + types + the
blob-only-nullable invariant + the schema-version metadata key.
- `manifest_rows_cover_provenance_dep_and_artifact` — a known captured build
yields the expected provenance / dep / artifact rows; `content_id` == sha256.
- `in_mem_sink_round_trips_rows` — what is emitted comes back byte-identical.
- `arrow_batch_matches_schema_and_values` — the RecordBatch shape + cell values.
- `emits_functional_rows_for_build_info` (feature `testmatrix`) — honest matrix
rows for the emit + round-trip.
## Slice 3 — native warehouse landing + at-rest seal (shipped)
The build-info feed no longer rides `test_results` as an interim; it lands in its
OWN warehouse table with its OWN submit path, and seals at rest:
- **`build_info` table** (`warehouse::build_info`, `iceberg_schema::build_info()`)
— the §f3 columns **1:1** (`build_id` · `step_id` · `ts_micros` · `kind` ·
`level` · `name` · `value` · `content_id` · `blob`nullable), the exploded
sibling of the one-row-per-build `build_provenance` ledger. `append_build_info`
/ `query_build_info` reuse `nornir_build_thing::build_info::BuildInfoRow`
**verbatim** — the same struct the emitter and the seal use, so live wire,
at-rest blob, and warehouse row are one shape (no second format).
- **Submit RPCs** (`Telemetry`, mirroring `SubmitTestResults`):
`SubmitBuildProvenance` ships each `BuildProvenance` as JSON (record can gain
fields with no proto bump) → `append_build_provenance`; `SubmitBuildInfo` ships
typed `BuildInfoRowPb` rows (`has_blob` distinguishes a NULL blob from an empty
one) → `append_build_info`. Client helpers: `remote::submit_build_provenance` /
`submit_build_info`. A dwarves build on the dev's hardware ships both to the
locked server warehouse, exactly like bakeoff/test/release rows.
- **At-rest seal** (`build::seal::seal_build_info_beside_artifact`, feature
`build-provenance`) — the §f3 rows sealed as **znippy over Arrow-IPC** →
`<artifact>.build-info.znippy`, the airgap-portable sibling of
`<artifact>.build-provenance.znippy`.
- **Viz** — the generic warehouse deck lists tables via `table_names()`, so
`build_info` + `build_provenance` show up natively with no bespoke pane.
Guarded: `submit_build_info_and_provenance_land_natively`
(`tests/data_submit_rpcs.rs`) submits real rows to an ephemeral server and reads
the exact values back over `Warehouse.Scan`; `warehouse::build_info` unit tests
round-trip the manifest + the nullable blob; `seal_build_info_writes_znippy_
beside_artifact` proves the seal. The root `build-info-arrow` feature forwards to
`nornir-build-thing/build-info-arrow`, and `tests/cargo_features_guard.rs` fails
RED if any feature ever references an undeclared `dep:`.
## Slice 4 — jera pipeline-DAG sub-jobs (§f5, shipped)
A build/pipeline is **not one job** — it is a **root jera job whose body is a
DAG, and every step is a sub-job**. Modelled in `nornir::build::pipeline`:
- **The DAG is data — reuse, don't re-roll.** A `PipelineSpec` (a `name` + a list
of `PipelineStep{ id, name, needs, kind }`) projects onto **funnel-core's
`PlanDag`** verbatim (`PipelineSpec::to_plan_dag`): **nodes = sub-jobs**, **edges
= `needs` dependencies** (`EdgeKind::Dep`, `need → step`). So `topo_order()` /
`critical_path()` / `depths()` (the ready-wave layering) all work for free — no
new graph type. A one-step spec is a one-node zero-edge "DAG", so the existing
single-job path is unchanged (additive).
- **The ledger is the sub-job hierarchy — reuse, don't re-roll.**
`PipelineSpec::run_on_ledger` starts a **root** `nornir-jobs` job (kind
`ci_build`) that OWNS the DAG, then admits **every step as a child sub-job**
(`parent_id` = root's `job_id`) `queued` up front, promotes each `running` in
ready-wave order, and terminates it `done`/`failed`. A step whose `needs` did not
all reach `done` is left blocked → `failed` without running. The `needs` edge set
is stamped into each sub-job's `detail_json`, so the DAG is recoverable from the
ledger, not only the projection. The viz Jobs tree already folds children under
their parent by `parent_id` — no viz change needed.
- **build_id / step_id contract.** The root `job_id` is the §f3 `build_id`; each
sub-job's `job_id` is the §f3 `step_id`. `PipelineOutcome::build_info_for(rows,
step)` filters a slice-2 stream to one sub-job (`build_id == root && step_id ==
sub-job job_id`) — "each step's log/metric stream is the f3 feed filtered by
`step_id`."
- **One engine for nornir AND dwarves** (LAW: reuse) — the same `root job →
pipeline DAG → sub-jobs` model, not two schedulers.
Guarded (`src/build/pipeline.rs`, RED-when-broken):
`three_node_dag_makes_root_plus_three_subjobs_with_needs_and_status` (a fetch →
compile → test DAG creates 1 root + 3 sub-jobs on the ledger with the right
`needs` edges + per-step `done`), `failed_step_blocks_dependents_and_fails_root`
(compile fails → test blocked → root failed),
`single_step_is_a_one_node_dag_additive`,
`build_info_stream_filters_per_step_id_to_each_subjob`,
`cyclic_pipeline_is_rejected`, `diamond_dag_reuses_funnel_core_critical_path`.
Emits `nornir_testmatrix::functional_status` rows for the three headline surfaces.
## Slice 5 — the Flight-backed sink + consumer (§f1/§f2/§f5, shipped)
The arc closes: the same build-info stream is now **in-process (default) OR Flight
(remote)** behind the ONE `BuildInfoSink` trait, exactly as the design says.
- **`FlightBuildInfoSink`** (`nornir::build::flight_sink`, feature
`build-info-flight`) — the remote counterpart of `InMemBuildInfoSink`. It
serializes each `BuildInfoRow` batch to the §f3 Arrow `RecordBatch` and ships it
via `edda/nornir-flight`'s `do_put`. It lives in **nornir** because it is the one
place that deps BOTH `nornir-build-thing` (the trait + rows) and
`edda/nornir-flight` (the raw transport) — no cycle (edda knows nothing of
build-info; build-thing knows nothing of Flight). `emit` is best-effort: rows are
buffered to a worker that groups by route key and `do_put`s one batch per key; a
transport error is surfaced via `flush() -> FlightStats`, never failing the build.
- **Route key = per sub-job `step_id`.** `route_key(build_id, step_id)` →
`build_id` for root/manifest rows (empty `step_id`), else `build_id/step_id`. So a
consumer subscribes by ticket **per sub-job** (§f5: "each step's log/metric stream
is the f3 feed filtered by `step_id`").
- **Consumer.** `subscribe(endpoint, token, key)` does `do_get(key)` and decodes
the batches back into `BuildInfoRow`s **byte-identical** to what was emitted.
- **Auth via nornir-service.** The sink and consumer carry a bearer token; the
`FlightServer` composes the shared `nornir_service::auth_interceptor`. An
unauthenticated `do_put`/`do_get` is rejected — proven RED-when-broken.
- **The arrow version boundary — ⚠ IT IS GONE (verified 2026-08-24).** This used to
read: "`nornir-flight` rides **arrow-56** (its `arrow-flight` → `tonic 0.13`
matches `nornir-service`; `arrow-flight 57+` needs `tonic 0.14`), while
`build-info-arrow` is **arrow-58**. Those are semver-incompatible majors…"
Measured against the tree, **both sides are arrow 59**:
`nornir-flight/Cargo.toml:42-44` declares `arrow-array`/`arrow-schema`/
`arrow-flight` all `"59"` on `tonic 0.14`, `nornir-build-thing:109-110` declares
`arrow-array`/`arrow-schema` `"59"`, and edda's `Cargo.lock` carries exactly one
arrow family — `59.2.0`, `arrow-flight 59.2.0` included. The tonic 0.13/0.14 split
that forced the gap resolved itself when `nornir-service` moved to 0.14.
`rows_to_batch` / `batch_to_rows` **stay**, and are not dead code: converting
`BuildInfoRow` to a `RecordBatch` is the serialization layer, needed at any arrow
version. What changed is the REASON — they are no longer a bridge across
incompatible majors, so nobody should preserve them as one or go looking for the
arrow-56 half. The test pinning their column shape against the canonical
`build_info_schema` still earns its keep: it is now a same-major drift guard.
- **Additive.** The whole module is gated behind `build-info-flight` (OFF by
default): the in-process `InMemBuildInfoSink` stays the default with zero
transport tax and no second arrow in the default build.
Guarded (`src/build/flight_sink.rs`, RED-when-broken):
`flight_schema_matches_canonical_f3_columns` (the §f3 shape + no-drift),
`batch_round_trips_rows_byte_identical` (arrow-56 columnize/decode lossless incl.
the NULL-vs-present `blob`),
`flight_sink_ships_and_consumer_receives_byte_identical` (a mixed manifest+feed
stream shipped over `do_put`, consumed per route key over `do_get`, byte-identical),
`auth_is_enforced_on_the_build_info_wire` (no bearer ⇒ `do_put`/`do_get` rejected),
and `emits_functional_rows_for_flight_build_info` (feature `testmatrix` — honest
`do_put`/`do_get` matrix rows).
## Slice 5 tail — the viz build-info LIVE-TAIL pane (shipped)
The arc's last open thread — surfacing the received stream *in the viz* — is now
wired: a **build-info live-tail** pane in the 📡 Live Run area subscribes the §f3
stream **per sub-job `step_id`** and shows the rows arriving as they land.
- **`BuildInfoTail`** (`src/viz/build_info_live.rs`, feature `viz`) — a **pure
fold** over `BuildInfoRow`s that knows nothing about *how* the rows arrived.
`ingest(&[BuildInfoRow])` folds each row into the per-sub-job state keyed by
`step_id`: a `step` row's status (carried in the §f3 `level` column —
`queued`/`running`/`done`/`failed`) drives the sub-job's **status + a recorded
transition** when it changes; `log`/`metric` rows append to that sub-job's
**tail**; manifest rows (`provenance`/`dep`/`artifact`, empty `step_id`) are
counted, not tailed (they are the static build record, not the live feed).
- **The live seam is `flight_sink::subscribe`** — `BuildInfoTail::pull(endpoint,
token, key)` (gated `build-info-flight`) subscribes one sub-job's ticket
(`route_key` = `build_id/step_id`) over `do_get` and folds the decoded rows
through the SAME `ingest`, so live-wire and injected rows fold identically. The
default `viz` build keeps the injectable `ingest` path and pays no Flight tax.
- **Reuse (LAW #5)** — the pane paints with the shared facett palette (`Theme`)
and the vendored facett widgets (`facett_ui::status_chip`, zebra rows); nothing
is re-rolled. It renders as data (`state_json()["build_info_live"]`), folded
into the app just like `bench.live`, so it is robot-testable, not pixel-only.
- **Additive** — a new `viz`-gated module + a field on the app; existing viz
unchanged. Drawn beneath the release-run feed in the 📡 Live Run tab.
- **Continuous auto-refresh (the timer, shipped).** The pane no longer needs a
manual `pull`: a **background poller** (`BuildInfoTail::start_poller`, gated
`build-info-flight`) re-`pull`s every tracked sub-job ticket
(`route_key(build_id, step_id)`) on a fixed `POLL_EVERY` (1500 ms) cadence on
its **own OS-thread tokio runtime** — the same pattern `FlightBuildInfoSink`
uses, so a `pull` never blocks the egui frame nor needs an ambient runtime.
Decoded rows come back over a plain `std` channel (the **inbox**); `draw` drains
the inbox through the SAME synchronous `ingest` every frame and calls
`request_repaint_after(POLL_EVERY)` so rows that land mid-build appear with no
user input (mirrors `bench_live`'s warehouse re-read). The poller forwards only
rows **past a per-key high-water mark**, so re-pulling the full stream never
double-folds — the append-only `ingest` (and every existing inject test) is
unchanged (L2). The **live-ticket set** is tracked (`subscribe_ticket`) and
**auto-grown** as the fold discovers each sub-job; the app starts the poller in
remote mode. Both the tick count (`poll_count`) and the live tickets are exposed
in `state_json`.
Guarded (RED-when-broken): `ingest_grows_tail_and_tracks_status` +
`manifest_rows_counted_not_tailed` +
`auto_refresh_drains_inbox_and_advances_ticks` (a simulated interval pull GROWS
the tail + ticks `poll_count`, no re-inject) + `empty_interval_pull_is_not_a_tick`
(`src/viz/build_info_live.rs`); `build_info_pane_tails_feed_per_subjob` +
`build_info_pane_auto_updates_on_poll_timer` (the REAL app auto-folds delivered
batches across egui frames) + `poller_pulls_live_flight_rows_across_intervals`
(feature `build-info-flight` — the poller re-`pull`s a live `edda/nornir-flight`
server across intervals, deduped by the high-water mark) + a wgpu snapshot
(`viz_snapshot_build_info_live`) (`tests/viz_build_info_live.rs`). Emits
`nornir/viz/build-info-live` `functional_status` rows (feature `testmatrix`).
## Deferred (later slices)
`dep` rows currently carry the enabled **features** (the consumed inputs present
on `BuildProvenance`); a resolved crate-graph dependency row is a later slice.
(**Continuous auto-refresh** of the live-tail pane — the timer that re-`pull`s
each ticket while the pane is open — is now **shipped**; see the "Slice 5 tail"
section above.) `run_on_ledger` runs steps sequentially in topo
order today (each wave's dependents wait on the prior); parallel intra-wave
execution of independent sub-jobs is a later refinement (the `PlanDag` already
exposes the waves via `depths()`).