<!-- nornir:class: reasoning -->
# CI/CD market inspiration — what dwarves borrows (anchor: TeamCity)
**Why this doc.** dwarves is the nordisk **TeamCity-in-Rust**. Survey the CI/CD field —
long-popular tools and new-and-beloved ones — and pull the genuinely good ideas, even from
tools we dislike. dwarves' differentiators stay: **pipelines are Rust `xtask`s (zero YAML)**,
the DAG is **presented by facett** (`facett-plandag`/`facett-brokkrview` — a visual edge YAML
tools don't have), and everything is **remote-controllable + robot-tested in code**.
## Established tools
| Tool | Beloved feature | Borrow → dwarves |
|---|---|---|
| **TeamCity** (anchor) | **Build chains** (snapshot + artifact deps), **agent requirements**, **pre-tested / personal builds**, **investigations/mute** | brokkr `[[pipeline.depends]]` = the chain; content-addressed artifacts flow along DAG edges; **agent requirements → jera worker-group capability match**; pre-tested = run the DAG on a worktree before landing; "investigation" = attach an owner + mute to a red node (funnel/plan-DAG already has assignment). |
| **Jenkins** | ubiquity, plugin ecosystem, scripted pipelines | lesson = *don't* be Jenkins (Groovy/plugin sprawl); our "plugins" are just Rust crates. Keep the declarative `brokkr.toml` seed + the programmable xtask. |
| **GitLab CI** | `needs:` **DAG** (out-of-order), **dynamic child pipelines**, runners + **tags** | ready-frontier drain (already sindri); **dynamic pipelines** = an xtask *emits* its DAG at runtime (native, since it's code); runner tags → jera group/label match. |
| **GitHub Actions** | **matrix** fan-out, **reusable workflows/composite actions**, **self-hosted runner labels**, **environments + required reviewers** | matrix = `BuildVariantSet` (target×feature×profile); reusable = brokkr pipeline templates; runner labels → jera worker-groups; **environments + approval gates → dwarves release gates** (manual/approval node before ship). |
| **CircleCI** | orbs (reuse), test splitting/parallelism, fast caches | test-splitting across jera agents (gatling within an agent, jera across agents); layer/action cache already in brokkr (`ActionCache`). |
| **Buildkite** ⭐ | **self-hosted agents + hosted control-plane** — you own the compute, they own orchestration | **THE model for dwarves**: `dwarves-server` = control plane (schedule/observe), `dwarves agent` = your machines run the work. Maps 1:1 onto the jera-multi-machine build below. |
| **Argo / Tekton** | k8s-native DAG, `Task`/`Pipeline` CRDs, step-as-container | container/VM steps via **draupnir** (KVM/OCI/Redfish) submitted as **jera** jobs — get k8s-style isolation without owning k8s. |
| **Azure DevOps** (borrow selectively) | **YAML templates** (parameterized reuse), **environments + gate checks**, **artifact feeds/retention** | template reuse → brokkr pipeline templates; **gate checks** (approval / external-signal before a stage) → release gates; artifact feeds → holger (the registry) with retention. Skip the rest. |
## New & beloved
| Tool | Why loved | Borrow → dwarves |
|---|---|---|
| **Dagger** ⭐ | **pipelines-as-code in a real language** + content-addressed caching, runs anywhere | validates our thesis: **Rust xtask pipelines, zero YAML**. Steal the caching model (cache by input digest — brokkr already does content-addressed). |
| **Earthly** | repeatable containerized builds (`Earthfile`), strong caching | reproducibility = draupnir container step + content-addressed artifacts; byte-identical re-run is already the bench/roundtrip law. |
| **Depot / Namespace / Blacksmith** | **fast remote builders + shared remote cache**, ephemeral runners | remote builders = jera worker-groups on beefy boxes (Loki/Odin); shared cache = the content-addressed action cache served from the warehouse. |
| **Woodpecker** | lean, self-hostable, open Drone-style | the "small & self-hosted" ethos — dwarves is one binary + agents, no SaaS. |
## dwarves synthesis — the top features to build (mapped)
1. **Distributed agents / worker-groups (jera multi-machine)** — the keystone; sketch below. *(Buildkite/GHA/TeamCity-agents/Depot)*
2. **DAG with `needs` + matrix fan-out** — brokkr DAG + `BuildVariantSet`; sindri drains the ready frontier. *(GitLab/GHA)*
3. **Build chains: snapshot + artifact deps across pipelines** — `[[pipeline.depends]]`, content-addressed artifacts along edges. *(TeamCity)*
4. **Dynamic pipelines** — an xtask emits its own DAG at runtime. *(GitLab child pipelines)*
5. **Approval / environment gates** — a manual/external-signal gate node before ship. *(GHA environments / Azure gates)*
6. **Pre-tested / personal builds** — run the DAG on a worktree branch before it lands. *(TeamCity)*
7. **facett presents the DAG** — `facett-plandag`/`facett-brokkrview` render the live pipeline (stations light up); the visual edge over every YAML tool. *(our own)*
8. **Content-addressed cache + reproducible container steps** — already seeded; extend with a warehouse-served shared cache. *(Dagger/Earthly/Depot)*
**Two standing differentiators:** pipelines are **Rust xtasks (zero YAML)**; the whole thing is
**remote-controllable and robot-tested in code** (self-driving, per the autotest law).
## jera multi-machine — design sketch (the keystone build item, ties H1)
Today jera = local job ledger + process/VM/container runners; brokkr's `AgentPool` is in-process.
Target = the **Buildkite model**: hosted control-plane + your own agents, over gRPC.
```
control plane: dwarves-server (sindri scheduler + brokkr DAG + jera ledger)
│ gRPC (bearer + dwarves-workspace header)
┌──────────────────┼───────────────────────┐
dwarves agent dwarves agent dwarves agent ← your machines
--group loki --group odin --group nuc
caps=[gpu,12c] caps=[32c] caps=[6c]
```
- **Agent register:** each machine runs `dwarves agent --group <name>`; registers
`{id, host, group, capabilities[], cores, mem, capacity}` with the server (mirrors the bench
machine-roster discipline — record machine+cores+mem on every job).
- **Schedule (sindri drain):** each ready step declares required `{group, caps}`; sindri matches
it to a **capable idle agent in the target group** (+ the exclusive gate), dispatches over gRPC.
- **Bidirectional kill / lifecycle:** the job lives in the ONE jera ledger; cancel job → kill remote
work; agent/work dies → job → `Killed` (never a silent stall — RAGNARÖK).
- **Artifacts:** flow between agents along DAG edges, content-addressed (`artifact::content_id`) so a
downstream agent pulls exactly the produced inputs.
- **The seam that keeps it clean (Law #1 — jera owns it, shared by nornir/dwarves/holger):**
an `Agent` dispatch trait with a `LocalAgent` (in-process, single-machine — ship first) and a
`GrpcAgent` (remote worker-group — add second) impl. sindri/jera schedule against the trait, so
going distributed is *adding an impl, not a rewrite* — and it lands in **jera (edda)**, not dwarves,
so all apps get it. Read jera + gatling from source; never hand-roll threads.
**Verdict:** jera-multi-machine is the #1 dwarves build item and the biggest single lever — it's what
turns dwarves from a single-box forge into a real CI/CD fleet, and (per the reuse-first law) it must
be built **into jera** as the shared `Agent`/`GrpcAgent` seam, not bolted onto dwarves alone.