<!-- nornir:class: design -->
# draupnir boot-status readback — `await_power_state` / `boot_and_await`
Added draupnir 0.1.5 (additive). The cross-backend **boot-status readback** seam: a
`Boot::boot` fires an instance up but returns *before* it has actually powered on, so
a consumer that needs to *know* the instance is up polls its `Lifecycle::status`
until it reports the wanted state. This is the power-lifecycle analogue of the
container `run_to_completion` drive loop, but written against the plain `Lifecycle`
trait so it drives **any** backend — KVM, container, Redfish — and a mock alike.
## Why it exists
The chain is `app -> jera (job) / Skidbladnir (service) -> draupnir (provision)`.
`Boot::boot` returns a `Machine` the instant the backend accepts the launch:
- a **KVM** guest returns as soon as tunnr has the QEMU process up — the kernel is
still booting;
- a **Redfish** node returns after the power-on REST call is accepted — the metal is
still POSTing / PXE-ing, minutes from ready;
- a **container** returns once the runtime schedules it — it may still be starting.
So `boot()` succeeding does **not** mean the instance is *up*. Before this seam a
consumer had only `Lifecycle::status` — a single instantaneous read — and had to hand-
roll its own poll-until-up loop (and its own timeout / never-comes-up handling) per
call site. That is exactly the duplicated orchestration draupnir exists to own once.
`await_power_state` provides the loop once; `boot_and_await` is the one-call
"provision and confirm up" seam jera / Skidbladnir actually want.
## The API (the only design — no alternatives)
```rust
pub struct WaitOptions {
pub timeout: Option<Duration>, // None = wait forever; Some(d) = timeout after d
pub poll_interval: Duration, // default 200ms
}
// Poll a Lifecycle until it reports `want`; Ok(()) the moment it does.
// A live status() error propagates; awaiting PowerState::Unknown is an Err(Spec);
// a bounded budget makes a never-reached state an Err(Backend) timeout, not a hang.
pub fn await_power_state<L: Lifecycle>(
lifecycle: &L, machine: &Machine, want: PowerState, opts: &WaitOptions,
) -> Result<()>;
// The one-call provision seam: validate + boot, then await PowerState::On,
// returning the live Machine only once it is actually up.
pub fn boot_and_await<B: Boot + Lifecycle>(
backend: &B, spec: &BootSpec, opts: &WaitOptions,
) -> Result<Machine>;
```
Design decisions, and why:
- **Built on the always-compiled `Lifecycle` trait, generic over the backend.** No
reach into any backend internal, no new dependency — the default build stays pure
std. That is what makes it **mock-testable with no live instance**: a scripted
`Lifecycle` transitions `Unknown -> Unknown -> On` and the driver proves
`poll -> compare -> Ok` end to end, and a stuck-`Off` node proves the timeout.
- **`boot_and_await` reuses `boot()` (validate + Boot), not a bespoke path.** An
invalid spec is rejected at `validate()` before the backend — or any wait — is
ever touched (asserted:
`boot_and_await_rejects_an_invalid_spec_before_booting`), so the readback never
waits on an unbooted instance. One boot seam, no duplication.
- **Awaiting `PowerState::Unknown` is a `Spec` error, up front.** `Unknown` means
"not observed"; a target of `Unknown` is nonsensical (it would spuriously "succeed"
the first time a backend could not read the state). Rejected before the first poll.
- **A `status()` error propagates as `Err`.** If the backend can't read the state
(BMC unreachable, socket gone), the readback itself failed — that is not a state to
keep polling through, it is surfaced immediately.
- **Timeout is opt-in, mirroring `RunOptions`.** `Default` / `poll_every` wait
indefinitely (parity with `run_to_completion`); `bounded(timeout, poll)` returns a
clear `Error::Backend` naming the instance and the **last-observed** state — a
booted-but-dead node is a diagnosable timeout, never a silent hang. `WaitOptions`
deliberately mirrors `container::RunOptions` field-for-field so the two readback
loops read the same.
## Tests (all no-instance, RED-when-broken)
`src/lib.rs`:
- `await_power_state_returns_ok_once_the_state_is_reached` — polls past two
`Unknown`s to `On` and returns Ok (asserts it actually polled ≥3 times).
- `await_power_state_times_out_when_the_state_is_never_reached` — a stuck-`Off` node
is a bounded `Error::Backend` timeout. **RED-when-broken**: drop the `observed ==
want` gate (return Ok on the first poll) and this + the reached-Ok test fail.
- `await_power_state_rejects_awaiting_unknown` — `want = Unknown` is `Err(Spec)`
before any poll (`status_calls == 0`).
- `await_power_state_propagates_a_backend_status_error` — a `status()` `Err`
propagates.
- `boot_and_await_boots_then_confirms_power_on` — boot then confirm On, returns the
live `Machine` (`power == On`).
- `boot_and_await_times_out_when_the_instance_never_comes_up` — a never-up node makes
the one-call seam a bounded timeout, not a hang.
- `boot_and_await_rejects_an_invalid_spec_before_booting` — an ISO-on-container spec
is `Err(Spec)` before the backend / any wait is touched.
`tests/testmatrix_rows.rs`:
- `await_power_state_confirms_a_boot_readback_across_backends` — records the
`draupnir/lifecycle` matrix row (reached-Ok · never-up-timeout · reject-Unknown ·
`boot_and_await` confirms On).
## Fleet-level readback — `boot_fleet_and_await` (draupnir 0.1.5, additive)
The single-node `boot_and_await` confirms **one** instance. A multi-machine
dispatcher (jera) boots a *fleet* from one image and needs to know **who** came up —
not to stop at the first dead node. `boot_fleet_and_await` is the fleet analogue: it
`plan_fleet`s the spec into `n` members and drives **each through `boot_and_await`**
(no per-backend twin — L5), rolling up a per-member verdict plus aggregate tallies.
```rust
pub enum MemberOutcome {
Up(Machine), // booted AND confirmed On within the budget — ready
Timeout(String), // Err(Backend): readback timed out / backend couldn't bring it up in budget
Error(String), // Err(Spec|Unsupported): hard misconfig — waiting can't help
}
pub struct MemberReadback { pub name: String, pub outcome: MemberOutcome }
pub struct FleetReadback { pub members: Vec<MemberReadback> } // in fleet order
impl FleetReadback { fn ready(&self)->usize; fn failed(&self)->usize; fn all_ready(&self)->bool; }
// Boots n members from one spec, awaits each to On under `opts`, rolls up who is ready.
pub fn boot_fleet_and_await<B: Boot + Lifecycle>(
spec: &BootSpec, n: usize, backend: &B, opts: &WaitOptions,
) -> FleetReadback; // infallible by construction
```
Design decisions, and why:
- **Reuses `boot_and_await` per member, not a bespoke fleet loop.** Each member gets
the exact same validate → boot → confirm-On path; the fleet call only fans it and
classifies. One readback seam, no twin (L5).
- **Infallible — returns a rollup, never an early `Err`.** A dead or misconfigured
member is a per-member `Timeout`/`Error` line, so a **partial fleet is always
observable** (parity with `boot_fleet`). The aggregate is `ready()` = the `Up`
count, `failed()` = the rest, `all_ready()` = non-empty and none failed.
- **Bounded per-member timeout is what makes it safe.** Each member is awaited
independently within `opts`, so one node that never powers on rolls up as a
`Timeout` while the healthy members are `Up` — it cannot hang the whole rollup.
**Pass a `WaitOptions::bounded(..)`**; the default unbounded options would block
forever on the first dead node (documented, same contract as `await_power_state`).
- **Three honest buckets from `boot_and_await`'s result.** `Ok` → `Up`;
`Err(Backend)` → `Timeout` (the backend did not get the node to `On` in budget — a
readback timeout or a transient backend failure while booting/polling);
`Err(Spec|Unsupported)` → `Error` (a hard misconfiguration or an unavailable
backend — retrying cannot help). The split lets a dispatcher retry `Timeout`s and
fail fast on `Error`s.
How jera calls it: build the shared `BootSpec` + fleet size `n` from the
Roster/WorkPayload, pass a bounded `WaitOptions`, then read `FleetReadback`:
`all_ready()` to gate the whole fleet, `ready()`/`failed()` for the tally, and the
`members[]` lines (`name` → `Up`/`Timeout`/`Error`) to decide which members to retry
(the `Timeout`s) versus reject (the `Error`s).
Fleet tests (all no-instance, RED-when-broken) — `src/lib.rs`:
- `boot_fleet_and_await_rolls_up_one_dead_member_as_timeout_others_up` — a fleet of 3
with a dead middle member rolls up node-2 = `Timeout`, node-1/node-3 = `Up`, ready
2 / failed 1. **RED-when-broken**: neuter the `observed == want` gate (dead node
returns Ok → ready 3) or flip the `Err(Backend) -> Timeout` arm to `Error` (the
`matches!(.., Timeout)` verdict fails).
- `boot_fleet_and_await_reports_a_fully_ready_fleet` — every member `Up`, `all_ready`.
- `boot_fleet_and_await_rolls_up_an_invalid_spec_as_error_not_timeout` — an invalid
spec is `Error` on every member (rejected at `validate()` before the backend), never
`Timeout` — proves the Spec/Backend bucket split.
- `boot_fleet_and_await_of_zero_is_an_empty_not_ready_rollup` — `n=0` is an empty
rollup, `all_ready()` false.
`tests/testmatrix_rows.rs`:
- `boot_fleet_and_await_rolls_up_who_is_ready` — records the `draupnir/lifecycle`
matrix row (partial fleet node-2 timeout / others up, ready 2 / failed 1, plus a
full-fleet all-ready arm).
## Parallel fleet boot — `boot_fleet_and_await_parallel` (draupnir 0.1.5, additive)
`boot_fleet_and_await` boots+awaits its members **serially**, so a fleet's wall-clock
is the *sum* of the per-member waits: 8 nodes on a 5-minute budget is up to 40 minutes,
one node at a time. But the per-member wait is almost entirely the backend's own boot
latency (a metal node POSTing, a guest booting) — idle time draupnir spends sleeping in
`await_power_state`. `boot_fleet_and_await_parallel` overlaps that idle time: it fans
each member onto the **gatling fork-join pool** (one worker per member) so all `n` are
booted+awaited at once and the wall-clock collapses to roughly the *slowest* member's wait.
```rust
// Same arguments + same FleetReadback as the serial call — just concurrent.
pub fn boot_fleet_and_await_parallel<B: Boot + Lifecycle + Sync>(
spec: &BootSpec, n: usize, backend: &B, opts: &WaitOptions,
) -> FleetReadback;
```
Design decisions, and why:
- **`gatling::gatling_forkjoin::gatling_for_each` — the ONE sanctioned threading home.**
ROOT-LAW #0 (rayon-free / core-saturation) forbids a bare `std::thread::spawn` in any
non-engine crate; all concurrency originates in gatling. The fork-join pool is scoped,
so each member still borrows the shared `&backend` / `&opts` and the fleet plan off the
stack with no `Arc`/`clone`, and it returns results in **index order** (deterministic
rollup). Worker count is pinned to `plan.len()` — **one worker per member**, NOT the
default core count — because `classify_member` is blocking-IO (park on a boot), not CPU:
every member needs its own parked thread so the wall-clock collapses to the slowest, a
deliberate exemption from the core-saturation law (which governs CPU hot paths).
- **Identical rollup to the serial path, by construction.** Both paths classify every
member through the **one shared `classify_member` step** (validate → `boot` →
confirm-On → bucket), and the parallel path **joins the member threads back in fleet
order** (`node-1`…`node-n`), not completion order. So the `FleetReadback` — per-member
`MemberOutcome`, member order, and the `ready()`/`failed()`/`all_ready()` tallies — is
byte-for-byte the same whichever path booted it. The parallel path changes *when*
members are awaited, never *what* the rollup says. This is asserted directly
(`parallel_fleet_boot_is_identical_to_serial_for_a_mixed_fleet`).
- **Still infallible, still one-dead-node-safe.** Each member is awaited on its own
thread within the bounded `opts`, so a dead node times out on *its* thread and is
joined as a `Timeout` line while the healthy threads finish independently — one dead
node cannot hang the others (same guarantee as the serial call, now truly parallel).
**Pass a bounded `WaitOptions`** for the same reason.
- **`B: Sync` bound — the serial path stays the general fallback.** Sharing `&backend`
across threads needs `B: Sync`. All three real backends satisfy it
(`KvmBoot`/`ContainerBoot` are `Arc<Mutex<..>>`, `RedfishBoot` is plain owned fields).
A backend that is *not* `Sync` simply calls the serial `boot_fleet_and_await`, which
is kept unchanged as the fully-general path — **the working path is never replaced**
(L2). Two functions, one shared classification, no twin (L5).
Parallel tests (all no-instance, RED-when-broken, mutation-verified) — `src/lib.rs`:
- `parallel_fleet_boot_is_identical_to_serial_for_a_mixed_fleet` — the equivalence
guard: a mixed fleet of 5 (node-3 dead → `Timeout`, node-5 unbootable → `Error`,
node-1/2/4 `Up`) rolls up to a `FleetReadback` **equal** to the serial one.
**RED-when-broken**: reversing the join order (or any bucket/member divergence) breaks
the `assert_eq!`.
- `parallel_fleet_boot_actually_overlaps_the_members` — the concurrency guard: a barrier
backend that records peak in-flight `status` calls proves **peak == n** (all members
awaited at the same instant). **RED-when-broken**: pointing it at the serial
`boot_fleet_and_await` collapses the peak to 1.
- `parallel_fleet_boot_of_zero_is_an_empty_rollup` — parity with the serial `n=0`.
`tests/testmatrix_rows.rs`:
- `boot_fleet_and_await_parallel_matches_the_serial_rollup` — records the
`draupnir/lifecycle` matrix row (parallel == serial over the mixed fleet).
## Perf / bench
`await_power_state` and the serial `boot_fleet_and_await` are poll loops gated on the
backend's own `status` latency (a QEMU handle read, a container-state query, a BMC REST
round-trip) — no compute hot path to guard, and a real measurement would time the
backend / the network, not draupnir. **The parallel fleet boot IS a clean measure,
though**: it overlaps exactly that per-member latency, so `benches/fleet_boot.rs`
(criterion, dev-dependency only — the lib's runtime deps are untouched) benches serial
vs `_parallel` over a fake backend whose `status` sleeps a small fixed latency modelling
the readback round-trip. The serial arm pays it `n` times in series, the parallel arm
once:
| n | serial | parallel | speedup |
|----|--------|----------|---------|
| 4 | 8.30 ms | 2.13 ms | ~3.9x |
| 8 | 16.53 ms | 2.18 ms | ~7.6x |
| 16 | 33.05 ms | 2.29 ms | ~14.4x |
(Loki 12c/64GiB, 2 ms per-member latency, light `--sample-size 10` run; serial ≈
`n × latency`, parallel ≈ one latency.) These are **indicative** — the authoritative
HEAVY numbers are the Loki bencher's; noted in `draupnir_agents_problems.md`.
## Related — teardown + functional-status emit
`await_power_state` / `boot_fleet_and_await[_parallel]` each record their verdict as a
`draupnir/lifecycle` `functional_status` row, and the *undo* seam **`rollback()`** tears
a booted instance back down (power-off through the same `Lifecycle` trait), likewise a
GREEN/RED row. The full table of emitting lifecycle surfaces — `boot` · `rollback` ·
`await_power_state` · `wait_ready` · the seed builders — and the RED-when-broken
`failed_boot_and_rollback_emit_red_rows` test are documented in
[functional-status-emit](functional-status-emit.md).