# Growing the data set WITHOUT a restart — the runtime path, as built
Written 2026-09-14 against `gunnar` branch `data-set-grow` and `edda` branch
`data-set-grow`, the code half of `private-gunnar-ops/DATA-SET-GROWTH-FLOW.md`
§3 under ruling 1 (the appliance never goes down). `data-set-grow-drain.md`
beside this file is the design note it grew from; `DATA-SET-GROWTH-DESIGN.md`
§5 stays the contract for the engine's steps G1–G6 and their journal. This
file says what exists, where, and what the bench has to drive to prove it.
## The verbs
| verb | credential | does |
|---|---|---|
| `Admin.GrowStart{target_sectors_per_member, go_ahead, drain_timeout_secs, wait_members_timeout_secs}` | gunnar-admin | checks the go-ahead's structure, the signer policy and the nonce ledger; refuses if a growth is active, `/data` is not served, or the set is not complete; then runs the growth on its own thread and answers the first status |
| `Admin.GrowStatus{}` / **`Bootstrap.GrowStatus{}`** | Read / **none** | phase, `why`, epoch, target, the members (device, partition, disk/set/offered bytes, larger?), `members_verdict`, the journal, `engine_present`, `serving`, `identity_in_ram`, `in_flight`, `since`, `set_uuid`, `total_bytes` |
| `Admin.GrowResume{give_up}` | gunnar-admin | while `waiting-members`: look again now; `give_up`: stop waiting, assemble what is there, serve |
| `Mirror.FlushReplication{timeout_secs, resync}` | gunnar-admin | the watermark: flush, wait, report `{caught_up, in_flight, pending, bus_gaps, needs_full_resync, last_success, standbys, catalog_repos, verdict}` |
CLI: `gunnar control grow start|status|resume`, `gunnar control replication flush`.
`Bootstrap.GrowStatus` is the third credential-free RPC
(`gunnar_grpc::UNAUTHENTICATED_METHODS`), and T6 is why: the actor outside the
box that owns the volumes must read `waiting-members` / `failed` with nothing
to sign with, while the identity table is in RAM — or after a crash in that
window, when there is no identity table to sign against at all.
## The phases, and what each writes to `/serialdat/grow-journal`
The journal is the boot path's (`korp_installer::grow::Journal`, one line per
fsynced record, `epoch= target_sectors= member= step= at=`); the runtime steps
ride the same lines and `korp-start` reads past them to the open epoch.
| phase | on the box | journal |
|---|---|---|
| `draining` | the ONE admission gate is paused (`Refusal::Paused`, "gunnar.rs is growing its storage to make room for new members. Pushes are paused — please try again in a few minutes.", `EXIT_BUSY` over SSH, `503 Retry-After: 120` over HTTP); wait ≤ `drain_timeout` for `in_flight == 0` | on success `DRAIN` (epoch = `Journal::epoch_for(target)`); on timeout NOTHING — the gate reopens, `failed{aborted…}`, nothing changed (T1) |
| `unmounted` | `ControlDb::suspend` (documents, blobs, sealed secrets → RAM; `control.redb` closed); `ServerStores::suspend_repos` (handle cache emptied, engines' caches released, opens refused); `sync`; `umount2("/data")` (never lazy; `EBUSY` names the descriptors); a read-only tmpfs over the empty mountpoint; md `array_state ← clear`, the stale `/dev/md/korp-data` node removed | `UNMOUNT`, `STOP`, `WAIT`; a refused unmount → everything back, `ABORT` |
| `waiting-members` | `/sys/bus/pci/rescan` + survey every 2 s (15 s after the timeout) until every member is present, larger, equal within 1 % (`grow::detect`, the engine's own G1) and the same set uuid; `GrowResume` wakes it; the timeout only changes the words (T6) | `RESCAN` when eligible |
| `growing` | `korp_installer::grow::runtime::grow_and_mount`: G1–G3 → assemble → G4 → mount → G5 → G6; on any refusal the set is assembled and mounted at the size the members share | the engine's `G1 … G6`; `FAIL` when it did not grow |
| `mounting` | `ControlDb::resume` (file reopened, every document pointer compared with the RAM copy — drift is named in `why`), `resume_repos`, gate unpaused, `Ready` announced | `REOPEN` |
| `done` / `failed{why}` | `failed` still SERVES unless `why` starts with `FATAL` (cannot assemble or mount at all): then `serving: false`, identity stays in RAM, the gate stays paused, the control plane and `GrowStatus` keep answering | — |
`ABORT` closes an epoch (nothing was touched); `FAIL` keeps it open for the
next boot or the next `grow start`; a prelude-only epoch is joined by the
engine whatever target G1 measures, so a runtime growth is one epoch.
## How the torture rows are met
* **T1** — bounded drain, then abort with nothing changed; no transfer is ever
killed (`grow::tests::a_drain_that_times_out_aborts_with_nothing_changed`).
* **T2** — `gunnar_core::ControlDb`: both redb stores share one handle;
`suspend` copies the tables and closes the file (proven descriptor-free via
`/proc/self/fd`), reads answer from the copy, writes are `Error::OffStripe`
→ `UNAVAILABLE` (D7: refuse); `resume` verifies RAM == disk.
* **T6** — the phase never fails on the wait; `Bootstrap.GrowStatus` needs no
credential; a crash with volumes detached halts the next boot on a missing
member, and the actor outside re-attaches — then reboot finishes the epoch
from the disk, or `GrowResume` does.
* **T9** — every recoverable failure serves at the old size and names why;
only cannot-assemble/mount is fatal, and even then the control plane is up.
* **T11** — `Mirror.FlushReplication` between drain and unmount (step 0 of
§3): it proves every push the replicator saw is on the twin as of the
flush; it does NOT prove refs outside heads/tags, LFS, the control
database, visibility, tombstones, or repositories nobody pushed since the
twin existed unless `resync` ran. The verdict says both, in words.
## The go-ahead (D3)
`GrowStartRequest.go_ahead` is JSON: `{v: 1, signer: "operator"|"monetize",
target_sectors, nonce, issued_unix_ms, signature}`. The structure and the
nonce ledger (`/serialdat/grow-go-aheads`, one line per spent nonce) are
checked for every signer; WHO may sign is `gunnar_server::grow::GoAheadPolicy`.
The default `OperatorPolicy` accepts an unsigned `operator` go-ahead (the
gunnar-admin RPC is the approval) and refuses `monetize`; a metered build
plugs a policy verifying `signature` under monetize's Ed25519 key through
`GrowMachine::with_policy` (the entitlement facts' `monetize_embed::signing`
pair is the intended verifier). Not built here: monetize's minting side.
## `Draining{kind: growing}`
`Presence.Watch` does not exist in this tree. `GrowMachine::with_announcer`
is the hook: register the stream's fan-out and it receives
`Announcement::Draining{kind: "growing"}` before the pause and
`Announcement::Ready` after the reopen. Until then both are `tracing` rows on
`gunnar::grow` (`presence.draining`, `presence.ready`).
## The bench: a `--runtime` scenario for `gunnar-growbench`
What the bench should drive under KVM (software steps first, the hotplug
half when the kernel has it); the bench crate is another agent's and is not
edited here:
1. boot the appliance image built with `gunnar-server --features grow-runtime`
(needs the sibling edda checkout — see the Cargo.toml note) on 4 member
disks; bootstrap; push the manifest set; `replication flush` if a twin is
configured;
2. `gunnar control grow start --wait-members-timeout-secs 120` as
gunnar-admin; assert the answer is `draining` and that a `git push` started
now is refused with the growing sentence and exit 75 (and `git fetch` too);
3. poll `gunnar control grow status` (NO credential) until `waiting-members`;
assert `serving: false`, `identity_in_ram: true`, `/proc/mounts` on the
guest has no `/data` (a tmpfs is there instead), `/sys/block/md_korp_data`
is gone; the journal reads `DRAIN UNMOUNT STOP WAIT`; a control-plane
WRITE (`user add`) answers UNAVAILABLE naming the growth; a READ
(`user list`, `status`) still answers — the identity is in RAM;
4. resize every member: today's kernel (no hotplug) — QMP `block_resize` on
the four attached drives (virtio-blk raises a config change and `size`
updates in place); the hotplug kernel — QMP `device_del` ×4, resize the
backing files, `device_add` ×4, and assert the guest enumerates them;
5. optionally `grow resume`, else wait for the 2 s poll; poll status to
`done`; assert `total_bytes` grew, `Admin.Capacity` reads the new total
with `GROWTH_AT_RUNTIME`, the journal ends `… RESCAN G1 G2:0..3 G3:0..3 G4
G5 G6 REOPEN`, the manifest set verifies (design §7's "intact"), a push
lands, `replication list` shows the standby taking it;
6. the crash rows: halt the guest between `STOP` and the resize (C1b) —
assert the next boot HALTS on a missing member until the disks are
re-attached, then boots and finishes the epoch from the disk; halt during
`growing` — the boot path resumes at the next G-step (C2–C5 as today);
7. the refusal rows: resize 3 of 4 → status stays `waiting-members` with
`PARTIAL` in the survey; `grow resume --give-up` → `failed` (serving, old
size), the journal `FAIL REOPEN`; an unmount with a descriptor held open
→ `failed{aborted: … EBUSY … fd N → /data/…}`, journal `DRAIN ABORT`,
nothing changed.
Native tests cover every phase and every one of these transitions over a
scripted world (`gunnar-server/src/grow/tests.rs`); what only the bench can
prove is the real umount/md/xfs behaviour and, with the hotplug kernel,
step 4's detach/re-attach.
## What needs the hotplug kernel
Only step 4's second form. Everything else — drain, identity in RAM, close,
umount, md stop, the guard tmpfs, rescan (a harmless no-op), G1–G6, mount,
reopen — runs on today's kernel; a QMP `block_resize` of an attached virtio
disk is enough to exercise the whole runtime path under KVM. On UpCloud the
detach/resize/attach is the front-side watchdog's (D5), and the appliance's
half is complete.
## The registry — closed 2026-09-15
korp-installer **0.1.4** carries `grow::runtime` (released through `dwarves
crate-release` from edda `data-set-grow`), `gunnar-server`'s `grow-runtime`
feature resolves it from the registry, and the feature is in all four server
variants of `.nornir/build-variants.toml`. The developer patch config is gone.
## Proven under KVM (bench run 6, `DATA-SET-GROWTH-DESIGN.md` §14)
`gunnar-growbench` R1–R5: the clean runtime growth (QMP `block_resize` under
the running guest, today's kernel), the partial → give-up row, the power cut
in `waiting-members`, the drain timeout (T1), and the hot-plug form (unplug →
resize → re-plug ×4 on the `kernel-pci-hotplug` kernel, members behind PCIe
root ports). 128 → 256 GiB with no reboot on both kernels. What the run
found and fixed is in §14.