<!-- nornir:class: design -->
# REFRESH — the live-service upgrade verb (`Lifecycle::refresh`)
> The one command behind `korp-server --refresh` / `dwarves-full --refresh`: the whole
> build → install → migrate → restart dance as ONE idempotent, rollback-safe call.
> Materializes the super-plan §🔄 SERVICE REFRESH corrected design — **signed-bundle
> reinstall + self-migrate**, NOT an exe-swap — by reusing seams that already exist.
## Why
`Lifecycle` shipped install/start/stop/status/uninstall but **no refresh/upgrade** for a
LIVE service. Bringing a fresh HEAD onto oden meant a manual build → copy → restart dance
every time. `refresh` is that dance, made one verb, idempotent and rollback-safe.
## The seams it REUSES (no parallel machinery)
| need | reused seam |
|---|---|
| verify-on-receipt (integrity + optional ed25519 authenticity) | `fetch::fetch_and_verify` + `archive::TrustRoot` |
| versioned generation dirs + atomic `current` flip | `layout::Layout` (`prepare_product`, `activate`, `activate_version`, `previous_generation`) |
| unfold the whole product / roll a generation back | `install::{update_product-style unfold, rollback_product}` |
| restart without a shell | `systemd::Systemd` D-Bus (`reload` + `stop`/`start`) |
| health probe (TCP/HTTP) | `service::Lifecycle::health` |
| RON-orchestrated migration step | `orchestration::{Step::Migrate, MigrateModule}` |
New code is only the **glue** (`install::refresh_product`) + the **migration model**
(`migrate.rs`) + each product's `--migrate-config` verb.
## The migration model — stamped + ordered CHAIN (`src/migrate.rs`)
Migration code lives **IN THE PRODUCT**; Skidbladnir owns only the ordering.
1. **Stamp.** Every install writes the product's current schema version into
`<data_dir>/.migration-stamp.json` (a dotfile INSIDE the data root, so the uninstall
law erases it with the data — a wiped product forgets its schema too).
2. **Chain.** The product declares an ordered list of schema versions, oldest → head
(korp: `["korp-v0"]`; dwarves: `["dwarves-v0"]`). `migration_steps(stamped, chain)`
diffs the on-disk stamp against it:
- `stamped == head` ⇒ **no hops** — this is the idempotency guarantee.
- `stamped` earlier ⇒ each consecutive `(from → to)` hop up to head.
- `stamped` NOT in the chain ⇒ **hard error** (downgrade/corruption never silently
"catches up").
3. **Replay.** `replay_migrations` runs each hop through a `MigrateRunner`. Production is
`ProductBinRunner` — it `exec`s the NEW generation's own
`<bin> --migrate-config from=X to=Y` (direct argv, **zero-shell**: no `sh -c`, the same
direct-exec a systemd `ExecStart=` does; the product runs its own Rust migration). The
stamp is written after every SUCCESSFUL hop, so a failed hop leaves a **resumable**
stamp (retry resumes precisely, no double-apply, no skip). Tests inject a
`RecordingRunner` and assert the exact hops offline.
An absent stamp is treated as a brand-new data root (stamp head, migrate nothing) — an
honest statement, not a paper-over; a product introducing the stamp onto a pre-existing
deployment backfills the stamp first.
## The engine — `install::refresh_product` (§🔄 exact order)
1. **Verify-on-receipt** the bundle (refuse to unfold a single byte of a tampered one).
2. **Unfold** the whole product into `software/<newver>/<name>` — `current` NOT flipped
yet (data + log preserved).
3. **Migrate** — replay `stamped → head` against the preserved data using the INCOMING
binary, while the OLD generation is still live.
4. **Flip** `current` atomically to the new generation.
5. **Restart** over D-Bus (daemon-reload + hard stop→start; zero-shell).
6. **Health-probe** the running service.
7. **Roll back** on an unhealthy probe: flip `current` back to the prior generation
(atomic symlink swap) + restart. A rolled-back refresh is a **SAFE outcome** (the
service is healthy on the prior gen), reported as such — not a fault. The additive
forward-migration is honestly left in place after a binary rollback.
Restart + health are injected via the `RefreshControl` trait (`SystemdRefreshControl` in
prod; scripted in tests) so the whole orchestration — including rollback — runs offline.
## The product half
Each product owns its chain + a `--migrate-config from=X to=Y` verb that REALLY validates
the hop (identity or adjacent-forward only; unknown/garbage refused — never a stub that
fakes success). Today both korp and dwarves declare a single head, so the only legit hop
is an honest identity no-op and a fresh install's refresh runs zero hops. `--refresh`
(no arg) builds from HEAD via `cargo build` (a sanctioned build-host action, target on
`/scratch/cargo/<repo>` per the oden disk law) + packs the bundle; `--refresh <bundle>`
installs an appointed one.
## Tests (all green, offline)
- `migrate`: chain math (`v0→v1→v2`), idempotency (`stamped==head` ⇒ 0 hops), unknown
stamp refused, absent stamp = fresh, resumable failed hop.
- `install::refresh_is_idempotent_and_flips_current` — install v1, refresh→v2 flips
`current`, re-refresh is a proven no-op.
- `install::unhealthy_refresh_rolls_back` — an unhealthy new generation rolls `current`
back to the prior one, restarts on it, and leaves the forward-migration in place.
- `orchestration::ron_migrate_step_replays_via_module` — the RON `Migrate` step replays a
chain through `MigrateModule` + a runner.
- korp `migrate_config_validates_the_chain` (+ proven LIVE: identity ok, bad version
exits 1). dwarves mirrors it.
## Live status (oden, 2026-07-29)
`--refresh` is code-complete, builds, and its engine is proven by the offline
rollback/idempotency tests. A live full-server refresh of the running korp-server was
NOT run: it is a **root-owned system unit** (`/etc/systemd/system/korp-server.service`,
serving the map on :50051) and a second `rustfs-embed` server would collide on :9000.
`--migrate-config` is proven live. The clickable proof (map serving tiles + korp UI +
dwarves UI on the RTX 4090) is documented in the super-plan.
## KUBERNETES REFRESH — the K8s parity (`KubeRuntime::refresh`, `src/kube_runtime.rs`)
The same verb, one substrate over. A product promotes a new software version
IDENTICALLY on Metal (systemd `current`-flip + D-Bus restart) and on Kubernetes
(Deployment rolling-update over the apiserver) — the **same stamped, ordered
migration model** drives both. `src/kube_runtime.rs` is the K8s half; it reuses the
product-owns-its-migration doctrine and the pure chain math of `src/migrate.rs`
wholesale, changing only the substrate the five refresh steps run on.
### The mapping — each systemd step to its K8s equivalent
| systemd (`refresh_product`) | Kubernetes (`kube_refresh`) |
|---|---|
| **deliver** — unfold the new bundle into `software/<newver>` | SSA-apply the Deployment carrying the NEW container image (`rolling_deployment`) |
| **migrate** — replay `stamped→head` via the NEW binary's `--migrate-config` | a pre-upgrade migration **Job** per hop, image is the NEW image, args are `--migrate-config from=X to=Y` (`migration_job`); the rollout is gated on Job success |
| **swap** — atomic `current` symlink flip | the Deployment **RollingUpdate** (a new ReplicaSet supersedes the old) |
| **restart + health** — D-Bus restart, then the TCP/HTTP probe | the readiness/liveness gates — the rollout only completes when the new pods are Ready (`wait_rollout`) |
| **rollback** — flip `current` back to the prior generation + restart | re-apply the prior pod spec — k8s brings the prior **ReplicaSet/revision** back (the `rollout undo` equivalent), waited on again |
### The migration model is UNCHANGED — only the stamp's home moves
`migrate::migration_steps` (the pure `stamped → head` chain diff) is reused verbatim:
`stamped == head` yields zero hops (idempotency), a stamp outside the chain is a hard
error (no silent downgrade), and each hop is stamped forward on success so a failed
run resumes precisely. The one difference from the systemd side is where the stamp
lives: on Metal it is `<data_dir>/.migration-stamp.json` beside the data; on Kube it
is a small **ConfigMap** the engine reads before and writes after each good hop (the
migration Job mounts the product's PVC so `--migrate-config` reads and writes the SAME
data root the running pods do). The Job image is the NEW image because migrations are a
property of the incoming code — exactly as `ProductBinRunner` execs the NEW generation's
binary on Metal.
### The exact order (`kube_refresh`, §🔄 corrected design, K8s)
1. **Anchor.** Read the Deployment's current container image FIRST, before any mutation
— a rollback needs to know where it is going back to, and the moment after the apply
is too late to ask.
2. **Migrate.** Read the stamp; compute `migration_steps`. An absent stamp is a
brand-new deployment (stamp head, run nothing). Each hop runs a `migration_job` on
the new image; on success the stamp advances. A **failed hop aborts BEFORE any
delivery** — the old Deployment is untouched and still serving (outcome
`MigrationFailed`), the analog of a systemd migration failing before the flip.
3. **Deliver + swap.** SSA-apply `rolling_deployment` with the new image; k8s runs the
RollingUpdate (`maxUnavailable: 0`, `maxSurge: 1` — zero-downtime).
4. **Health.** Wait for the rollout to become Ready (updated == ready == available ==
desired).
5. **Roll back** on an unready rollout: re-apply the prior image's spec so k8s scales
the prior ReplicaSet back up, and wait again. No readable anchor yields `NoAnchor`
(nothing restored — never reported as a rollback); a failed rollback yields
`RollbackFailed` (the environment may be DOWN — never collapsed into "rollout
failed"). A clean `RolledBack` is a SAFE outcome (serving the prior image), reported
as such.
Restart/health/apply/migrate are injected via the `KubeRefreshOps` seam — the K8s analog
of the systemd `RefreshControl` + `MigrateRunner` injections — so the whole orchestration,
rollback included, runs offline against a scripted mock.
### Additive, gated, zero-shell
The engine (`kube_refresh`), the manifest renderers, and the injected-mock tests compile
in the lean core with NO `kube` (the seam is sync, `serde_json`-only). Only the ONE
production impl, `kube_runtime::live::KubeRuntime`, is behind the OFF-by-default
`kubernetes` feature; it bridges each sync seam call to async kube-rs over its own tokio
runtime (the same sync→async bridge `unfold`'s Talos target uses), reusing the existing
`k8s_apply` engine (`apply_json_manifests` + `run_job`) — no `kubectl`, no `helm`. korp
consumes Skidbladnir with `default-features = false`; its `kube` feature opts into
`skidbladnir/kubernetes` and nothing else changes.
### Tests (all green, offline)
`src/kube_runtime.rs`: golden renders — the migration Job (new image + exact
`--migrate-config from=X to=Y` argv + PVC mount + `backoffLimit: 0` + `restartPolicy:
Never`), the refreshed Deployment (new image + RollingUpdate + readiness/liveness), the
HTTP-probe form, the no-probe form, and the rollback target (prior-image Deployment,
everything-but-the-image identical so k8s reuses the prior ReplicaSet). The engine over a
scripted `KubeRefreshOps` mock: a good refresh migrates-then-delivers and reports the hops
(anchor read first, all Jobs before delivery, stamp advanced to head); `stamped == head`
runs zero Jobs; an absent stamp stamps head; a failed migration Job aborts before any
delivery and leaves a resumable stamp with the old image still live; an unready rollout
rolls back to the prior revision; a failed rollback is its own outcome; a missing anchor
says nothing was restored; a stamp outside the chain is refused; the dry run is pure.
### Live status
Offline-proven end to end (the five-step orchestration, the migration-Job + stamp
lifecycle, the manifest renders, and the rollback path). What still needs a live cluster
is the same class the systemd side defers: the real rollout timing and pod-readiness
cadence under the apiserver (the `wait_rollout` poll is `#[cfg]`-gated over kube-rs like
`k8s_apply::wait_ready`, not unit-tested against a live control plane). The `KubeRuntime`
production impl builds clean under `--features kubernetes`.
## WATCH-REFRESH — the deferred sub-verb (`Lifecycle::watch_refresh`, `src/watch.rs`)
The always-fresh oden: **poll git `HEAD`; refresh on change; no-op when unchanged.** It
adds no parallel machinery — it is a thin DECISION in front of the `refresh` engine above.
### The decision (`watch_refresh_once`)
1. **Read `HEAD`** through **gix** (`GitHead`, zero-shell — never `git rev-parse`),
peeling `HEAD` to its 40-hex commit id.
2. **Diff** it against the persisted `WatchStamp` (`<data_dir>/.watch-refresh-stamp.json`
— same dotfile-inside-data posture as the migration stamp, so the uninstall law wipes
it with the data).
3. **Unchanged** (`stamp == HEAD`) ⇒ honest, logged **no-op**; stamp untouched.
4. **Moved** (or **first-ever**, no stamp) ⇒ run the injected `RefreshTrigger`
(build-from-HEAD → pack → `refresh_product` in production), then advance the stamp to
the new `HEAD` **on success only**. A trigger error propagates and leaves the stamp
UNADVANCED, so the next tick retries the same commit — restart-safe, no double-apply,
no skip. `seed_watch_stamp` lets the deployer set the post-install baseline so the very
first tick is a clean no-op instead of a (safe, idempotent) redundant refresh.
### Timer, not an in-process daemon (the chosen mechanism + why)
The refresh engine is already idempotent + crash-safe with ALL resumable state on disk
(the schema stamp + the watch stamp). A long-lived in-process loop therefore buys nothing
and costs supervision, reboot-persistence, and a runtime held alive only to sleep — and on
a crash it must re-read the stamp anyway. So the wired production mechanism is a **systemd
`.timer` + `Type=oneshot` `.service`** (`render_watch_units` / `install_watch_refresh_timer`,
installed over the SAME zbus path as everything else): each tick is one `watch_refresh_once`
that reads the stamp, acts, advances it, and exits; systemd owns the cadence,
restart-on-failure, and (`Persistent=true`) firing a tick missed while the host was off.
The oneshot is `Restart=no` (the *timer* re-runs it) and carries no `[Install]` (the timer
pulls it). A bounded `watch_refresh_loop` (injected `Ticker`; `SleepTicker` parks the
thread — no busy-wait) is a documented foreground/dev alternative; a product that already
runs a tokio runtime can drive the same atom from a `tokio::time::interval`.
### Additive + gix behind a feature
The HEAD reader is an injected `HeadSource` seam, so the whole poll/diff/stamp decision +
the timer renderers compile in the lean core with NO gix. Only the ONE production impl
(`GitHead`) is behind the OFF-by-default `watch-refresh` feature (`dep:gix`, the lean
`revision`/`sha1` leaf). korp/dwarves consume Skidbladnir with `default-features = false`
and do not enable it, so their builds are unchanged — a product opts in when it wires
`--watch-refresh`.
### Tests (all green, offline)
`src/watch.rs`: unchanged-HEAD no-op (trigger never runs); moved-HEAD invokes the refresh
with the prior commit + the stamp advances, then a second tick is a proven no-op;
first-ever tick refreshes with `previous=None`; a failing refresh leaves the stamp for a
precise retry (recovering retry re-runs the SAME hop); the bounded loop refreshes only on
change; the oneshot/timer renderers are asserted (Type=oneshot, Restart=no, OnUnitActiveSec,
Persistent, `Unit=`, user-scope drops `User=`). Behind `--features watch-refresh`, a REAL
gix test inits a repo, commits, and proves `GitHead` resolves the exact `HEAD` and sees a
second commit as a MOVE.
### Live status
Offline-proven end to end (the decision, the stamp lifecycle, the unit render, and the real
gix HEAD read). What still needs a live host: the actual timer *tick cadence* under systemd
(the `.timer`/`.service` install is `#[cfg(unix)]` over zbus, not unit-tested, exactly like
`install_systemd`) and a full live build-from-HEAD refresh — both gated by the same
root-owned-`korp-server` / `:9000` collision that held back the live `--refresh` above.