# TWIN DESIGN AUDIT — 2026-09-21 (lane T1)
Read of the code at `master ac02956`, plus branch `agent/stink-leaks` (`4d38319`, the
follower-supervision fix, NOT on master). Nothing here is quoted from a design doc without a
`file:line` behind it — several recorded claims in this estate have turned out stale, so every
answer below is read off the source.
**Scope note.** The owner ruled mid-audit: *"gunnar twin must be built EXACTLY like the main, it
MUST use the SAME iso"* and *"DO not build a special gunnar TWIN iso"*. §7 audits the code
against that target. The un-metered-twin question is closed by that ruling and is not argued
here.
**The short version.** The restore path EXISTS, end to end, and it has a name:
`gunnar control twin replay --twin <twin>:50051 --into <fresh>:50051`. It has been driven under
KVM and reached its last step. That is the good news and it is the thing the owner asked about.
The design is sound in its core and wrong in three specific places: the twin can never be the
recovery target (§5), the pairing has no notion of WHICH log it follows (§2, D1), and nothing
refuses a restore from a stale twin (§4, D2).
---
## 1. What does the twin actually hold?
The twin holds **the primary's transaction log, byte for byte, at the primary's own sequence
numbers**, and applies every frame through the UNLOGGED halves of its stores so it originates
nothing (`crates/gunnar-server/src/twin.rs:23-27`, `apply_raw` at `twin.rs:210`).
**Replicated** — the twelve frame kinds, `gunnar-core/src/txlog.rs:101-146`:
| Kind | What it carries | Covers |
|---|---|---|
| `Pack` (1) | pack bytes as the client pushed them | every git object |
| `Refs` (2) | every ref kind incl. `HEAD`, atomic and plain | the ref graph |
| `ControlBlob` (3) / `ControlDoc` (4) | the control store's blobs and document pointers | **catalog, identity table (principals, keys, grants, tokens, memberships, gates), PKI, mirrors, flow, pulls, policy, ENTITLEMENTS** (`txlog.rs:112-116`) |
| `Secret` (5) | the sealed bytes exactly as the secret store holds them | host key, issuer key, every minted secret |
| `Kek` (9) | the 32-byte key-encrypting key, **seq 1 of every primary's log** | without it every `Secret` frame is noise (`txlog.rs:128-133`) |
| `LfsPut`/`LfsDelete` (6/7) | LFS object bytes and tombstones | LFS |
| `RepoStore` (8) | store created / purged | purges propagate; a restore does not resurrect a deleted repo (test `twin/tests.rs:1035`) |
| `Append` (10) | one line of a signed append-only log | attestations, transparency heads, forge events, builds, membership |
| `File` (11) | whole documents under the data root | `replication.json`, `webhooks.json` (`txlog.rs:1452-1463`) |
| `SnapshotAt` (12) | never in a log — the snapshot stream's terminator | — |
Entitlements are a control document (`gunnar-core/src/entitlement.rs:49`, `ENTITLEMENT_REF`), so
the tenant/entitlement state IS replicated. That matters for §7.
**NOT replicated**, by construction (`txlog.rs:1420-1446`):
* `is_cache_path` — `gc/`, `quarantine/`, `meta/`, `objects.exploded*`, `objects.graph`,
`serialdat/`, `.tmp`, `incoming-*`, and **`grow-journal`**. The last one is real state: the
data-set growth journal is per box and a restored box starts with none.
* `is_per_box_path` — **`crash/` and `log/`**: the box's crash records and its console/audit
trail. A restored main comes back with NO audit history. Nothing warns about this.
* The metadata mirror dir is actively DELETED on a twin at join
(`twin.rs:671-688`, `forget_mirrors`) — identity mirror, `replication.json`/`webhooks.json`
mirror copies. They are derived, and the `File` frames carry the real ones, so this is correct.
* **Everything outside the data root**: boot params (`GUNNAR_NET`, `static_ip`, `hub.conf`),
`/etc/gunnar-clock/time_ed25519`, the medium's own configuration. These are *install*
parameters, not data — they come from the medium, not from the twin.
**Does any absence make a restore incomplete or silently wrong?** Three:
1. **`log/` — the audit trail is silently gone.** Every other class is either replicated or
re-derivable. The audit/console trail is neither, and the restore reports success without
mentioning it. *Rank: D6.*
2. **`grow-journal`** — a box mid-growth when it died restores with no journal.
Deliberate (`is_cache_path`), but undocumented as a restore consequence. *Rank: D7.*
3. **The entitlement facts survive but their VERIFIER does not.**
`EntitlementSlot::open` re-verifies every stored fact against `--monetize-pubkey` and, on a
fact that does not verify, logs `entitlement.boot.refused` and leaves the namespace **Free**
(`gunnar-server/src/entitlement.rs:311-324`). It is a `tracing::error!`, not a `?`. So a
restore onto a box handed a DIFFERENT monetize key silently downgrades every paying tenant to
Free, and the restore still says "restored". *Rank: D3 — this is the "incomplete or silently
wrong" case the question asks for.*
Positive finding worth recording: a fresh box's OWN secrets are given up at
`prepare_replay`/`forget_own_secrets` (`twin.rs:526-545`) and the unopenable-row case is dropped
loudly rather than bricking the box (`twin.rs:919-952`). Both were measured failures; both are
now handled. The KEK is re-minted at `reseal_after_restore` (`twin.rs:881`) so the dead
primary's key leaves the disk, and each step is framed so the twin following the restored box
gets the new key.
---
## 2. The pairing
**Join.** `gunnar control twin join --at <twin> --primary <main>` →
`Bootstrap.BecomeTwin` (`gunnar-cli/src/control/twin.rs:77-88`,
`gunnar-grpc/src/service/bootstrap.rs:161-187`) → `twin::become_twin`
(`twin.rs:644-664`). It mints `twin.key` (ed25519, 0600, per box, never framed), writes
`twin.json`, runs `prepare_replay` (which refuses unless the box is FRESH: bootstrap window
open, no enrolled key, no repository — `twin.rs:485-506`), marks the box a twin, forgets its
mirrors, sets `browse` and `writes_frozen`, and returns the public key line. The operator then
enrols that key on the primary as a machine principal in group `twin`.
**Who authenticates whom.**
* **Twin → primary: signed.** Every `Mirror.LogHead` / `LogFetch` / `SnapshotFetch` is an
`sshsig` from `twin.key` (`twin.rs:1267`, `1274`, `1304`). The primary authorises via
`require_log_reader` (`gunnar-grpc/src/service/mirror.rs:36-54`): **membership of group `twin`
is sufficient and bypasses the grant check entirely.**
* **Primary → twin: NOTHING.** The twin verifies no identity of the primary at all. The channel
is `plain_channel` (`twin.rs:1221`) — no TLS, no pinning, no host key. The twin trusts whatever
answers at the `host:port` in `twin.json`.
* **`Bootstrap.BecomeTwin` and `Bootstrap.Restore` are UNAUTHENTICATED RPCs**
(`gunnar-grpc/src/auth.rs:1832-1833`). Their only guard is that the target box is fresh. That
is the same trust model as `Bootstrap.Enroll` and is defensible — but it means the CONTENT of
a restore is unauthenticated: anyone who can reach a fresh box's control port can stream it an
arbitrary log, and it comes up as a "primary" with attacker-chosen admin keys and KEK.
**Re-join.** Refused. `become_twin` is idempotent for the SAME primary (returns the same key)
and errors for any other: *"this box is already the twin of X; a twin is never re-pointed"*
(`twin.rs:647`).
**Key rotation.** There is no twin-key rotation verb. `twin.key` is minted once at join and
never rotated (`twin.rs:652-657`). Rotation is done on the PRIMARY side by removing and
re-enrolling the principal — that is what `gunnar-cli/tests/key_rotation.rs:8-21` records having
been done by hand on the live appliance. *Rank: D8.*
**A re-imaged TWIN.** Works, at a price. The new box is fresh, `become_twin` succeeds, a NEW key
is minted and must be re-enrolled on the primary. Its head is 0, so if the primary has trimmed
retention at all, `Follower::step` takes the `first > wanted` branch and pulls a whole
`Mirror.SnapshotFetch` — which holds `Log::freeze()` for the ENTIRE copy
(`twin/snapshot.rs:216`). On a 44 GiB estate the primary's framed writes are stopped for the
whole transfer. There are stall deadlines on both halves (`mirror.rs:506`,
`control/twin.rs:564-585`), so it cannot hang for ever, but re-imaging a twin is a
production-freeze event and nothing says so at the door.
**A re-imaged MAIN — this is the defect.** `twin.json` pins an ADDRESS and nothing else.
`LogHeadResponse` carries `head_seq`, `first_seq`, `bytes`, `twin`, `applied_seq`,
`primary_head_seq`, `caught_up`, `state`, `why`
(`gunnar-proto/proto/gunnar/v1/mirror.proto:192-211`) — **no log identity, no box UUID, no
epoch.** So:
* a main restored FROM THIS TWIN continues at the twin's own numbers, and the twin follows on
correctly — this is the designed happy path and it works;
* a main re-imaged FRESH at the same address (wrong order of operations, or restored from a
different source) starts at seq 1. The twin holds head N. `Follower::step` computes
`wanted = N+1 > primary_head`, the fetch loop never runs, and it returns
`Progress::CaughtUp` (`twin.rs:1146-1179`). The twin reports state `following` and simply
waits. **When the new main's head passes N, the twin will begin appending the NEW box's frames
at seq N+1 on top of the OLD box's state — two histories spliced, silently.** The gap rule
protects against holes; nothing protects against a different log with the same numbers.
**Symmetry.** None. The relationship is entirely one-directional: the twin pulls, the primary
answers and records a heartbeat row. The primary has no notion of "my twin" beyond
group membership, and **any** member of group `twin` may pull the whole log — which is every
sealed secret plus the KEK (`mirror.rs:36-54`, and `mirror.proto:147` says so out loud). Group
`twin` is, in practice, root on the box. The design does assume main is immortal in exactly one
sense: it assumes the box at that address is always the same box.
---
## 3. The restore path — the thing the owner actually wants
**It exists. The verb is `gunnar control twin replay`.**
```
gunnar control twin replay --twin <twin>:50051 --into <fresh-main>:50051 [-i KEY] [--ignore-room]
```
`gunnar-cli/src/control/twin.rs:90-113` (`replay`) and `:154-217` (`pump_into`). Traced
concretely:
1. `replay` reads the twin's `Mirror.LogHead` as the operator (`control/twin.rs:95-98`).
2. If the twin's log starts above 1 (it installed a snapshot once), `pump_into` first drains the
twin's own `Mirror.SnapshotFetch` — items with `seq 0`, `SnapshotAt` last
(`control/twin.rs:165-176`).
3. Then the twin's log `first..=last` in ranges of 32 MiB (`control/twin.rs:136`), every chunk
stamped `restore_head = last` and `restore_bytes = head.bytes`.
4. Both are streamed straight into the fresh box's **unauthenticated**
`Bootstrap.Restore` (`service/bootstrap.rs:189-276`). The fresh box runs `restore_begin`
(`twin.rs:707-726`: refuses a twin, requires FRESH, writes the `txlog-restoring` marker),
checks room ONCE before the first byte lands (`twin.rs:765-805`,
`bootstrap.rs:209-218`), then `append_at` + `apply` per entry, durable first
(`twin.rs:824-827`).
5. `restore_finish` (`twin.rs:837-856`) refuses an empty stream and refuses a stream that ended
below the announced head — a SHORT stream is not a restore and the box stays resumable. It
writes the `txlog-reseal` marker before clearing `txlog-restoring`, so no open can see
"restored" without "re-seal".
6. Answer: `restart_required: true`. The box restarts and
`reseal_after_restore` (`twin.rs:881-977`) re-seals every secret under a KEK of its own,
framing each step so the next twin gets them.
A broken fetch mid-stream **poisons** the request body rather than cancelling it
(`control/twin.rs:150-153`, `Frame::Poison`), because tonic reads a client RST_STREAM(CANCEL) on
a request stream as a CLEAN end — that is exactly how this verb used to hand a fresh box a short
stream it then finished as a restore. Good fix; worth keeping.
**What `restore_drill.rs` proves.** `crates/gunnar-growbench/src/restore_drill.rs` scenario
**R6** (`:38-45`, `R6_NAME` at `:457`) is the real thing, under KVM, in Rust: the primary's data
disks are destroyed and the medium re-installed onto its system disk; the box must read FRESH
(bootstrap OPEN, the old admin refused); `gunnar control twin replay` streams the twin in; the
box restarts only if the answer said so; and then **every repository clones `fsck`-clean with
every file sha256-identical, the user list and the repository list equal what the primary held,
and the twin follows the restored primary.** Beside it: **R9** proves an interrupted replay
RESUMES rather than starting over (`:373`), **R10** proves a primary killed mid-push leaves no
listed-but-unclonable repository, **R8** proves a retention hole is closed by a real snapshot
(the twin's own `first_seq` must move off 1, which only `Log::restart_after` does).
**What it ASSUMES.** Three things, and they are named in the source:
* the media it installs are composed from `0bd1779`, which **predates `restore_head` entirely**
(`.nornir/twin-transaction-log.md:466-467`), so the short-stream refusal is NOT proven under
KVM — only by `pump_into` unit tests (`restore_drill.rs:27-36`);
* it assumes **two media** — `--iso` and `--twin-iso` (`growbench/src/main.rs:7,73,170`). Under
the one-medium ruling this becomes one `--iso` plus a role param. See §7.
* it has never run in the estate. The twin box is dark.
**The one thing that does NOT exist: an orchestrating verb.** `restore_drill.rs:38` describes
itself as *"the steps `cargo xtask restore main` takes"*. **`cargo xtask restore` does not
exist** — there is no `restore` arm in `xtask/src/main.rs` and no `restore.rs` in `xtask/src/`.
The re-image half is `cargo xtask reinstall-appliance` (`xtask/src/reinstall.rs`); the feed half
is `gunnar control twin replay`; nothing joins them, nothing checks the order, nothing checks
that the twin was current before the data disks were destroyed. The bench does that
orchestration in its own Rust and no shipped verb does. *Rank: D4.*
So the honest answer to the owner: **yes, main can be fed from the twin, the command is
`gunnar control twin replay`, and it has been driven to its last step under KVM. What is missing
is a single verb that does the whole rescue in the right order, and a check that the twin was
current when you started.**
---
## 4. Liveness and lag
**Two halves, both real.**
* *Primary's view.* Every twin poll carries a `TwinReport` — applied head, `statvfs` of the data
root, the follower's state word (`twin.rs:1240-1248`). `heartbeat::Registry::record`
(`gunnar-grpc/src/heartbeat.rs:158-178`) keeps the last one per principal plus a ring of
`behind_polls + 1` samples; `view` (`:181-231`) decides `ok` / `behind` / `silent` **at answer
time**, never on a timer — a twin that died cannot report its own death, so absence has to
read as `silent` (`heartbeat.rs:10-13`). `silent` after 60 s; `behind` only on a stuck head or
a monotonically growing lag, so a healthy constant small lag reads `ok` (`:105-132`). A group
member that has never polled gets a `silent` row (`:195-201`). This is well built.
* *Twin's view.* `primary_view` (`heartbeat.rs:243-264`) — `silent` when the primary has not
answered inside the bound. Surfaced on `Bootstrap.Status` (`bootstrap.rs:313`).
**Failure mode after this morning's fix.** Before `4d38319`, `start_follower` called `run()`
ONCE: any failure before the poll loop (no `twin.json` yet, an unreadable `twin.key`, a channel
that would not build) killed the thread for the life of the process and nothing retried it. The
commit's own measurement: the live twin at `applied 0 of 19264`, with an empty state word beside
it, blocking every growth in the estate for ever because `flushing` refuses to unmount a set
whose twin is behind. The fix retries every `FOLLOWER_RETRY` (5 s, no back-off ladder, on
purpose) and writes the failure INTO the status with the attempt count, so the heartbeat carries
`unreachable` instead of `""`. `SnapshotNeeded` stays a stop — it needs an operator's
`twin catchup` and re-running would write the same refusal every five seconds. This is the right
shape. **It is on `agent/stink-leaks` and NOT on master** (verified: `git merge-base
--is-ancestor 4d38319 master` fails).
**Residual: the registry is in-memory.** `Registry.rows` is a `Mutex<BTreeMap>`
(`heartbeat.rs:139`) with no persistence. A primary restart resets every row to "never polled
since this server started". Correct-by-default (reads `silent`), but it means the primary can
never tell you how long a twin has been dead ACROSS a restart. *Rank: D9.*
**Can a stale twin be used for a restore without anybody noticing? YES.** `replay` reads the
twin's `LogHead` and, if it is not caught up, prints **a warning on stderr and proceeds**:
```
crates/gunnar-cli/src/control/twin.rs:99-101
if head.twin && !head.caught_up {
eprintln!("warning: the twin says `{}` ({}); restoring what it holds, {}..={}", …);
}
```
There is no `--force`, no confirmation, no refusal. And it is worse than it looks in the case
that matters: **in the real disaster the primary is dead, so the twin's state is `unreachable`
and `caught_up` is false ALWAYS** — the warning fires on every genuine rescue and therefore
carries no signal. Nor can the operator measure the staleness: `LogHeadResponse` has no
`last_answer_unix_ms` field (`mirror.proto:192-211`), and `primary_head_seq` is itself the
twin's stale memory from its last successful poll. A twin that stopped following three weeks ago
and a twin that was level ten seconds before the primary died are indistinguishable to `replay`.
*Rank: D2 — this is the second most important finding in this audit.*
---
## 5. Promotion
**Promote-in-place is not supported, and it is refused by construction in three separate
places.**
1. `twin.rs:2` — the module's first sentence: *"applies them in order into stores of its own,
and is never promoted."* Repeated at `twin.rs:220` as the reason a twin may keep a frame kind
it cannot apply while a box being RESTORED may not.
2. `restore_begin` — `anyhow::ensure!(!stores.is_twin(), "a twin is never restored into; restore
into a fresh primary")` (`twin.rs:722`).
3. `become_twin` — *"a twin is never re-pointed"* (`twin.rs:647`). There is no un-twin verb:
nothing deletes `twin.json`, and `is_twin_root` (`twin.rs:463`) is read at every
`ServerStores::open` (`stores.rs:736`), which sets `writes_frozen` and `browse`
(`stores.rs:852-853`). A twin comes up frozen, for ever, by the presence of one file.
`Mirror.SetRole(PRIMARY)` (`service/mirror.rs:350-366`,
`control_plane.rs:5289-5343`) belongs to the **replication** subsystem — standby / browse
replica — which is a different and older mechanism. It does not promote a txlog twin, and a
browse replica refuses its own promotion by name (`control_plane.rs:5302-5313`). The two
mechanisms share the word "replica" and nothing else; that overlap is a documentation hazard in
its own right.
**Consequence, stated plainly.** The twin's own hardware can never become main. Recovery
requires a THIRD box — either main re-imaged, or a new VM. You cannot re-image the twin to make
it main, because re-imaging it destroys the only copy you are rescuing from. Under the owner's
words — *"if the whole main dies, the twin has config + data"* — this is correct as a RESTORE
SOURCE and incomplete as a recovery plan: the plan silently requires that a fresh box is
obtainable at the moment main dies. In an estate where Scaleway/UpCloud capacity has been
out-of-stock (memory: `scaleway-out-of-stock-2026-09-09`), "obtain a fresh box" is not free.
*Rank: D5 — this is a design decision to affirm or change, not a bug.*
The ruling of 2026-09-21 (one medium, role as an install parameter) makes promote-in-place
*cheap to add later if it is ever wanted* — the binary on the twin is already the same binary —
but it does not by itself create it. The refusals above are three explicit lines of code, not an
accident of packaging.
---
## 6. What is untested
In-process coverage is genuinely good: `crates/gunnar-server/src/twin/tests.rs` holds 25 tests
including late join, restart mid-range, a torn tail, the retention hole, restore rebuilding
every object AND every document, "the same box not just the same repositories" (`:623`), the
too-small-data-set refusal, the empty and short streams, an apply that failed being re-applied,
LFS delete and repo purge not resurrecting, the KEK re-seal and the twin following the new key,
snapshot catch-up then follow, unknown kinds in a log and in a snapshot, a broken snapshot
stream not leaking into the next install, a crash mid-install, and a fresh box's own secret not
bricking the box. Plus `gunnar-server/tests/twin_heartbeat.rs`,
`twin_snapshot_memory.rs`, `gunnar-cli/tests/twin_push_is_framed.rs`, and the KVM drills
`twin_drill.rs` / `restore_drill.rs` (H1, R8, R9, R10, R6).
**What nothing exercises:**
1. **A twin following a DIFFERENT log at the same address** (§2, D1). No test, no bench. The
failure is silent by construction.
2. **A restore from a demonstrably STALE twin** (§4, D2). No test asserts what `replay` does
when the twin is hours behind — because nothing refuses it.
3. **The whole rescue as one ordered procedure.** No verb (§3, D4), therefore no test of the
verb. R6 proves the STEPS; nothing proves an operator following a documented order gets them
in the right order.
4. **The short-stream refusal under KVM.** Named as unproven by the bench itself
(`restore_drill.rs:27-36`) — the media predate `restore_head`
(`.nornir/twin-transaction-log.md:466-467`). Proven only by `pump_into` unit tests.
5. **A restore whose entitlement facts do not verify** (§1, D3). Nothing drives
`EntitlementSlot::open` with a restored fact under a different monetize key.
6. **`Bootstrap.Restore`'s unauthenticated door.** `tests/auth.rs` asserts it is on the
unauthenticated list; nothing drives a hostile restore into a fresh box.
7. **Twin-key rotation** (§2, D8). `key_rotation.rs` drives the PRIMARY-side principal churn;
nothing rotates `twin.key` itself, because no verb does.
8. **`log/` and `grow-journal` absence after a restore** (§1). No test asserts what a restored
box has lost; R6 asserts only repos, users and catalog.
9. **A twin re-image against a trimmed primary**, i.e. the production freeze a full
`SnapshotFetch` costs. R8 makes a hole deliberately with `LogRetain`; nothing measures the
freeze duration on a real-sized set as an operational number.
10. **Two twins.** `require_log_reader` admits any member of group `twin`; the heartbeat
registry is keyed per principal and would show both. Nothing exercises it, and nothing
refuses it.
---
## 7. Against the one-medium ruling: where does the code assume the twin is a different KIND of box?
**Answer: almost nowhere. The ruling is cheap to satisfy, and that is a genuinely good finding.**
The role is **already** a runtime fact, not a build fact:
* **`GUNNAR_ROLE` does not exist anywhere in this repo** — zero hits across `*.rs`, `*.toml`,
`*.env`. Nothing is keyed off it today, so introducing it costs one env read.
* The real source of truth is **`<data-root>/twin.json`**, written at runtime by
`Bootstrap.BecomeTwin`, read by `is_twin_root` (`twin.rs:463`) at every
`ServerStores::open` (`stores.rs:736`), which then sets `twin`, `writes_frozen` and `browse`
(`stores.rs:852-853`). That file is in `is_per_box_path` (`txlog.rs:1445`), so it is never
framed and never travels — correct.
* The secondary source is the **boot argument `--role twin`**
(`gunnar-common/src/cli.rs:286-291`, `gunnar-server/src/lib.rs:337`). It does exactly two
things: it binds the git listener to loopback:0 (`serve.rs:994-999`) and it makes
`Bootstrap.Status` report `role = "twin"` (`control_plane.rs:1707`). It does NOT freeze
writes, does NOT start the follower, does NOT set `stores.is_twin()`. So a box booted
`--role twin` that has not yet joined is an ordinary fresh primary with no git listener —
which is exactly what `prepare_replay` then requires.
* **No runtime check reads a variant or medium name.** Grep for `GUNNAR_VARIANT`,
`variant_name`, `build_variant` finds only `gunnar-recipe` plumbing and a container-robot
test. The server never asks what it was built from.
* Identity, keys, data layout and config paths are all derived from the data root and from boot
params, never from the image. `twin.key` is minted at join, on the box, from
`gunnar_core::mint_ed25519()` (`twin.rs:653`).
**What actually encodes "a twin is a different image", and all of it is declaration, not code:**
| Where | What it says | Cost to remove |
|---|---|---|
| `.nornir/build-variants.toml:301` | `[[build.variant]] name = "gunnar-twin"` | delete the row (T2's `one-medium`) |
| `.nornir/build-variants.toml:2339-2360` | `[[build.installer]] name = "gunnar-twin"`, `variant = "gunnar-server"` (UNMETERED), `volume_id = "GUNNAR_TWIN"`, `korp.args=serve,…,--role,twin`, `ZNIPPY_GIT_EXPLODE=off`, `ZNIPPY_GIT_GRAPH_CACHE=off`, `korp.data.disks=vdb` | the ONLY differences from `gunnar-installer-metered` (`:1596-1613`) are the variant, the volume id, the `--role twin` arg and two cache switches. Everything else is byte-identical. |
| `crates/gunnar-cli/src/control/twin.rs:1460,1465,1474` | three refusal strings naming "the `gunnar-twin` variant" as the build with the `server` feature | three string edits |
| `crates/gunnar-growbench/src/main.rs:7,73,170,207` | `--twin-iso` — the bench installs TWO media | collapse to one `--iso` + a role param |
| `xtask/src/build.rs:106`, `brokkr.toml:207` | `gunnar-twin` named in the variant census / no-step list | census bookkeeping |
The two cache switches (`ZNIPPY_GIT_EXPLODE=off`, `ZNIPPY_GIT_GRAPH_CACHE=off`) are **already
redundant**: a twin turns both off itself at runtime — `mark_twin` calls
`engine.hold_no_graph_memo()` (`stores.rs:1119-1123`), `objects.exploded*` and `objects.graph`
are `is_cache_path` so no frame can create them (`txlog.rs:1428-1430`), and `forget_mirrors`
removes the mirror dir. So the one medium loses nothing by not carrying them.
**Does anything in the join/pairing path break when both boxes are demonstrably the same build?
No.** `become_twin` requires only that the target is fresh; the follower needs only `twin.json`
+ `twin.key`; `require_log_reader` checks only group membership. Nothing compares builds,
versions or images in either direction. Two boxes off the same medium pair exactly as two boxes
off two media do.
**The one thing the ruling DOES require of the code**, and it is small: `--role twin` must reach
the server. Today it reaches it as `korp.args=serve,…,--role,twin` on a twin-only installer row.
Under one medium it must come from an install parameter — `--param role=twin` →
`korp.env.GUNNAR_ROLE=twin` — which means adding an env read that maps `GUNNAR_ROLE=twin` to
`ServeRole::Twin`, defaulting to `Primary` when absent. That is one env lookup beside
`args.role` at `gunnar-server/src/lib.rs:337`. Nothing else in the server changes.
**Headline: nothing in this design fights the one-ISO ruling.** The code was already
role-at-runtime; only the build declarations and the bench's two-medium assumption are not.
---
# VERDICT
**The twin design is SOUND in its core and SALVAGEABLE at its edges. It is not wrong.**
The core — one ordered transaction log, the gap rule, durable-before-apply, apply through the
unlogged halves, the same `apply` serving follow and restore, the KEK as seq 1, the re-seal on
restore — is a good design, carefully reasoned, and visibly built out of measured failures
rather than out of a diagram. Several of the nastiest cases in this class of system (torn tail,
half-installed snapshot, short stream read as complete, a fresh box's own secret bricking the
restore, a frame kind a newer primary invented) are already found, fixed and regression-tested.
The restore path the owner asked about exists, has a name, and has been driven under KVM to its
last step.
What is wrong is not the mechanism. It is that **the twin is trusted without being checked**:
nothing verifies it is following the log it thinks it is, nothing verifies it was current when
you restored from it, and nothing walks the operator through the rescue in the right order. All
three are additive fixes. None of them requires restructuring.
**The right shape, concretely — four additions, no rewrites:**
1. **Give the log an identity.** A per-box UUID minted at first open, framed as part of seq 1
beside the KEK (or carried as a new `LogHeadResponse` field). The twin records it at join in
`twin.json` and refuses to follow a `LogHead` that reports a different one, by name. This
closes D1 and costs one field on the wire plus one comparison in `Follower::step`.
2. **Put a clock on `LogHead`.** Add `last_answer_unix_ms` (the twin's own
`TwinStatus::last_answer_unix_ms`, which already exists at `twin.rs:65` and is simply not on
the wire) to `LogHeadResponse`. Then make `replay` **refuse** a twin whose last successful
poll is older than a bound, with `--force-stale` as the named override and the age printed in
the refusal. This closes D2.
3. **Write the rescue as one verb.** `cargo xtask restore main` — the name
`restore_drill.rs:38` already uses. Order: prove the twin holds the watermark → re-image the
dead main → prove it reads FRESH → `twin replay` → wait for the control plane (the 180 s the
bench learned it needs) → verify a clone. The bench's R6 already IS this procedure in Rust;
lifting it out of the bench into a shipped verb is refactoring, not new design. This closes
D4 and makes D2's check unskippable.
4. **Make the entitlement mismatch loud.** `EntitlementSlot::open` should count refused facts
and, when the count is non-zero on a box that has just been restored
(`reseal_pending`/post-restore), refuse to start rather than run every tenant Free. This
closes D3.
Leave promote-in-place refused (D5), but **write the recovery precondition down where an
operator meets it**: the rescue needs a third box, and the twin is never it.
---
## Defects, ranked by what it costs to leave them
| # | Defect | Evidence | Cost if left |
|---|---|---|---|
| **D1** | **The twin has no idea WHICH log it follows.** `twin.json` pins an address; `LogHeadResponse` carries no log/box identity. A main re-imaged fresh at the same address is followed silently, and once its head passes the twin's, two histories are spliced at the same sequence numbers. | `twin.rs:420-443`; `mirror.proto:192-211`; `twin.rs:1094-1179` | **Silent, permanent corruption of the only copy.** The failure is invisible until the restore, and the restore then rebuilds the corruption faithfully. Highest cost of anything in this audit. |
| **D2** | **Nothing refuses a restore from a stale twin.** `replay` prints a stderr warning and proceeds; and in the real disaster the primary is dead, so the warning fires every time and means nothing. No wall-clock staleness is on the wire at all. | `control/twin.rs:99-101`; `mirror.proto:192-211`; `twin.rs:70-72` | **Silent data loss at exactly the moment the system exists to prevent it.** The operator gets a green "restored to seq N" for a box missing three weeks. |
| **D3** | **A restore with a mismatched monetize key downgrades every paying tenant to Free, silently.** `EntitlementSlot::open` logs `entitlement.boot.refused` per fact and continues. | `entitlement.rs:311-324`; `serve.rs:1063-1076` | Revenue leaking with a log line nobody reads, on the day of a rescue when nobody is reading logs. Stripe is LIVE. |
| **D4** | **No orchestrating restore verb.** `cargo xtask restore` does not exist; `restore_drill.rs:38` describes its steps as if it did. The pieces (`reinstall-appliance`, `twin replay`) are unjoined and the order is unchecked. | `xtask/src/main.rs` (no arm), `xtask/src/` (no `restore.rs`); `restore_drill.rs:38` | A rescue run by hand, under pressure, from a doc comment inside a bench. Destroying the main's data disks before checking the twin is one wrong order away. |
| **D5** | **The twin can never be the recovery target.** Promote-in-place is refused in three places and there is no un-twin verb. Recovery silently requires a third box. | `twin.rs:2`, `:647`, `:722`; `stores.rs:736,852-853` | A correct design decision with an unwritten precondition. Costs an outage of unbounded length when no fresh box is obtainable (cf. the Scaleway out-of-stock window). Affirm it and document it, or change it. |
| **D6** | `log/` (the console/audit trail) and `crash/` are per-box and never replicated; a restored main has no audit history and the restore does not say so. | `txlog.rs:1439-1446`; `bootstrap.rs:269-275` | Forensics gone after the one event worth investigating. |
| **D7** | `grow-journal` is a cache path; a box that died mid-growth restores with no journal. | `txlog.rs:1433` | A growth that cannot be reasoned about after a rescue. Deliberate but unrecorded. |
| **D8** | **No twin-key rotation verb.** `twin.key` is minted once at join and never rotated; rotation is done by hand on the primary side. | `twin.rs:652-657`; `gunnar-cli/tests/key_rotation.rs:8-21` | A long-lived credential that reads the whole box (secrets + KEK) and cannot be rolled without an operator improvising. |
| **D9** | The heartbeat registry is in-memory only; a primary restart forgets how long every twin has been dead. | `heartbeat.rs:139,161-169` | Fails safe (reads `silent`), but the primary can never say "this twin has been dead for six days". |
| **D10** | **Group `twin` is root.** Membership alone grants `Mirror.LogFetch` / `SnapshotFetch`, bypassing the grant check — the whole log, every sealed secret, and the KEK. | `service/mirror.rs:36-54`; `mirror.proto:147` | A machine principal added to one group can exfiltrate the entire box. Correct for the twin; too broad as a mechanism. |
| **D11** | Re-imaging a twin forces a full `SnapshotFetch`, which holds `Log::freeze()` for the whole copy — the primary's framed writes stop for the duration. Nothing warns at the door. | `twin/snapshot.rs:216`; `twin.rs:1096-1111` | A twin rebuild taken as routine becomes a production freeze measured in hours on a 44 GiB set. |
| **D12** | `Bootstrap.Restore` is unauthenticated, so the CONTENT of a restore is unauthenticated: a fresh box on a reachable network can be fed an arbitrary log and comes up with attacker-chosen admin keys and KEK. | `gunnar-grpc/src/auth.rs:1832-1833`; `bootstrap.rs:189-198` | Same trust model as `Bootstrap.Enroll` and defensible — but the blast radius is larger, and the fresh-box window is exactly when the box is on a network nobody is watching. |
| **D13** | The short-stream refusal (`restore_head`) is not proven under KVM; the bench's media predate the field. | `restore_drill.rs:27-36`; `.nornir/twin-transaction-log.md:466-467` | A named, deliberate hole. Closed by composing the media from a commit that carries `restore_head`. |
| **D14** | `4d38319` (follower supervision — the fix for a dead follower blocking every growth for ever) is on `agent/stink-leaks`, **not on master**. | `git merge-base --is-ancestor 4d38319 master` → false | The measured failure (live twin, `applied 0 of 19264`) is still live on master. T2 owns the merge. |
**On the one-medium ruling (§7): it is cheap.** Nothing in the server code assumes the twin is a
different kind of box. `GUNNAR_ROLE` does not exist yet, the role is already `twin.json` +
`--role twin`, no runtime check reads a variant, and the two twin-only cache switches are
already redundant at runtime. The work is: delete two declaration rows, map `GUNNAR_ROLE=twin` →
`ServeRole::Twin` beside `lib.rs:337`, fix three refusal strings, and collapse the bench's
`--twin-iso` into one `--iso` plus a role param. No part of this design fights the ruling.