<!-- nornir:class: idiot -->
# PROMOTE, for idiots
You have the same application running in several places — a test one, maybe a
"nearly-real" one, and the real one. You need to move new code up through them
without ever wrecking the real one, and sometimes copy the real one's data down
so testing runs against something true.
That is all this is. Nothing below assumes you know Rust, Kubernetes or what a
"slot" is.
---
## The words
**Slot** (also: site, environment, rung). One running copy of your application.
`bench`, `staging`, `apex`. Each has a namespace (where it runs) and one or more
hostnames (what users type).
**Ladder.** The slots in order, with arrows showing where code goes next.
**Guarded.** A slot the tool refuses to touch unless you turn two keys. In
practice: the real one.
**Generation.** A dated copy of a slot — `staging-20260721-1432`. Switching
between generations is how you roll back in seconds instead of restoring a
backup.
**Bundle.** A sealed file holding a copy of a slot's data.
---
## The one picture
```
bench ──► staging ──► apex CODE goes this way (up)
◄────────────◄───────────── DATA goes this way (down)
guarded ─┘
```
Two rules fall straight out:
* **Code goes up.** Try it on bench, then staging, then the real one.
* **Data comes down.** Copy the real one's data into a test slot. **Never the
other way** — that replaces the real data with test rows. The tool refuses it,
and there is no flag to make it agree.
---
## Step 1 — write down your ladder
One file. Copy this and change the names.
```yaml
apiVersion: skidbladnir.nordisk/promote.v1
meta:
customer: Contoso # anything you like; the tool just carries it
slots:
- name: bench
namespace: contoso-bench
hosts: [bench.contoso.example]
role: development
guarded: false
promotes_to: [staging] # from bench, code goes to staging
- name: staging
namespace: contoso-stg
hosts: [stg.contoso.example]
role: pre-production
guarded: false
promotes_to: [apex]
- name: apex
namespace: contoso
hosts: [contoso.example, www.contoso.example]
role: production
guarded: true # ← the real one
```
### Things that will bite you
**You must write `role:` and `guarded:` on every slot.** Leaving one out is an
error, not a shrug. That is deliberate: a slot that never said what it was should
never be assumed harmless.
**The name means nothing.** Calling a slot `prod` does not make it production.
`role: production` does. And a slot called `harmless` with `role: production`
**is** production. The tool reads what you wrote, never what you called it.
**`role: production` must have `guarded: true`.** Anything else is refused as
ambiguous.
**List every hostname.** A hostname you did not list belongs to no slot, and the
tool will refuse to act on it rather than guess.
**Do not number your rungs.** The arrows are the order. The tool works out
"depth" itself.
---
## Step 2 — the two keys
To do anything to a guarded slot you need **both**:
1. the `--allow-prod` flag (whatever your tool calls it) on this exact command,
**and**
2. an environment variable: `PROMOTE_ALLOW_PROD=1`
One alone does nothing. A flag on its own is a typo in a script; two are a
decision.
It must be exactly `1`. `PROMOTE_ALLOW_PROD=true` does **not** work. That is not
fussiness — it is so a leftover line in your shell profile cannot quietly arm
production.
If you get refused, read the message. It tells you which of the two keys is
missing.
---
## Step 3 — moving code up
```
promote bench → staging ✅ one step, fine
promote staging → apex ✅ one step (needs both keys — apex is guarded)
promote bench → apex ❌ refused: "it skips staging"
promote apex → staging ❌ refused: "that is the wrong way along the ladder"
```
The skip **can** be forced if you really mean it — and even then the tool hands
back the list of rungs you jumped so it ends up in the log. You cannot make a
skip invisible.
The wrong-direction one cannot be forced at all.
---
## Step 4 — copying data down
```
copy apex → staging ✅ normal
copy apex → bench ✅ normal
copy bench → apex ❌ always refused
```
Before it loads anything, the tool asks the target: *have you changed since this
backup was taken?*
Three possible answers, and **two of them stop you**:
| answer | what happens |
|---|---|
| "nothing newer here" | it proceeds |
| "yes, someone wrote here since" | it stops and asks you to confirm |
| **"I don't know"** | **it stops and asks you to confirm** |
That last row is the important one. "I don't know" is **not** a green light. If
the tool cannot prove the target holds nothing newer, it behaves exactly as if it
knows there is something newer. Guessing in that situation is how a day's work
disappears.
There is also a second check you might not expect. A timestamp only moves when
someone **writes**. If someone **deleted** two hundred records, the timestamp
does not move — it goes away with them. So the tool also compares **counts**,
because a count is the thing that goes *down* when records vanish. If the target
has fewer records than the backup, it stops.
---
## Step 5 — backups that are honest
The most dangerous thing in the world is a backup that looks fine and isn't.
The old version of this tool had exactly that bug: if it could not reach the file
storage, it backed up the database only, printed a small warning, and **exited
successfully**. Three days later that file looks identical to a good backup.
Now the backup file carries a verdict of its own:
* **complete** — every part came through.
* **incomplete** — something did not, and it says which part and why.
* **unknown** — nothing was even declared. (A backup that never really ran looks
like this. It is *not* treated as good.)
And when you go to restore:
```
$ restore --from backup.tar.zst
⛔ refusing to unseal backup.tar.zst: the capture that produced it was NOT whole.
verdict: incomplete
objects: skipped — object-store credentials were not configured
```
Note what it did **not** say: it did not say the file is corrupt. The file is
perfect. Every byte checks out. The *contents* are only half there — and a
checksum can never tell you that, which is why the verdict is written into the
backup at the moment it is made.
You can override it. You have to say so out loud, and the verdict still gets
printed afterwards.
---
## Step 6 — rolling out, and rolling back
When you push a new version, four things can happen. Your tool should say which:
| you see | it means | what to do |
|---|---|---|
| **rolled** | the new version is live | nothing. Done. |
| **rolled back** | the new one failed; the old one is back and serving | site is fine. Go fix the new version. |
| **rollback failed** | the new one failed **and** putting the old one back failed | **the site may be down. Go look, now.** |
| **no anchor** | the new one failed, and there was no old version to restore | the site may be down. Nothing was put back. |
Rows 3 and 4 are not "it failed". They are different phone calls at 2am, which is
why they are different words.
---
## Step 7 — generations, and the fast way back
Instead of overwriting a slot, you can stand a **new dated copy** of it next to
the old one and then point the hostname at the new one.
```
staging-20260715-0900 ← the old one, still there, still complete
staging-20260721-1432 ← the new one
stg.contoso.example ──► points at one of them
```
Rolling back is then: **point the hostname back**. Seconds. Nothing was
destroyed, so nothing has to be restored.
The switch happens in a fixed order, and it matters:
1. **wake** the new copy and wait for it to actually be up;
2. **point** the hostname at it;
3. **remove** the hostname from the old copy.
If you point first and wake second, there is a stretch of time — seconds to
minutes — where the hostname resolves to something that is not running. Users see
errors. That is the whole reason the order is fixed and not a matter of taste.
The old copy is **not** shut down by a switch. Leaving it running is what keeps
"switch back" fast. Shutting it down is a separate, later decision.
---
## Step 8 — the screen that shows you all this
The ladder can be drawn as a picture with a colour per rung. Four states, and
they are **not** three-plus-one:
| state | meaning |
|---|---|
| **never run** | nothing has been observed here yet |
| **green** | the last run here worked |
| **failed** | the last run here failed — and it always says why |
| **unknown** | something tried to look and could not — and it always says why |
**Never-run is not green. Unknown is not green.** If nobody has ever deployed to
a rung, the screen says so instead of showing a hopeful colour. If the tool could
not reach a rung — the machine was off, say — it says *that*, rather than either
"fine" or "broken".
Failed and unknown always carry a short reason code (`rollout-timeout`,
`node-unreachable`). You never get a red box with nothing in it.
---
## The five things to remember
1. **Write down `role` and `guarded` for every slot.** Names mean nothing;
what you declare is everything.
2. **Two keys for the real one.** A flag alone never works, and the value is
exactly `1`.
3. **Code up, data down.** Data upward is refused and cannot be forced.
4. **"I don't know" stops you.** Every time. That is the feature.
5. **A backup that is byte-perfect can still be half empty.** Read the verdict,
not the checksum.
---
## When it refuses you
Every refusal names what would make it work. If you hit one that does not, that
is a bug in the refusal and worth reporting — a message that only says "no" is
treated here as a broken message.
```
⛔ refusing: no slot is registered for namespace 'stagng'.
Registered slots: apex, bench, contoso, contoso-bench, contoso-stg, staging, …
Either you meant one of those, or this slot belongs in the ladder file — add it
there (with its `role` and `guarded`) and it becomes usable.
```
That is what a good one looks like: what you typed, what exists, and what to do.