<!-- nornir:class: reasoning -->
# kubernetes-target — reasoning (what was considered and rejected)
> Companion to [`kubernetes-target-design.md`](kubernetes-target-design.md), which
> states only the chosen shape. This file holds the alternatives, the why-nots, and
> the sources the analysis rests on.
## The frame Rickard set
> *"kubernetes has hooks that could solve stuff the deployer does, but it's just
> another way of solving them — both are right I think."*
That framing did the analytical work. Once you stop asking "which is better" the
table falls out on its own: for each capability there is a Kubernetes spelling, a
deployer spelling, and a question of whether our neutral vocabulary has a word for
it. Three of the most interesting findings only appear under that frame:
- Kubernetes **declines** to offer cross-service ordering. Under a "k8s is better"
frame that reads as a gap in us to close; under this frame it reads as a
capability we have and they refused, which is the correct reading.
- systemd's `StartLimitBurst=` has **no Kubernetes equivalent**. CrashLoopBackOff
is not tunable. The init-system family is richer in exactly one place, and it is
worth knowing which one.
- The deployer's observed worst failure (a node "sat in `booting` forever",
`HANDOFF-NEXT-MACHINE.md:45-51`) is precisely the bug class Kubernetes bounds
with `progressDeadlineSeconds` — and which Skidbladnir also does not bound. Two
designs sharing a hole is stronger evidence than either one having a feature.
## Alternatives rejected
### 1. Build a reconciliation daemon (an operator for Skidbladnir)
**Rejected.** The README's load-bearing claim is "no agent, no SSH, no shell on
the target". A reconcile-forever loop *is* an agent. What survives from the
operator pattern is the level-triggered **idempotency contract** — re-render and
re-apply must be a no-op when the target already matches — which is a testable
discipline rather than a process. `ensure_system_user`, `k8s_apply`'s server-side
apply, and the layout symlink swap are already idempotent; the work is to write
the contract down and assert it, not to build the loop.
### 2. Make lifecycle hooks a systemd-only knob
**Rejected**, and this was the closest call. `ExecStartPre=`/`ExecStartPost=`/
`ExecStop=`/`ExecStopPost=` map so cleanly that a systemd-shaped hook type is
tempting. But the four-slot model turns out to be honoured by **seven of eleven**
backends (systemd ×2, Quadlet, OpenRC, SysV, and — for `pre_start`/`post_stop` —
runit and s6-rc), and Kubernetes has an exact semantic twin for three of the four
slots. A knob at that coverage is not a systemd knob wearing a hat.
The one place we deliberately did *not* follow systemd: systemd allows an
arbitrary number of `ExecStartPre=` lines with `-` prefixes for
failure-tolerance. `HookAction` has no failure-tolerance flag, because Kubernetes
has none either ("If either a `PostStart` or `PreStop` hook fails, it kills the
Container") and a hook whose failure is ignored is a hook nobody can reason
about. If it is needed later it is additive.
### 3. Give `readiness` a host meaning
**Rejected.** The temptation is to define host readiness as "the health probe
passed", which would make the field render everywhere and look tidy. It would be
a lie: readiness only means anything when something *routes traffic* based on it.
On a host, nothing does. Declaring the degrade is worth more than a field that is
technically populated and operationally meaningless — this is the same principle
that keeps `renderer_for`'s `Option`.
`startup`, by contrast, does have a host meaning (a bounded poll before `start`
is reported successful, or `Type=notify` + `TimeoutStartSec=`), which is why it is
in and readiness is in-but-degraded.
### 4. Model `replicas` on the host as "run N copies"
**Rejected.** It would break the constellation law that every service runs as its
own dedicated user, it would require a supervisor Skidbladnir does not have, and
it would make `Lifecycle::health()`'s single `127.0.0.1:<port>` probe meaningless.
`replicas` stays k8s-scoped with a declared degrade.
### 5. Emit a Helm chart instead of raw manifests
**Rejected.** Njord's deployer *is* Helm and it works, so this was a real option.
Against it: (i) it re-introduces a templating layer between `ServiceSpec` and the
apiserver, and the exact-bytes contract tests in `k8s.rs` exist precisely because
we want the mapping to be inspectable; (ii) it puts a Go binary in an airgap
bundle, against L9's spirit; (iii) `k8s_apply` already talks to the apiserver
directly with server-side apply and needs neither `kubectl` nor `helm`.
What we *take* from Helm is the **hook tier**, which Kustomize deliberately lacks
("its lifecycle is left to kubectl/GitOps"). Skidbladnir's `RenderedService` is
Kustomize-shaped (pure manifests) while `install::apply` is Helm-shaped (an
ordered verb list with a release notion) — we already have both halves and had
simply never named the hook points between them (B5 in the design).
### 6. Render `after`/`needs` as init-container wait-gates by default
**Rejected as the default, kept as opt-in.** The semantics do match —
"the kubelet repeatedly restarts that init container until it succeeds" is exactly
a wait-gate. But it requires the dependency to be resolvable *by DNS from inside
the pod*, which silently converts a declared build-time ordering into a runtime
network dependency. The default is therefore to record the degrade and keep the
ordering guarantee where it actually holds — in `orchestrate::start_waves`, which
drives the applier.
### 7. Add the missing knobs per-backend as they come up
**Rejected** — this is the failure mode L5 exists to prevent, and the reason B8
(`RenderedService::degraded`) is sequenced first. Without an observable degrade
channel, "add it where it fits" produces a spec whose meaning depends on which
backend you happened to render, which is exactly the per-init bespoke install
scripts the crate set out to delete.
## On reading the Njord deployer read-only
Everything cited in the design chapter comes from `/home/rickard/talos-njord/`,
which is the cluster bring-up bundle plus the Helm values — not the application
repo (that checkout is on the T9 volume and was not mounted). Nothing was edited,
committed, or pushed there (L10). Secret **values** were deliberately not quoted;
only key *names* from `migrate/old-secrets.env`, because the shape of the secret
set (flat `KEY=VALUE` env, referenced not embedded) is the load-bearing fact and
the values are not.
Two things the bundle taught that no Kubernetes document would have:
- **Physical identity needs a stable key.** Talos selects the OS disk by WWID
because "sd-letters renumber when the boot-USB is pulled". This is the same
lesson as B6's numeric `uid`: a *name* that the environment can renumber is not
an identity. `unfold`'s Talos generator already gets this right.
- **Operational knowledge belongs beside the knob.** The single most valuable line
in `njord-new-values.yaml` is the comment explaining that nginx defaults
`client_max_body_size` to 1m and uploads therefore 413. A spec field with
nowhere to write down *why it is set that way* loses that knowledge on the next
redeploy.
## Sources
Kubernetes:
[container lifecycle hooks](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/) ·
[pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) ·
[init containers](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) ·
[Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) ·
[StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) ·
[DaemonSet updates](https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/) ·
[ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) ·
[Secret](https://kubernetes.io/docs/concepts/configuration/secret/) ·
[ServiceAccount](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) ·
[Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) ·
[Disruptions / PDB](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)
Helm / Kustomize:
[chart hooks](https://helm.sh/docs/topics/charts_hooks/) ·
[Kustomize vs Helm (Spacelift)](https://spacelift.io/blog/kustomize-vs-helm) ·
[Kustomize vs Helm (IBM)](https://www.ibm.com/think/insights/kustomize-vs-helm)
Talos:
[editing machine configuration — apply-config modes](https://www.talos.dev/v1.10/talos-guides/configuration/editing-machine-configuration/)
Njord bundle (read-only): `/home/rickard/talos-njord/{njord-new-values.yaml,
HANDOFF-NEXT-MACHINE.md, worker-join/README-ADD-SLAVE.md,
migrate/old-helm-values.yaml, migrate/old-secrets.env}`