Gunnar

nordisk/Skidbladnir

.nornir/socket-deployment-for-idiots.md

<!-- nornir:class: idiot -->
# Sockets, for idiots

## The one idea

You built a thing. A **bulb**. It might be a program, a VM image, a bootable ISO, a network
appliance. To find out whether it actually works, you have to plug it into something: a screen,
a network, a DHCP server, a cluster. That "something" is the **socket**.

A light bulb does not carry its own socket around. Neither does your build artifact. You say
what KIND of thing it is, and the right socket gets screwed in for you.

That is the whole idea. Everything below is detail.

## Two words you need

- **Bulb** = the thing you built. Its *type* is a string like `exe`, `iso`, `tunnr-appliance`.
- **Socket** = everything the bulb needs around it to run and be watched: a network, a DHCP
  server, a DNS resolver, a WireGuard tunnel, some consoles.

Each piece of that surrounding stuff is a **fixture**.

## Doing it, keystroke by keystroke

### 1. Turn the feature on

In your `Cargo.toml`:

```toml
[dependencies]
skidbladnir = { version = "0.1", features = ["sockets"] }
```

It is off by default. Nothing about your existing build changes when you turn it on.

### 2. Say what kind of thing you built

```rust
use skidbladnir::socket::{BuildThingType, socket_for, Mode};

let ty = BuildThingType::from_kind("tunnr-appliance").unwrap();
```

`from_kind` returns `None` if you typo it. It does not guess.

### 3. Get the socket

```rust
let socket = socket_for(ty);
```

You did not choose a socket. You said what the bulb was, and the right socket came back. That is
the point.

### 4. Look before you boot

```rust
for row in socket.plan(Mode::Test).topology() {
    println!("{} -> {}", row.kind.as_str(), row.endpoint.as_deref().unwrap_or("(no address)"));
}
```

```
test-net       -> (no address)
dhcp           -> 10.9.0.1:67
dns            -> 10.9.0.1:53
wireguard-peer -> 10.9.0.1:51820
nocloud-seed   -> (no address)
kvm-boot       -> (no address)
```

Nothing has booted yet. This is the *plan*. You can put it on screen immediately, which is why
the panel is never blank while you wait.

### 5. Light it

```rust
let mut dep = socket.light(Mode::Test);
```

Now ask what you got:

```rust
println!("{}", serde_json::to_string_pretty(&dep.json()).unwrap());
```

Every fixture will say **`ready`**. Not `serving`. **Read the next section, it is the one that
matters.**

## The rule that will confuse you, explained

> **`ready` does not mean it works. It means we set it up and nothing has told us otherwise.**

Software lies about this constantly. A dashboard shows green because a config file existed, not
because anything ever answered a request. This module refuses to do that.

To get from `ready` to `serving`, you must show it **proof from the actual boot**:

```rust
let console = /* the text your VM printed while booting */;
dep.confirm(&console);
```

`confirm` reads that text. If it finds `lease of 10.9.0.137 obtained`, DHCP becomes `serving`,
because a machine really did get an address. If it finds nothing, DHCP stays `ready` — *not*
failed, just **not proven**.

And if it finds proof of *failure*, the fixture goes **`down`**, and tells you the exact line:

```
dns   down   the boot console shows dns did not come up: resolve: Temporary failure in name resolution
```

Three outcomes, and you should be able to recite them:

| It found | Result |
|---|---|
| proof it worked | `serving` |
| proof it failed | `down`, quoting the line |
| nothing conclusive | unchanged — and it tells you what it looked for |

Failure always beats success. If the console shows a lease *and* shows the tunnel dying later,
you get `down`. We would rather alarm you wrongly than reassure you wrongly.

### "Proof" is stricter than you think

This is where people write a fake console for a test and get confused.

**Trying is not succeeding.** These lines look like success and are not:

| line | what it really is |
|---|---|
| `udhcpc: broadcasting select for 10.9.0.157` | asking for that address. Not having it |
| `udhcpc: broadcasting discover` | shouting into the void |
| `Sending handshake_initiation` | said hello. Nobody replied yet |
| `HANDSHAKE(REKEY_TIMEOUT)` | gave up and is retrying. This is a **failure** |

The lines that *do* count:

| line | why |
|---|---|
| `udhcpc: lease of 10.9.0.157 obtained from 10.9.0.1` | it was actually given the address |
| `Received handshake_response` | the other end answered. The tunnel exists |

And a lease only counts if the address is **inside the pool** the socket handed out. If your VM
fell back to a static `10.9.0.55`, that is not a lease no matter how much it looks like one.

If you need a realistic console for a test, do not invent one — copy from the `transcripts`
module in `socket.rs`. Those are real captures.

## Test mode vs Prod mode

Same socket, two postures.

- **`Mode::Test`** — open network, DHCP handing out addresses, a WireGuard key panel you can
  paste keys into, no passwords. Convenient, and completely unsuitable for real use.
- **`Mode::Prod`** — closed network, static addressing, real credentials, no key panel.

Now the part people trip on:

```rust
let dep = socket.light(Mode::Prod);
assert!(!dep.all_ok());   // <- this is CORRECT
```

**A Prod socket is currently unhealthy on purpose.** Here is why, in plain terms.

The test network is provided by QEMU's built-in "user mode" networking. It has a little DHCP
server and a little DNS forwarder baked in. They are excellent for testing. They are *not* a
real network — nothing outside the VM can route to them.

So when you ask for Prod, the honest answer is not "sure, here you go" while quietly handing you
the test fixtures. The honest answer is a refusal, per fixture, with reasons:

```rust
for (kind, why) in dep.refusals() {
    println!("{}: {}", kind.as_str(), why);
}
```

```
test-net: refused in Mode::Prod: test-net would be served by the test-grade runtime
          'slirp-builtin', and a Prod socket does not serve test fixtures
dns:      refused in Mode::Prod: dns would be served by the test-grade runtime
          'slirp-builtin', and a Prod socket does not serve test fixtures
nocloud-seed: a Prod NoCloud seed must carry real credentials; this socket layer mints
          none, and it will not ship the Test seed's auth-less image
```

Not everything is refused. The WireGuard peer is real software (boringtun) and the VM boot is the
real boot, so those stay healthy. Prod hardens; it does not blanket-fail.

**Show those lines to the user.** Do not show a spinner. Do not show a "Deploy" button that does
nothing. The refusal *is* the information.

**Every refusal tells you how to fix it.** Look for `TO FIX:` in the message — the network one
says "implement `NetworkBackend` with `is_test_grade() -> false`", the seed one says "wire in
`pki` or `openbao`". That is the actual work, named. Nothing in this system is allowed to say
"nope" without saying "…and here is what would change it".

## The WireGuard key panel (Test mode only)

```rust
dep.add_wg_peer("xTIBA5rboUvnH4htodjb6e697QjLERt1NAB4mZqp8Dg=", None, vec!["10.9.0.0/24".into()])?;
```

It checks the key is real base64 decoding to exactly 32 bytes. Junk is rejected loudly. The same
key twice is rejected. A bad key never silently lands in the list.

After a boot where the tunnel came up, `confirm` flips that peer to `established` and the
indicator goes green — you do not poll for it.

The peer light follows the tunnel, in both directions:

| the peer says | it means |
|---|---|
| `never` | you added the key, nothing has happened |
| `handshaking` | it said hello, nothing answered yet |
| `established` | it is up |
| `stale` | it *was* up, and the last handshake is over 3 minutes old — WireGuard has given up on those keys |
| `lost` | it was up, and the tunnel has since been proven dead |

`lost` and `never` are deliberately different words. "It broke" and "it never worked" send you
to different places.

**About `stale`:** WireGuard refuses to use a session's keys after 180 seconds without a
handshake, so we do too. But we can only *see* the age if the console contains `wg show`
output (`latest handshake: 6 minutes, 12 seconds ago`). A normal boot log does not have it. So
if you never capture `wg show`, staleness cannot be detected — we say so rather than inventing
a timer and pretending.

In Prod there is no panel, and `add_wg_peer` returns an error rather than pretending.

## Kubernetes: read this before you get excited

There is a `k8s-workload` socket. It composes a control plane, two workers, a CNI and a registry.

**It cannot currently start a cluster.** Light it and you get:

```
k8s-control-plane  not-implemented  no in-VM control-plane provisioner: unfold::talos::
                   MachineConfig renders the machineconfig and talos_apid pushes it, but
                   nothing boots the node
```

That is deliberate and it is written down rather than hidden. What *does* work today is the
checking half — given a real cluster's console output, the probes will honestly tell you whether
the nodes are ready and whether your workload rolled out. They are strict:

- a node listed as `Ready,SchedulingDisabled` is **not** ready (it is cordoned; it will run
  nothing new);
- a workload whose console says "successfully rolled out" while the replica count reads `1/3`
  **fails** — the count beats the claim;
- a console that says nothing about a rollout **fails**. Nobody saw it come up, so it did not.

## When you are done

```rust
let order = dep.teardown();
```

Consumes the deployment and throws away the keys you typed in. The bulb's boot is released first,
the network it rode on last.

## If something looks wrong

| Symptom | Cause |
|---|---|
| everything says `ready`, nothing says `serving` | you never called `confirm()` with a console |
| Prod says `all_ok == false` | correct, see above. Read `refusals()` |
| `qemu_netdev_arg()` gave you `None` | that socket has no shared network (a plain exe/VM) |
| a fixture went from `serving` back to `down` | the console showed it failing later. Believe it |
| a peer says `handshaking` forever | it is sending initiations and nothing is answering |
| a peer went `established` → `stale` | over 3 minutes since the last handshake. The session is dead |
| your test console "obviously works" but nothing goes green | you probably used an *attempt* line. See "Proof is stricter than you think" |
| `from_kind("...")` gave `None` | typo. It will not guess for you |