# holger.rs's control plane is cleartext on the public internet, and the estate
# says it is not
*MEASURED 2026-09-23 by the monetize lane, read-only, while giving monetize's
outward doors TLS. Nothing was changed on the box. The fix is the owner's to
route — this is the evidence and the recommendation.*
## What the estate says
```toml
# private-holger-ops/estate.toml:330
grpc = { bind = "0.0.0.0:50051", public = "grpc.holger.rs", via = "direct" }
# ↑ "gRPC on its own port and NOT through Caddy: holger-server terminates
# its own TLS there (mTLS is one of its authentication methods, and a
# proxy that terminated TLS would take the client certificate away)."
# estate.toml:308
"tcp/50051 any", # gRPC — holger-server's OWN TLS and its own auth …
# estate.toml:620
server_endpoint = "https://grpc.holger.rs:50051"
```
Three statements: the port is open to the world, holger-server terminates TLS
on it, and mTLS is one of its authentication methods.
## What the box does
**No TLS.** `openssl s_client -connect grpc.holger.rs:50051` reports
```
no peer certificate available
tls_get_more_records:packet length too long
```
and a rustls client gets `received corrupt message of type InvalidContentType`
— both are a TLS client meeting a **plaintext** server. A plaintext h2 gRPC
call to the same port succeeds and returns real data.
*(The earlier reading that this port had a valid certificate came from
`Verify return code: 0 (ok)`, which openssl prints even when **no certificate
was received**. That line fooling a reader is exactly how a box comes to be
believed encrypted when it is not.)*
**No mTLS.** The emitted appliance config carries one authentication method and
it is a static bearer:
```ron
# deploy/generated/server/holger.ron:31
auth: Some((methods: [StaticToken(subject: "console", token_file: …)],
require_auth_for_reads: false)),
```
So the stated reason for keeping gRPC out of Caddy — *a proxy would take the
client certificate away* — describes a client certificate that no client
presents.
**Reads are open to anyone who can reach the port, and the port is open to
everyone.** From an ordinary developer machine, with no credential:
```
tenants: bundles, crates, dev, docker, migtest, prod
crates {artifacts: 1, artifact_bytes: 777, archive_bytes: 802, archive_files: 2}
docker {artifacts: 1, artifact_bytes: 899}
```
**The write credential travels in that cleartext.** `require_auth_for_reads:
false` means the bearer is only sent when something is written — and when it
is, it is sent in the clear, over the public internet, to a port open to `any`.
Anyone on the path has the console's write credential.
## How the console reaches it today, which is the question that decides the fix
`holger-ui-server` is given `HOLGER_UI_HOLGER=https://grpc.holger.rs:50051`
(`deploy/generated/web/holger-ui.env:25`, emitted from `estate.toml:620`) and
it **works**: `https://holger.rs/api/public/releases` answers with the six
repositories, which can only have come from holger-server over gRPC.
It works because **the scheme is ignored**:
* `holger-ui/crates/holger-ui-server/Cargo.toml:30` — `tonic` with
`["codegen", "transport", "router"]` and **no TLS feature**;
* `holger-ui/crates/holger-ui-server/src/grpc.rs:214` — `connect_lazy` keeps
the `https://` prefix and never calls `tls_config`.
A tonic built without TLS dials the port in plaintext whatever the URI says. So
the console is not "configured with a different address", and nothing in front
terminates TLS: **the declaration is decoration, and the one component that
reads it cannot honour it.**
That also means monetize was the only thing on the estate that ever told the
truth about this address — it REFUSED `https://` by name rather than quietly
dialling plaintext, which is how the discrepancy surfaced at all.
## The trap in the obvious fix
Giving holger-server a certificate on 50051 makes the declaration true and
**takes the console down the same minute**: `holger-ui-server` cannot speak
TLS, so every call it makes would fail. Any sequence that starts with the box
is a sequence that starts with an outage.
## A third finding, one line: the gate reads a stale state
`private-holger-ops/terraform/terraform.tfstate` is gitignored, dated
**2026-09-17**, and holds ONE `front` server while `estate.toml` has declared
three machines since before `e73a5ff` — so three terraform-gate tests answer
questions about an estate that no longer exists, and have been red for days
without meaning what they appear to mean. (The true state is on loki.) A gate
that reads a stale state is worse than no gate: it fails for the wrong reason,
and the day it passes will be a coincidence.
## Recommendation, in this order
1. **Correct the declaration now, because it is false.** `estate.toml:620` →
`server_endpoint = "http://grpc.holger.rs:50051"`, and the comments at
`:308` and `:327-330` stop claiming TLS and mTLS that are not there. This
changes no behaviour — the traffic is already plaintext — and it stops the
file telling every future reader that the control plane is encrypted. A
declaration that is believed is worse than one that is absent.
2. **Close the port.** `estate.toml:308`: `tcp/50051 any` → the addresses that
actually dial it. The console is on the estate's own network; monetize
reaches it from the front. A cleartext control plane with open reads should
not be on the public internet at all, and this is the change that costs
nothing and removes the exposure today.
3. **Then make TLS real — and after the ruling above, this step is NOT what it
first said.** It is no longer "give holger-ui TLS, then a certificate on
holger-server's own 50051". It is:
1. **a terminator in front of the appliance's gRPC door** (Caddy already
terminates for `holger.rs` and `repo.holger.rs` on the same box), serving
`grpc.holger.rs:50051` over TLS and forwarding plaintext h2c to
holger-server;
2. **the console keeps dialling plaintext**, on the box it shares — it needs
no TLS, no `ring`, and no change beyond the one already made (`622e172`:
a `https://` endpoint is refused by name instead of dialled in the
clear, and the start line says which);
3. **then flip `estate.toml:620` back to `https://`**, because by then it is
true. monetize already speaks TLS and needs no further change;
4. and **re-read the mTLS condition above** before doing any of it.
4. **Rotate the console's bearer afterwards** — see the next section. It has
been sent in cleartext over a public port for as long as this has been
deployed.
## What is NOT recommended
Moving gRPC behind Caddy (`via = "caddy"`). It is tempting — Caddy already
terminates TLS for `holger.rs` and `repo.holger.rs` — and the estate's stated
objection (mTLS) does not currently apply. But it is the largest change of the
four, it breaks the day mTLS *is* configured, and steps 1–2 remove the
exposure without it.
## Item 3's code half is done, and it cost nothing — but TLS itself needs a ruling
holger-ui `622e172` (branch `lane-w1/console-tls`, **not deployed**): the
console now **refuses** a `https://` endpoint by name instead of dialling it in
the clear, states `tls = true|false` on its start line, and leaves `http://`
and bare addresses exactly as they were. The lock is unchanged — no new
package, no new edge — and 151 tests pass.
**The TLS feature itself is not there, and that is a measurement.** Adding
`tls = ["tonic/tls-ring", "tonic/tls-webpki-roots"]` — even OFF by default —
puts `tonic -> tokio-rustls -> rustls -> ring` into `Cargo.lock`, and
`holger-ui-server/tests/no_c_crypto.rs` walks the LOCK as a graph, feature-blind
by design. Measured: with the feature declared and off, the suite was 137 green
and `this_binarys_graph_carries_no_c_crypto` RED. The law is not decoration —
the passkey plane is ES256/P-256 over RustCrypto so that the one process that is
the way in links no C TLS stack.
So the order we established has a fork in its first step, and it is the owner's:
1. **the law bends** — widen `FORBIDDEN`, accept `ring` in the console;
2. **rustls gets a RustCrypto provider** — a new, early-stage dependency in a
security-critical place;
3. **TLS is terminated in front of the appliance** — the console keeps dialling
plaintext on the box it SHARES with holger-server, and external clients get
TLS. This costs no crypto in the console at all, and the measured topology
already points at it: all three names are one machine.
**RULED 2026-09-23: (3), terminate TLS in front of the appliance.** Taken by the
coordinating lane rather than parked, and flagged to the owner as theirs. The
reasoning, so the question is closed rather than left open:
* **The topology decided it.** All three names are one machine, so the
console's call to holger-server never crosses a wire. Plaintext between two
processes on one box is not the exposure; the public port is. A terminator in
front closes exactly what is open and nothing else.
* **(1) is the worst trade available.** `no_c_crypto` exists so the one process
that is the way in — the passkey plane — links no C TLS stack. Widening it to
reach a box on the same machine spends a real property to solve a problem
that is not there.
* **(2) puts an early-stage crypto provider in the security-critical
position**, to serve a hop that needs no crypto at all.
**⚠ THE CONDITION UNDER WHICH THIS RULING IS WRONG.** The estate's own stated
reason for keeping gRPC out of a proxy was mTLS — *a proxy that terminated TLS
would take the client certificate away*. That objection does not apply today
because no client presents a certificate (measured: `holger.ron` declares one
method, a static bearer). **The day mTLS is actually configured, (3) is wrong
and this must be revisited.** Whoever configures it inherits this decision.
## What monetize did about it, which is nothing to the box
* Both monetize builds now dial `https://` over rustls against the compiled-in
Mozilla roots, or a CA named by `[products.<id>] tls_ca`. Zero new packages.
* `guests/holger/tests/against_the_live_holger_rs.rs` holds the measurement:
it is `#[ignore]`d, it names this document's finding in its failure message,
and it **turns green by itself** the day holger-server answers TLS on 50051.
That is deliberate — a test that goes green on its own is worth more than a
note in a file nobody opens.
---
# Item 2, prepared and NOT applied: closing `tcp/50051`
*Prepared 2026-09-23. The apply is the owner's to route. Everything below is
evidence or a named gap in the evidence; nothing here is reasoning dressed as a
fact.*
## What the evidence says, with no credential
1. **The three public names are ONE machine.**
`holger.rs`, `repo.holger.rs` and `grpc.holger.rs` all resolve to
**95.111.208.165**.
2. **The three-machine estate is declared, not applied.**
`private-holger-ops/terraform/terraform.tfstate` (local copy, 2026-09-17,
serial 4) holds exactly `upcloud_server.front`, `upcloud_storage.front_data`
and `upcloud_firewall_rules.front` — one server — while `estate.toml`
declares `web` · `server` · `twin`. The two terraform-gate tests that assert
three machines are red for this reason and were red before this lane
touched anything.
3. **So the console and holger-server are on the SAME box.** The BFF dials
`grpc.holger.rs:50051`, which is that box's own public address; the kernel
delivers it locally. **No firewall rule is needed for the console at all**,
and `estate.rs` would refuse one anyway — a machine-scoped rule open to
itself is *"a rule with nothing on the other end of it"*
(`xtask/src/estate.rs:676`).
4. **`admin` is currently the whole internet.**
`terraform/estate.auto.tfvars.json:2-3` — `admin_ipv4_start = "0.0.0.0"`,
`admin_ipv4_end = "255.255.255.255"`. **Changing `any` to `admin` would
close nothing** until those two values are narrowed. This is the part that
would look like a fix and not be one.
## What could NOT be established, and why
**Who has actually dialled 50051 over the retained window.** That is
`AuditService.Query`, and it is gated by `authorize_admin_read`
(`holger/server/lib/src/grpc/services.rs:2599`) — the same gate that refused
`Admin.Capacity` with *"[holger-auth-1] valid credentials required to read the
audit log"*. The credential is the static bearer at
`/var/lib/holger/admin-token`, on a machine declared `shell = false`.
**I did not take that credential and did not run that query.** The owner (or
anyone holding the console's token) completes the evidence with a read-only
call — `AuditService.Query` over the same plaintext port, filtered to the
retained window, reading `source_ip` off each record. Two caveats that must
travel with the result, or it will be read as more than it is:
* **Reads are unaudited** on this deployment (`require_auth_for_reads: false`,
and the flow door is documented as unaudited because it serves aggregates).
An audit-log enumeration therefore shows who WROTE, not who dialled. The
`cargo`/`pip`/`docker` pulls do not appear at all — they arrive on :443
through Caddy, not here.
* A window that has rolled over shows nothing for a client that has been quiet,
and "quiet" is not "gone".
## The allow-list as it stands, and what breaks if an entry is missing
`estate.toml:323`, today:
```toml
"tcp/50051 any", # gRPC — the product's public door …
```
| entry | why | what breaks without it |
|---|---|---|
| **(nothing for the console)** | same box; local delivery, never through the firewall | nothing — and `estate.rs:676` refuses a self-scoped rule |
| `tcp/50051 admin` **with the range narrowed** | the operator's `holger` CLI and `holger agent` from a known address | every CLI call and every agent run from outside that range; `:22` moves with it, since both read the same two variables |
| `tcp/50051 <monetize's address>` | monetize's poller, once it is pointed at holger | metering goes dark: `list_tenants` and every reading become `Unavailable`, the console's reachability dot goes red, and no invoice is produced. **monetize is not pointed at holger today** — there is no `[products.holger]` block on the deployed front — so this entry is for the day it is |
| `tcp/50051 server` (twin → primary) | already declared at `estate.toml:526` for the twin's follow traffic | the twin cannot follow — but no twin is deployed today (see (2)) |
**The change that actually closes the door is two lines, not one:**
```
estate.toml:323 "tcp/50051 any" → "tcp/50051 admin"
terraform/estate.auto.tfvars.json:2-3 admin_ipv4_start / admin_ipv4_end
0.0.0.0–255.255.255.255 → the operator's range
```
and `estate.auto.tfvars.json` is emitted, so the narrowed range is written in
`estate.toml` and re-emitted rather than edited in place.
## The risk, stated plainly — and then removed
Narrowing `admin` narrowed **`tcp/22`** on the same two variables
(`estate.toml:581`): getting the range wrong would close the one shell in the
estate at the same moment it closed the gRPC door, on a box with no console
access of its own.
**That coupling is gone** (private-holger-ops `3a81909`). The gRPC door has its
own declared range, `product_ipv4`, set to exactly what it had before — the
whole internet — so nothing changed on the wire. The rule was a LITERAL
whole-internet range in `main.tf` until then, which is why it had never been
narrowed: the only ways to close it were editing HCL by hand or borrowing the
admin range, and the second would have taken SSH down with it.
So the decision is now one line and it touches nothing else:
```toml
# estate.toml, [estate]
product_ipv4 = ["<first>", "<last>"] # then: cargo xtask estate emit, and apply
```
SSH stays where it is, on `admin_ipv4`, as a separate decision to take
separately or not at all. The gate tells the two ranges apart and refuses
:50051 scoped to the admin one, with the reason in the failure message — before
that, every `var.` reference read as `admin` and the mistake would have passed
silently.
---
# Item 4, specified and NOT performed: rotating the console's bearer
*Specified 2026-09-23. Nothing was rotated, nothing was read off the box, and
no token appears anywhere in this document.*
## What rotates
**ONE secret in TWO files**, and that is the whole difficulty:
| where | owner | what reads it |
|---|---|---|
| `/var/lib/holger/admin-token` | `holger:holger`, 0600 | holger-server's only declared auth method — `StaticToken(subject: "console")`, `deploy/generated/server/holger.ron:31` |
| `/opt/holger-ui/holger-token` | `holger-ui:holger-ui`, 0600 | the console; `HOLGER_UI_TOKEN` names the PATH, never the bytes |
They are two files because two service users read them; they are **one secret**
because holger authenticates one subject (`private-holger-ops/xtask/src/front.rs`
steps 8 and 16 place the same 32 bytes in both). Neither travels through git.
**What does NOT rotate with it**, and must not be confused with it:
`HOLGER_UI_ADMIN_TOKEN` — the break-glass bearer that enrols the FIRST passkey
and retires once one exists. It lives in `~/holger_secrets` and is appended to
`/etc/holger-ui.env`. Rotating the wrong one takes out the console's login
instead of its credential.
**Who else holds a copy:** nobody, as far as this repository declares —
`[products.holger] bearer_token` is NOT configured on the deployed monetize, so
monetize does not hold it today. Any operator who was handed it for a `holger`
CLI call does, and that is not written down anywhere; the rotation is the moment
to find out.
## What must be updated in the same minute
**The console reads its file ONCE, at start** (`main.rs::load_credential`, called
from `run()`), and holds it in `BearerAuth` for the life of the process. So
writing the new bytes into `/opt/holger-ui/holger-token` does **nothing** until
`holger-ui-server` restarts.
**The appliance re-reads its file on EVERY authenticated request.** Settled from
the code, read-only, with no call against the box and no token — the open
question in the first draft of this section is closed, and the answer is that
**holger-server does NOT need restarting**:
| hop | where |
|---|---|
| the gate calls the dispatcher per request | `holger/server/lib/src/grpc.rs:1539` — `auth::validate_request(&self.auth_config, bearer, cn).await`, inside `authorize_admin_identity` |
| the dispatcher builds the provider set INSIDE itself | `edda/crates/nornir-auth-policy/src/lib.rs:830` — `let providers = build_providers(config);` |
| the static-token provider RESOLVES its secret | same file `:738` — `BearerProvider::resolved(subject, token, env_var, file, …)` |
| resolving reads the file | `edda/crates/nornir-service/src/lib.rs:675` — `std::fs::read_to_string(path)` |
So the sequence is shorter than the cautious one, and this is the version to
follow at three in the morning:
1. write the new bytes to `/opt/holger-ui/holger-token` — **inert**, nothing
reads it until the console restarts;
2. write the same bytes to `/var/lib/holger/admin-token` — **live on the next
request**;
3. `systemctl restart holger-ui-server` — immediately after (2), ideally on the
same command line;
4. verify: the console's start line says `credentialled=true` and `tls=false`,
then perform ONE write through the console.
**The window is (2) to (3) and nothing else** — as long as a service restart,
not as long as a deploy.
**⚠ AND THE WRITE IN (2) MUST BE ATOMIC.** Because the file is read per request,
a request landing while it is half-written reads a partial or empty secret and
is refused — and an empty one is the fail-closed case the appliance's own
`refuse_disabled_static_token` describes. Write to a temporary file in the SAME
directory and `rename(2)` it over; `chmod 600`, `chown holger:holger` on the
temporary file before the rename, never after.
## What breaks if the two are not simultaneous
**The failure is quiet on the half that matters, which is why it needs saying.**
`require_auth_for_reads: false` — so with a mismatched bearer:
* **every READ keeps working.** The catalog, the artifact lists, the front page,
the whole console *looks healthy*.
* **every WRITE comes back `Unauthenticated`** — and so does every migration,
promotion and replication RPC. The console's own logs show a credential being
sent, which is exactly the shape that sends somebody looking at the network
instead of at the token. The emitted env file warns about this state in those
words; this is how it is entered.
* **`Admin.Capacity` fails too** (it is gated like the audit plane), so the day
monetize is pointed at holger its `servable` goes `Unmeasured` and no invoice
can be drawn from a capacity nobody could read.
**The zero-window alternative** — now much less likely to be needed, since the
window is one service restart rather than two, but recorded because it is the
right answer if even that is unacceptable:
`AuthConfig.methods` is a LIST. Declare the old and the new as two
`StaticToken` methods, place the RON, restart, cut the console over, then remove
the old method and restart again. It costs two deploys instead of one and holds
both secrets valid in between — which is also its risk, and why it is written
here as the alternative rather than the plan.
## What this does not fix
Rotating replaces a secret that has been sent in the clear over a public port
with another secret that will be sent in the clear over the same port. **It is
worth doing only after, or with, the port being closed and TLS terminated in
front** — otherwise the new bearer joins the old one on the wire the same day.
That ordering is the reason this is item 4 and not item 1.