> **RULE: Never add Co-Authored-By or any AI attribution to commits. Ever.**
# holger
Secure artifact server for airgap environments backed by znippy archives.
Repository: https://git.gunnar.rs/nordisk/holger
---
## Workspace layout
```
traits/ shared types, ZnippyPlugin discriminants (→ znippy-common)
server/lib/ gRPC service, wiring engine, auth, storage
server/cli/ holger server binary
server/repository/ one crate per package ecosystem:
znippy-{maven,python,rust,npm,nuget,go,gem,deb,rpm,helm,docker,conda,composer,package}
⚠ THE CRATE NAME IS OFTEN NOT THE NAME YOU SEARCHED FOR. A capability carries up
to four names and the docs' name is usually not the directory's:
PyPI (docs/UI) = "pip" (ron_repo_type) = ArtifactFormat::Pip = znippy-python/ (crate)
Docker (enum) = "oci" (alias) also file-oci/
Znippy (enum) = "znippy" but the crate is znippy-package/
Rust znippy-rust/ is package holger-rust-znippy-repository (words swapped)
`grep -ri pypi server/` finds NOTHING in the implementation. Use
`ArtifactFormat::home()` (traits/src/lib.rs) — the format→crate map, kept honest
by `every_format_names_a_crate_that_exists`.
agent/lib/ airgap / export / push logic
agent/cli/ holger agent binary
agent/connector/nexus/ Nexus connector (basic auth)
agent/connector/world/ external registry connector (crates.io, PyPI, …)
agent/connector/artifactory/ Artifactory connector
mannequin/ mock server: PKI + OIDC + Nexus + auth for dev/test (GUI behind `gui`)
plugin/ wasm package-handler + meter lane: abi, host, handlers, guests
holger-front/ holger.rs public front page (mounted by server/lib)
holger-estate-front/ the estate front (own workspace)
estate/ THE ESTATE WORKSPACE (H1): holger-fleet (the ONE monetize pull channel,
server-lib `metered`), holger-live; members outside it by `package.workspace`:
xtask, the build declaration, crates/holger-monetize-guest, estate-tests
crates/ holger-monetize-guest (an estate/ member)
holger-recipe/ holger-build-thing/ build-space/ holger-build/ the build declaration (estate/ members)
update-bundle/ errcode/ signed update bundles; the frozen error-code registry
provenance/ the offline provenance verifier (DSSE/in-toto/SLSA, Sigstore, PEP 740, npm, cosign)
bench-scenarios/ bench arms (dev-dependency only)
distribution/ Dockerfile
xtask/ cargo xtask verbs (test, gate, build, bench, gen-proto, …; `cargo xtask` lists them)
```
---
## Design
### Vision
Holger replaces Nexus/Artifactory in environments where security, auditability, and airgap compliance are primary requirements:
- **Airgap transfer** — move packages across network boundaries as immutable znippy archives
- **Strong authentication** — OIDC and mTLS, not just username/password
- **Immutable storage** — append-only, content-addressed, auditable
- **Multi-format** — Rust, Python, Java, Go, npm, NuGet, Gem, Deb, Helm, Docker from one server
### Security model
**Authentication:**
```
Client ──TLS──▶ holger-server
Option A: mTLS cert → extract CN as identity (CA-rooted trust)
Option B: Bearer token → validate via OIDC userinfo endpoint
```
**Authorization** (prototype → production):
| Role | Permissions |
|---|---|
| `denied` | nothing — not even reads (the per-repo carve-out) |
| `reader` | download, list |
| `writer` | upload, create repos |
| `admin` | promote, delete, manage config |
**Separation of duties** (the intake chain, `server/lib/src/promote/duties.rs`): the same
`auth` block grants `duties` / `group_duties` — `Developer` (reads AVAILABLE only),
`Importer` (ingests), `SecurityApprover` (approves, quarantines, releases, blocks),
`RepositoryAdmin` (configures, never approves), `Auditor` (reads audit + evidence, changes
nothing). The importer never approves its own import; `promotion.intake.approvals: N` needs N
distinct approvers; each approval is an attested statement in the approval evidence slot.
Wire verb: gRPC `PromotionService.AdvanceArtifact` / `GetArtifactState`, HTTP
`POST /-/intake/advance`, `GET /-/intake/state`, CLI `holger-server intake`.
**Stored evidence is read back** (W5): S9's `PromotionService.GetArtifactEvidence` (by
coordinate; HTTP `GET /-/intake/evidence?sha256=` and `GET /-/intake/sbom?sha256=` resolve the
coordinate holding the bytes and call the same `duties::evidence_as`; CLI `holger-server intake
evidence --name`) also returns every stored record, its DSSE envelope, the signer certificate,
the stale reason and the SBOM link, under the read duty of GetArtifactState.
SCANNED and the `osv` gate reuse a FRESH stored record (same scanner ids and versions, intel
sequence not newer) and audit it as a `scanned` step with `reused=true`; missing or stale ⇒
scan again. **One quarantine** (W6): `HOLGER_QUARANTINE_PROPERTY` is retired and refuses the
boot (`holger-intake-14`); the hold is the intake QUARANTINED state.
**npm through the chain** (lane P5, `server/lib/src/promote/npm.rs`): `promotion.intake.npm`
names an archive-less `ron_repo_type: "npm"` repository (the estate's `npm`). `npm publish` there
is INGESTED into `incoming` at the tarball's registry path (`<package>/-/<file>.tgz`, the
coordinate the walk names); the packument (each manifest read from the SEALED tarball, sha512
integrity), the tarballs and `dist-tags` are a view of the AVAILABLE set; unpublish and rewrites
are refused by policy (`holger-npm-1..5`).
**Pull-through into the chain** (`server/lib/src/pull_intake`): `python`, `oci`, `npm`, `maven`,
`crates` and `apk` (with `url`) sources fetch on a client's request (one jera `PullIntake` job per want),
INGEST as `pull-through:<source>`, walk SCANNED (scanners + `holger-provenance`) and POLICY_CHECKED,
wait for approval (or `auto_approve`), and are served from the sealed store once AVAILABLE; until then
each client gets its ecosystem's "not yet" (503 + Retry-After, then 404 / 403), and every index
(PEP 503, tag list, packument, `maven-metadata.xml`, sparse index file, APKINDEX) lists only AVAILABLE items.
Paths: `/production/<source>/…` (OCI: `/v2/production/<source>/…`).
**Provenance of what comes in** (`provenance/`, `server/lib/src/scanner/provenance.rs`, lane G3a):
the `provenance` scanner slot (id `holger-provenance`) verifies, OFFLINE, the attestations that
came with an artifact (`<data_root>/holger-provenance/<sha256>/`): DSSE + in-toto bound to the
sha256/sha512, SLSA v1/v0.2 (builder, build type, invocation, resolved dependencies into the
evidence), keys (Ed25519, ECDSA P-256) and keyless Sigstore (Fulcio chain to a pinned root,
issuer + subject identity, SCT, Rekor SET / inclusion proof against a signed checkpoint) — for
Sigstore bundles, PyPI PEP 740, npm attestations and cosign `.sig`/`.att` layers (Chainguard's
images). Policy: `promotion.intake.scan.provenance: (rules: <floor>, sources: {name: require(...)
| prefer | off})`; a source only tightens the floor. The trust root is a signed
`holger.sigstore-trust/v1` bundle (`rules.trust_root`, verified with `promotion.intake.trust_policy`).
An airgap export carries the material; the import verifies it again under the secure side's policy
(gate `airgap`). Codes `holger-provenance-1..13`.
**APK provenance** (lane G3, `provenance/src/apk.rs`): Wolfi and Chainguard packages are signed
only through `APKINDEX.tar.gz`. A package verifies when the index's `.SIGN.RSA256.<key>` (or
`.SIGN.RSA.`, SHA-1) verifies over the raw index stream with a PINNED apk key (before the index
is decompressed), the index entry for its `pkgname`/`pkgver` names its control segment's hash
(`C:Q1`/`Q2`) and size, and `.PKGINFO`'s `datahash` is the data segment's sha256 (hashed, never
unpacked). It is one provenance candidate (`apk-index:<file>`, builder `apk-key:<name>`, no SLSA
level, so `slsa_min > 0` refuses it). The keys ride in the `holger.sigstore-trust/v1` bundle as
`apk-keys/<name>` payloads (`TrustBundleOptions.apk_keys`, `TrustRoot.apk_keys`) and reach the
scan worker in the job (`ProvenanceJob.apk_keys`). `sources: [(name: "wolfi", kind: apk)]` is an
upload source whose artifacts also get `<inbox>/apk/<source>/*` as material (exported with the
airgap bundle); `require(apk_keys: [...])` narrows the keys.
**APK pull-through** (lane T8, `server/lib/src/pull_intake/apk.rs`): a `kind: apk` source WITH `url`
(e.g. `https://packages.wolfi.dev/os`) pulls `<arch>/-/pull/<name>` (the newest version and its
dependency closure) or one `<arch>/<file>.apk`. The upstream `APKINDEX.tar.gz` is verified against a
pinned key (`pull_through.rules.apk_keys` or the trust bundle's `apk-keys/`) before a package is read
from it; each `.apk` is held to that signed index before it is INGESTED. Holger serves its OWN
`APKINDEX.tar.gz`, re-signed with Holger's per-source key (`/production/<source>/-/apk-key-name`, then
`/production/<source>/<key>` into the client's `/etc/apk/keys`), listing only AVAILABLE packages with the
upstream's stanzas. Tests: `server/lib/tests/pull_through_apk.rs` (suite-check `trust:apk-pull`) and its
ignored `real_client_apk_add_through_holger`, a real `apk add jq` in `cgr.dev/chainguard/wolfi-base`
against the real Wolfi repository (the demo's row, `cargo xtask demo tests`; the client reaches Holger
through pasta `-T <port>`, never `--network host`).
**Maven PGP provenance** (lane T9, `provenance/src/pgp.rs` + `pgp/rpgp.rs`, rpgp `pgp =0.21.0`, pure Rust, no
default features): Maven Central's detached `.asc` beside a file verifies OFFLINE when its issuer
fingerprint (or key id) names a PINNED OpenPGP key's primary or signing subkey and the signature
(binary or text, SHA-224 or stronger) verifies over the exact bytes. An unpinned signer is
"unverified" (`holger-provenance-4`), never a pass. It is one provenance candidate
(`pgp-signature:<file>`, builder `pgp-key:<name>`, no SLSA level). The keys ride in the
`holger.sigstore-trust/v1` bundle as `pgp-keys/<name>` payloads (`TrustBundleOptions.pgp_keys`,
`TrustRoot.pgp_keys`, the job's `ProvenanceJob.pgp_keys`) or in the floor's `keys` as `alg: openpgp`;
`require(pgp_keys: [...])` narrows them per source (intake `sources` and `pull_through` sources).
The Maven pull-through fetches `<file>.asc` beside every POM and JAR.
It is OPT-IN (T9b): holger-provenance `pgp`, server-lib and CLI `maven-pgp`, carried by the CLI's
`online` and `full`; the default `holger` does not have it (rpgp is about 96 crates; the default tree
is bounded at 345). server-lib's own `online` does not carry it (a weak feature would put rpgp in
every nested lock, cargo#10801): test the Maven `.asc` with `--features jobs,online,supply-chain,maven-pgp`. Without it every pin FAILS CLOSED by
name, `[provenance-pgp-not-built]`: a trust bundle's `pgp-keys/<name>` and a floor key of
`alg: openpgp` are refused at load, so is a source's `require(pgp_keys: [...])`, and an `.asc` is
unverified (`holger-provenance-4`). Tested both ways: `provenance/tests/pgp_build_feature.rs`,
suite-check `trust:maven-pgp`, `trust:maven-pgp-intake`, `trust:pgp-not-built`, `trust:pgp-intake-not-built`.
**Secrets in what comes in** (`server/lib/src/scanner/secrets.rs`, lane HOLGER-G3): the
`secrets` slot (id `holger-secrets`) runs in the scan worker and walks the artifact with the
INGEST walk's own limits (`holger_inspect::Inspector::visit`: zip, tar, gzip, nested archives,
every image layer, deleted files included) against a rule set that is DATA
(`holger.secret-rules/v1` RON; built-in `scanner/secrets-rules.ron`, or the operator's file at
`promotion.intake.secrets.rules` / `steps.secrets.rules`, read and checked at boot and handed to
the worker in the job): private keys, AWS/GCP/Azure credentials, GitHub/GitLab/Slack/Stripe/npm/
PyPI/crates.io tokens, `.env` files, kubeconfigs, high-entropy strings with context. A finding is
`secret:<rule id>` with the path and line, never the value. The scanner warns; the policy decides:
`rules.secrets: block | warn | off` (default `block`) and a source's `gates.secrets` (tighten
only); `block` refuses high and critical findings at SCANNED by name.
**Every scan runs in a container** (lane SANDBOX-SCAN, owner 2026-10-23; `server/sandbox`,
`server/scan-worker`, `server/lib/src/sandbox`): the server never unpacks, parses or scans an
artifact itself. The INGEST walk (bombs, nesting, traversal, MIME, executables), YARA, the model,
licence and wheel-SBOM scanners, the `osv` gate's licence read, an image's package catalog (apk, dpkg,
rpm sqlite, python, npm, Go build info, cargo-auditable; `server/inspect/src/image.rs`) and
`holger_provenance::verify_artifact` (DSSE, Sigstore, DER/X.509, Rekor) run in `holger-scan-worker`,
one hardened rootless podman container per artifact (`--network none`, the artifact read-only, a
capped noexec tmpfs, a read-only root, `--cap-drop all`, no-new-privileges, the default seccomp,
uid 65532, CPU/memory/pids limits, a wall-clock deadline). The worker answers one JSON report on
stdout, capped and validated field by field; the server keeps the advisory match (an image's CVEs,
each with its severity, so `cve_max` holds), the evidence and the signatures. ClamAV is clamd in its
own sandboxed container (`scripts/clamd-sandbox.sh`); the server streams bytes to its socket.
`promotion.intake.rules: (sandbox: required)` is the default; `off` only in a test build
(`holger-sandbox-6`); no engine refuses every scan (`holger-sandbox-1`), never an in-process
fallback. The image is minted FROM scratch (`cargo xtask scan-worker image`) and crosses the airgap
as an OCI archive with its sha256 (`scan-worker save`); the secure side never pulls. Engine: the
`podman` program, or podman's API socket (`sandbox: (url: Some("unix://…"))`) for a Holger that
runs in a container. Codes `holger-sandbox-1..12`; container arms `cargo xtask scan-worker tests`.
**On the box** (lane HOLGER-MEDIA): the main and twin media carry the sandbox in the app archive
(`holger-recipe` `sandbox_payload_entries`: `sandbox/{manifest.toml,crun,holger-scan-worker.rootfs.tar,
clamav.rootfs.tar}`, staged by `cargo xtask scan-worker stage` into `.sandbox-out/`, packed by the
forge beside the app, so the installer, the keep medium and the live app all carry it). The OS-less
appliance has no podman: the engine there is the static `crun` (1.9 MB, sha256-pinned) with Holger
writing the OCI bundle (`holger_sandbox::crun`: a user namespace, uid 65532 mapped to host 100000+,
a new empty network namespace, the same limits), over the worker rootfs laid under
`/data/holger/sandbox` and checked by sha256 before the server serves (server-cli `sandbox_boot`,
server-lib `sandbox::media`); clamd runs the same way (`sandbox::clamd`, uid 100, 2 GiB, supervised).
The estate declaration says `sandbox: Some((engine: crun, …))` and `clamav: Some((…required: true))`;
`GET /-/sandbox/probe` (admin-read) is the worker's probe from inside, the install test's `sandbox` row.
The kernel rows every holger medium adds for it (`container_kernel_requires`): MULTIUSER (the
appliance-boot contract's one-user default, lifted by name), the namespaces, cgroup v2 with memory,
pids and cpu, seccomp.
**The airgap** (`server/lib/src/promote/airgap.rs`, `docs/demo-airgap.md`): `holger-server
export` takes the AVAILABLE set of the intake chain's production repository with its stored
evidence, signed attestations, SBOMs and intake records as a bundle candidate whose verdict is
COMPUTED from that evidence (an artifact without passing evidence is refused by name);
`holger-bundle approve`/`finalize` sign it; `holger-server bundle import` on the secure side
verifies every payload hash and evidence attestation and lands each artifact AVAILABLE, audit
first, served through a PEP 503 index (`/production/simple/`). `network: Offline`
(`server/lib/src/egress.rs`) refuses a config that needs the network and every outbound dial.
`scripts/demo-airgap.sh` runs the whole 1.0 demo.
**The sync protocol** (Holger v2.0 H4, `sync/` + `server/lib/src/sync.rs`, feature `sync`,
default-off; `docs/sync.md`): every sync from an upstream Holger (a pull, a stick, or the
`evidence/holger-sync.json` an export carries) is a signed manifest (Ed25519; generation,
packages added, every invalidation: coordinate, sha256, `cve-after|malware|pulled|policy`)
verified under the `sync:` block's trust list, anti-rollback. The downstream holds each
invalidated package at once through `IntakeChain::invalidate` (QUARANTINED from any state,
bytes kept, audited first with the `invalidated` trail step, never released, every index drops
it) and signs an ack (outbox + `--ack-out` stick; pull: `POST /-/sync/ack`). The upstream's
ledger keeps per downstream the generation applied, the lag and the invalidations reached;
`GET /-/sync/board` is the board's data; an invalidation not acked within `ack_deadline_secs`
raises one ALARM (audit, board flag, the `holger::alarm` log line), cleared by the ack. CLI
`holger-server sync key|invalidate|manifest|apply|ack|board|pull`; the `sync.tick` jera job.
Codes `holger-sync-1..10`.
**The box's import doors** (`server/lib/src/import_door.rs`, lane W4): a box has no shell, so
intel and update/airgap bundles also arrive over gRPC `AdminService.ImportIntelBundle` /
`ImportBundle` (client streams), HTTP `POST /-/intel/import` / `POST /-/bundle/import`, and the
`<data_root>/intel.inbox` / `bundle.inbox` drop directories (swept by the `holger_import_inbox`
jera job, then `archive/` or `refused/` + `.why.txt`; a `FIRST-IMPORT` marker sets an empty
floor). One body of code with the CLI verbs: verify, floor, audit first, fail closed; Repository
Admin or Security Approver only. Bundles land AVAILABLE through `IntakeChain::land_imported`
(trust policy `promotion.intake.trust_policy`); the old `bundle import --repo` raw-directory
write is refused by name (`holger-import-1`). An online box (`intel.offline: false`, the
sold estate's default) runs the `intel.refresh` jera job once at boot and daily: it fetches OSV,
NVD and KEV (or `intel.feed_mirror`), signs the set with the box's own refresh signer and lands it
through the same import (`ImportDoor::refresh_intel`); server-lib `intel-refresh`, named only by
the two metered MVP rows (server-cli `intel-fetch`); the airgap tiers keep zero egress. A scan never dials; it reads the local database.
Reads are **open by default** (airgap-mirror posture: the network is the perimeter, and
unmodified `pip`/`cargo`/`docker` carry no credential). Set `require_auth_for_reads: true`
in the RON `auth` block to gate them behind `reader` — opt-in, fail-closed, health probes
stay open. See `.nornir/auth-rbac-design.md` §R3.
**Repository tokens** (`server/lib/src/repo_token.rs`, lane P3): what cargo, pip, npm and
maven carry instead of a passkey. A person signed in with a passkey (stepped up) mints one over
gRPC `holger.token.v1.TokenService` (console page `/tokens`, CLI `holger-server token
mint|list|revoke`) for named repositories, read or read-write, 1 to 366 days. The secret
(`holger_rt_…`) is shown once; `holger-passkeys/repository-tokens.json` keeps `blake3` hashes,
the last use, and revoked rows. Only the repository door takes a token
(`auth::validate_request_for`; every other door refuses it): it acts as its owner as they stand
now, narrowed to its repositories and scope. Refusals `holger-token-1..11` (revoked, expired,
owner gone, out of scope, read only, …); an unknown token is the plain 401. A gated rust
repository says `"auth-required": true` in `config.json`, so cargo sends the token on reads.
Roles mapped from OIDC token claims or mTLS CN → role config.
**Transport**: gRPC over HTTP/2 (tonic). No HTTP/REST API surface.
### Server architecture
Config is a **directed graph** of three node types:
```
ExposedEndpoint ←── Repository ──→ StorageEndpoint
(gRPC) │ (disk / znippy archive)
▼
Repository (upstream proxy)
```
The wiring engine (`wire_holger`) resolves in multiple passes: index nodes → wire forward refs (repo→storage, repo→exposed) → wire reverse links (exposed→repos) → attach FastRoutes.
**Request flow (gRPC):**
```
gRPC request (tonic)
→ HolgerGrpc service
→ FastRoutes.lookup(repo_name)
→ RepositoryBackendTrait.fetch() / .put()
├── RustRepoFile (disk)
├── RustRepoZnippy (znippy archive)
└── MavenRepoZnippy (znippy archive)
```
**Direct Rust bindings** (no serialization for in-process callers):
```rust
holger.fetch("repo-name", &artifact_id) // returns Option<Vec<u8>>
```
### Agent architecture
The agent moves artifacts between systems:
```
Source ──▶ [Agent Logic] ──▶ Target
Sources: crates.io, Nexus, Holger, .znippy archive, directory
Targets: .znippy archive, directory, Nexus, Holger
```
All sources/targets implement `ConnectorTrait`:
```rust
async fn list_repositories() -> Result<Vec<RemoteRepository>>
async fn list_assets(repo) -> Result<Vec<RemoteAsset>>
async fn download_asset(asset) -> Result<Vec<u8>>
async fn upload_asset(repo, path, data) -> Result<()>
```
Auth per connector:
```rust
enum ConnectorAuth { None, Basic { username, password }, OidcToken { token }, Mtls }
```
### Storage
- **znippy archives** — immutable, content-addressed, blake3 checksums. One archive per repository version. Carriable across airgap on USB/optical.
- **Filesystem backend** — simple `{repo}/{path}` tree, used for dev/testing.
- **Promote workflow** — DEV (mutable, live ingest) → `promote` → PROD (immutable). The seal is content-addressed (blake3) into `<data_root>/holger-prod-store`; the ref IS the seal and is written `create_new`, so a sealed coordinate can never be overwritten. Signing 🟡.
### Mannequin (test infrastructure)
`mannequin/` mocks the entire external world for dev and integration tests:
```
holger-mannequin
├── PKI — CA mgmt, cert issuance, regen
├── OIDC — /token, /userinfo, discovery endpoint
├── Nexus — proxy + upload (replaces real Nexus in bench)
└── Auth — users, roles, validate, issue
│
└── egui GUI (--gui flag) — live system map + request logs
```
Used in-process (integration tests, no network) or as a standalone server (`mannequin serve`).
Also used for Nexus benchmark comparison — `mannequin` impersonates Nexus so holger and Nexus run against the same mock backend.
### What's production-ready vs prototype
| Area | State |
|---|---|
| gRPC serve + fetch | production-ready |
| Modern auth (OIDC + Bearer + mTLS) | PLUGGABLE feature-gated providers (task #57): each method is its own sub-crate impl'ing the shared `nornir_service::AuthProvider` seam (edda). `nornir-auth-oidc` (JWT-vs-JWKS offline / `/userinfo` online / device-code CLI login — Entra ID·Okta·Auth0·Google·Keycloak·Zitadel), `nornir-auth-bearer` (static token / API-key, reuses the shared `resolve_token`+constant-time compare). mTLS-CN stays in-tree (transport-level). `validate_request` dispatches a bearer to the enabled providers; build-thing pins the set per tier (embedded/mini = bearer-only tiny; `full` = +oidc). Wired into both gRPC + HTTP write paths. See `.nornir/auth-providers.md` |
| Authorization / ACLs | W2 RBAC write-gate wired (gRPC + HTTP), fail-closed; off until `roles`/`default_role` configured. R3 READ gate shipped: `require_auth_for_reads` gates every read door behind the `reader` floor (repo-scoped, `Denied` role for per-repo carve-outs), opt-in so existing deployments are unchanged; gated reads audit the real subject instead of `anonymous` |
| Maven, Python, Rust repos | working |
| npm, NuGet, Go, Helm, Docker, Composer, Gem, Conda | real protocol (Gem = real RubyGems compact-index, Conda = real `repodata.json`) |
| Deb, RPM | native index synthesized from the packages present (Deb `Packages`+`Release`, RPM `repomd.xml`+`primary.xml`, real SHA256); control fields parsed from the package bytes — Deb via `ar`+`control.tar.{,gz,xz,zst}`, RPM via the binary header tag table (epoch + `<format>`) — with filename-derivation only as fallback; passthrough wins when the archive carries upstream metadata |
| Generic / Raw repo | writable path-addressed blob bucket (`ron_repo_type: "raw"`) — disk-backed loose-file store, traversal-closed, SHA-256-verified delete; HTTP `PUT/GET/HEAD/DELETE` + reuses search / retention / replication / metrics unchanged (`raw.rs::RawBackend`) |
| AI models | GGUF, Safetensors and ONNX in a `raw` repository, told by content (not extension) by the intake's `model` scanner slot (`scanner/model.rs`): header checks (length limits, offsets inside the file, no overlapping tensors) block a malformed file; architecture, parameters, tensors, dtype and quantization go into the evidence and the `model.*` properties (property search). The pickle family (pkl/pickle/pt/pth/joblib, PyTorch zips holding `data.pkl`) is blocked by default; `models: (allow_pickle: ["repo"])` makes it a warning in the named repositories only. Codes `holger-model-1..8` |
| Maven metadata | passthrough / partial |
| Promote workflow | DEV→PROD seal shipped on BOTH surfaces: `PromotionService.PromoteArtifact` gRPC (ADMIN gate scoped to the destination repo, `promoted_by` from the authenticated identity) + `holger-server promote --from --to` LOCAL CLI (in-process like `replicate`; gated by filesystem permissions only, `promoted_by` = the OS user — see `.nornir/promotion-design.md` §3.1). Content-addressed, immutable (re-promote refused), dry-run by default, same fail-closed policy + property gates on both paths. Bundles/staging-list/history 🟡 |
| Audit log | append-only, **hash-chained** JSON-lines sink (`FileAuditLog`: `seq`/`prev`/`hash` per line, a checkpoint every `audit.checkpoint_every` records signed by the attestation signer) behind the `audit::AuditLog` trait. The **per-artifact trail** by sha256 (imported, hash_verified, scanned per scanner, policy_passed/denied, approved per approver, sealed, exported, airgap_imported) is a view of the one log: gRPC `AuditService.GetArtifactTrail`/`VerifyAuditChain`, HTTP `GET /-/audit/trail?sha256=`/`/-/audit/verify`, CLI `holger-server audit trail|verify|export`; Auditor or Admin only (`holger-audit-10`); verify names the first break (`holger-audit-4`). **SIEM export** (`audit.siem`, every sink off by default): JSON lines, RFC 5424 syslog over UDP/TCP/TLS, CEF, OTLP/HTTP JSON — a follower thread per sink with a cursor, never in the writer's way; local sinks only on the airgap side (`holger-audit-9`) |
| Replication | source→dest one-shot copy shipped — pure `replication::replicate_pairs` engine, content-id idempotent (skip-by-sha256), gated dry-run/`--execute`; `holger-server replicate --from --to` CLI (two file-backed stores) + `ReplicationService.Replicate` gRPC (admin-gated). Continuous/multi-site 🟡 |
### Key design decisions
1. **RON config** — config is a graph (repos → endpoints → storage), not a flat list. RON handles typed enums and nested refs naturally.
2. **gRPC for control** — streaming, strong typing, efficient binary. Package clients use the native HTTP format doors (and the OCI door).
3. **Dual access** — gRPC for network; direct Rust calls for in-process (agent, tests).
4. **Separate agent and server** — airgap requires offline operation. Agent runs internet-side, server runs secure-side. No direct connectivity needed.
5. **Trait-based backends** — `RepositoryBackendTrait` + `ConnectorTrait` add new formats without touching core logic.
6. **Mannequin** — mock the whole external world in one process. Replaces Keycloak + Nexus + CA for dev.
---
## Local dev setup
A lone clone builds from crates.io: no committed manifest or `[patch]` points out of
the repository (`xtask/tests/lone_clone.rs`). For estate work, check out `edda`,
`Skidbladnir`, `monetize`, `tunnr`, `draupnir`, `modgunn` and `facett` next to
`holger/` and run `cargo xtask siblings` once: it copies
`.cargo/estate-siblings.toml` (every sibling patch, ONE table) to the uncommitted
`.cargo/siblings.toml` that `.cargo/config.toml` includes as optional, and every
other `cargo xtask` verb refreshes it. `cargo xtask` always starts: the alias runs
`xtask/boot`, a detached launcher with no dependency, which refreshes the copy and
hands the verb to the real `xtask` (run it from the repository root). Add a sibling dependency by VERSION in the member's
manifest and its path to `.cargo/estate-siblings.toml`, never `path = "../<sibling>"`.
`estate-tests/` (the tests against the real ONE box) is a detached workspace that
reaches the siblings by path. znippy comes from crates.io.
---
## Build & test
Build and test in the `nordisk-cargo-toolchain:live` container. Run
`git lfs install --local && git lfs pull` first (the front's picture and the boot
logo are LFS; a test names the pointer file if you forget).
```bash
cargo xtask test # the wasm guests, then default + metered suites
cargo xtask gate # rows, builds, tests and the declared arms
```
Tests that load a wasm guest build it on demand (`traits::test_support::wasm_guest`:
host cargo, `--target wasm32-unknown-unknown`, into `<target>/wasm-guests`); no
container and no manual step. Shared test helpers (`fstatus`, `MemReader`,
`CountingReader`, the race-free `free_port` lease, the LFS check) live in
`traits::test_support` behind the `test-support` feature, a dev-dependency only.
### Benchmarks: measure THROUGH the door that persists
```bash
cargo xtask bench # measure + prove the row landed
cargo xtask bench --heavy # …including the #[ignore]'d container arms
NORNIR_BENCH_ONLY=holger.blake3 cargo xtask bench # one arm
cargo xtask bench-landed # the guard: newest receipt has a warehouse row
```
`cargo xtask bench` is a THIN door onto `nornir bench run holger` — it owns no
arm, no scale and no feature (those live in `xtask/examples/nornir-bench.rs` and
`edda/crates/nornir-bench`). It resolves the absolute
`workspace_nordisk/release/nornir.toml`, passes `NORNIR_BENCH_ONLY`,
`NORNIR_BENCH_SKIP`, `NORNIR_BENCH_FEATURES`, `NORNIR_MACHINE` and
`HOLGER_BENCH_SCALE` through, and then **counts `bench_runs` before and after**:
no new row is RED, whatever the exit code said. Running the example directly and
redirecting stdout — which is how every holger measurement was taken until
2026-09-20 — writes no row anywhere.
**Every arm here, no nornir: `cargo xtask bench --local`** (2026-10-23). It runs the same
example, writes `bench-results/receipt-<run id>.json`, `latest.json` and `latest.txt` (one
line per arm), appends the BenchRun to `bench_history.jsonl`, and checks the tracked
metrics (`ops_per_sec`, `mb_per_sec`, every `*_p50_ms`) against the baseline of the same
`NORNIR_MACHINE` (`bench_baseline.<machine>.json`; the legacy `bench_baseline.json` is oden's):
worse than its `threshold_pct` is red. `--only <prefix>`, `--fetch-fixtures` (the large intake
fixtures, once, online side), `--update-baseline` (refused without `NORNIR_MACHINE`), `--runs N`
(judge, or take a baseline from, the per-metric median of N runs), `--no-regress` (report the
verdict, never fail). **The bench machine is builder-64** (2026-10-25; oden is retired): judge a
phase with `NORNIR_MACHINE=builder-64 cargo xtask bench --local --runs 3`, without
`--no-regress`. In the toolchain container set `HOLGER_BENCH_FIXTURES=/target/bench-fixtures` (the
default is `<workspace>/target`, not `$CARGO_TARGET_DIR`): without the fixture cache (270k OSV
advisories, the large packages) the CVE arm and every `*_l`/`oci_layer` metric are missing and the
"real DB" arms measure an empty one. The baseline lists `noisy` metrics (they moved more than the
threshold between its own three quiet runs: the upstream fetches, the large-file seals,
`concurrent_upload`, `secure_update_*`), and medians under 1 µs (`MIN_JUDGED_MS`) are below the
timer's resolution; both are reported, never judged. The `holger.intake_*` arms (bench-scenarios `intake.rs`) time every step of
the intake chain per package type over real packages (`bench-scenarios/fixtures/intake`):
hash, inspect, malware (`HOLGER_BENCH_CLAMD=host:port` for ClamAV), CVE against the real
OSV dumps, licence, SBOM, provenance (holger-provenance and the primitives), the gate
catalog and the source gates, seal, the whole chain for one package and N in parallel, and a
cold znippy fetch from `HOLGER_BENCH_SLOW_DIR`. `holger.sandbox_scan` times the scan sandbox
(the per-package overhead of the worker's container against in-process, the chain in the
container, chainguard/static's CVE scan, EICAR through clamd's sandbox); `scripts/bench-sandbox.sh`
brings up the podman API socket and clamd's sandbox and runs it. Threshold 50% on both bench
machines (oden: two identical runs differed by up to 44%; builder-64: three quiet runs differed
by a median of 3.2%, the noisy metrics aside; see each baseline's `threshold_why`).
LOCAL vs CENTRAL is `NORNIR_SERVER` (or `--server <url>`): unset, nornir writes
the iceberg row here; set, the SERVER runs the bench and appends the row,
because a thin client cannot hold the single-writer redb catalog lock. The door
prints which mode it used and where the row landed.
One run writes the value twice: the `bench_runs` row and the
`bench_history.jsonl` receipt at the repo root. The receipt **is committed** —
it is the backup the doc mashup renders from when the warehouse cannot be opened
— and `.gitattributes` gives it `merge=union` so two branches' runs both
survive a merge.
---
## Release procedure
0. `cargo audit` and `cargo deny check` (policy in `deny.toml`). This step is
first because it is the one that decayed unnoticed: the 2026-06-14 review
recorded 0 advisories, the 2026-09-11 review found 29, and nothing ran
either tool in between. `.forgejo/workflows/security.yml` runs the audit
half on every push; `cargo deny` needs the sibling checkouts, so it belongs
here.
1. Remove all `[patch.crates-io]` path overrides
2. `cargo xtask bench` — holger vs Nexus via mannequin
3. Gate: holger `ops_sec` ≥ Nexus for every artifact type
4. Gate: no type drops >10% vs last same-machine `bench_history.jsonl` entry
5. Gate: agent push → store → pull round-trip passes (rust, jar, wheel)
6. Update `CLAUDE.md` / `README.md` if architecture or numbers changed
7. Append result to `bench_history.jsonl`, commit
8. `git tag vX.Y.Z && git push --tags`
9. Publish: `traits → server/repository/* → server/lib → agent/lib → server/cli, agent/cli`
10. SLSA Build L3: the crates.io release goes through dwarf's release runner
(dwarves `crates/dwarves-slsa/scripts/slsa-release.sh`: build, the owner signs on the
YubiKey, verify, publish the attested bytes only), and the bundle lands in
`attestations/` (docs/SLSA.md).