<!-- nornir:class: design -->
# holger-ds — the real Python journey — design
> **Class:** `design` · **Function:** `holger-ds` (`uses` `serve` for the native HTTP
> door, `registry` for `PipRepoZnippy`, `storage` for znippy extract, the agent
> `world` connector for ingest). The whole-system view is [`design.md`](design.md);
> the serving door is [`serve-design.md`](serve-design.md).
## What holger-ds is
**holger-ds** is Holger standing in as a real **Python package index** for a
data-science stack: a `uv`/`pip` install and an Airflow DAG pull wheels from Holger,
end to end, over the real package protocol — never in-process, never by shelling to
`uv`/`pip`. It is **additive**: it reuses the existing `PipRepoZnippy` backend, the
`start_http_gateway` serve path, the agent `world` PyPI ingest, and the served-bench
foundation (`bench-scenarios/src/served.rs`). No existing registry path changes; the
crate/jar/wheel arms and the just-landed consolidation refactor are untouched.
The journey has three legs, all over one immutable znippy archive:
1. **Ingest** — the agent `world` connector (`agent/connector/world/src/pip.rs`)
parses a `requirements.txt` or a `uv.lock`, resolves files+digests against the
PyPI JSON API, and lays each distribution at `packages/{normalized-name}/{file}`
inside a `.znippy` archive.
2. **Serve** — `PipRepoZnippy` exposes that archive as a **PEP 503** HTML simple
index and a **PEP 691** JSON index through Holger's native HTTP door
(`handle_http2_request`), plus the wheel-download path.
3. **Install** — a real consumer (uv/pip, or the pure-Rust journey client in tests)
fetches the index, resolves a wheel, downloads it, and **verifies the SHA-256** it
recomputes against the digest the index advertised.
## The serving surface (chosen design)
`PipRepoZnippy::handle_http2_request` routes four shapes after stripping the
`/{repo}` prefix:
| Request | Response |
|---|---|
| `GET /simple/` | PEP 503 root: one `<a>` per package name (HTML-escaped) |
| `GET /simple/{name}/` | PEP 503 detail: one `<a href="…/{file}#sha256={hex}">` per file |
| `GET /simple/{name}/json` | PEP 691 JSON: `{meta, name, files:[{filename,url,hashes:{sha256}}]}` |
| `GET /packages/{name}/{file}` | the distribution bytes (`application/zip` / `gzip`) |
### Content-addressing: SHA-256 on the PEP-facing surface
znippy content-addresses every stored file with **blake3**, but `pip`/`uv` do **not**
accept blake3 — they verify a download against a **SHA-256** (or sha384/512) named in
the `#sha256=` link fragment (PEP 503) or the `hashes` map (PEP 691). So the
PEP-facing digest is a real SHA-256 of the served file bytes, computed **once per
file and memoised** (`PipRepoZnippy::sha_cache`). This is the digest a real uv install
recomputes and aborts on mismatch — the "wheel content-addressing" guarantee.
The SHA-256 is computed **only** on the index-detail path (`/simple/{name}/[json]`),
never on the artifact-download hot path (`/packages/…`), which stays byte-identical
to before — so the benched serve hot path (`artifactory_load`/`nexus_load`) does not
regress.
### PEP 691 negotiation — the one caveat
`uv`/`pip` select PEP 691 JSON via an `Accept: application/vnd.pypi.simple.v1+json`
header. The `RepositoryBackendTrait::handle_http2_request` seam does **not** surface
request headers today, so a header-driven content negotiation would need a
cross-backend trait change (all ~13 `znippy-*` backends). To stay additive, the JSON
representation is reachable at the explicit `/simple/{name}/json` route — the format
is fully implemented and tested, and a real uv install still works over the valid
**PEP 503 HTML** path (now carrying hashes). Wiring true `Accept`-header negotiation
is the documented follow-up (see the problem log).
## Reachable as a HEAVY bench
`holger.pip_index_serve` (registered in `xtask/examples/nornir-bench.rs`, picked up by
`nornir bench run holger`) is the Python sibling of `artifactory_load`/`nexus_load`:
- builds a synthetic wheel corpus (`pip_corpus`, byte-budgeted on heavy tiers),
- stands up the **real** Holger HTTP gateway over `PipRepoZnippy`,
- drives the pure-Rust gatling GET pool (`served::run_pip_index_fetch`) over the
**wheel-download** path (headline `holger_served_mbs` + `ops_per_sec`) and the
**`/simple/{name}/` resolve** path (`holger_index_ops_per_sec`),
- keeps the in-process `get_file` ceiling as an internal Holger-vs-Holger reference
(`holger_inprocess_read_ceiling_mbs`), never beside a competitor,
- records host cores + memory like every heavy arm.
## Why this design
- **Reuse over duplication.** No new server, no new protocol code paths, no second
wheel builder — the same `make_fake_wheel` corpus feeds both the in-process
`pip_wheel_*` arms and the served `pip_index_serve` arm, and the same
`start_holger_gateway`/`bombard_http_get` foundation the crate/JVM arms use.
- **Honest measurement.** Delivery is measured over the package protocol (HTTP),
exactly as the Nexus/Artifactory head-to-heads are; the in-process figure is kept
separate as a ceiling.
- **Real verification.** The journey test recomputes SHA-256 and rejects a mismatch —
it exercises the exact failure a real uv install guards against.