<!-- nornir:class: design -->
# Deb/RPM native metadata — design
> **Class:** `design` · **Function:** `serve` (the Deb + RPM repository backends).
> Sibling how-tos: [`deb-rpm-native-metadata-guide`](deb-rpm-native-metadata-guide.md)
> (serve a repo) and [`deb-rpm-native-metadata-for-idiots`](deb-rpm-native-metadata-for-idiots.md)
> (point-and-go). Whole-system view: [`design.md`](design.md); serve path:
> [`serve-design.md`](serve-design.md).
## The problem — a mirror of bare packages
Holger serves a Debian (APT) repo and an RPM (yum/dnf) repo straight out of an
immutable `.znippy` archive. When the archive was built by the znippy `deb_native`
/ `rpm_native` plugin it carries a *typed sub-index* — a pre-parsed view of every
package's control/header fields — and holger just reads that. But an archive is
often a **plain mirror of package bytes**: someone dropped a folder of `.deb` /
`.rpm` files in, no plugin, no sub-index. The client (`apt`, `dnf`) still demands
a full index — APT `Packages`/`Release`, rpm-md `repomd.xml`/`primary.xml` — with
real fields it can resolve dependencies against.
Previously holger filled the gap by **guessing from the filename**: it split
`{name}_{ver}_{arch}.deb` / `{name}-{ver}-{rel}.{arch}.rpm` for coordinates and
emitted nothing else — no `Depends`, no `Maintainer`, RPM `epoch=0`, no `<format>`.
That is enough to *list* packages, not enough to *resolve* them. Both backends
carried an honest RED maturity-disclosure row saying so.
This change closes the gap: holger now **parses the real control stanza / RPM
header out of the served package bytes**, with no reliance on the znippy plugin.
Filename-derivation drops to last-resort fallback and both maturity rows flip green.
## Fallback order — three tiers, most-authoritative first
Both backends resolve each package's metadata in the same precedence:
| # | Source | Applies when |
|---|---|---|
| 1 | **Typed sub-index** (`as_deb` / `as_rpm` view) | the archive was built with the znippy `deb_native` / `rpm_native` plugin |
| 2 | **Native parse of the served bytes** — *new* | mirrored / plugin-less archive; the package body itself is decoded |
| 3 | **Filename derivation** | tiers 1 & 2 both miss (unreadable/hostile package) — coordinates only |
There is a **fourth, outer tier that wins over all of these for the index files
themselves**: authoritative *path-passthrough*. If the archive already carries a
real `Release`/`Packages` or `repomd.xml`/`primary.xml` (a byte-exact upstream
mirror), holger serves that verbatim and never synthesizes. Only when the index
path misses does synthesis run — and synthesis is where the three tiers above pick
each package's fields.
- Deb passthrough-vs-synthesize dispatch: `server/repository/znippy-deb/src/lib.rs:347`
(`handle_http2_request`) → passthrough `get_file` at `:364`, then
`synthesize_index` at `:369`/`:275`.
- Deb per-package precedence (tiers 1→2→3): `server/repository/znippy-deb/src/lib.rs:194`,
the `controls.get(...).or_else(|| control::control_from_deb(&bytes))` chain.
- RPM per-package precedence: `server/repository/znippy-rpm/src/lib.rs:169`, the
`meta.get(...)` → `header::parse_rpm_header(...)` match.
## Deb — the `ar` + `control.tar.*` parser
A `.deb` is an `ar(5)` archive with exactly three members in order: `debian-binary`,
`control.tar.{,gz,xz,zst}`, `data.tar.{,gz,xz,zst}`. Only the control tarball is
needed, and it holds a single `./control` file — the Debian control stanza
(`Depends` / `Maintainer` / `Description` / …). Module:
`server/repository/znippy-deb/src/control.rs`.
Pipeline (`control_from_deb`, `control.rs:32`):
1. **Hand-parse the `ar` container** (`control_tar_member`, `:41`). The `ar` format
is trivial — an 8-byte global magic (`!<arch>\n`, `AR_MAGIC` `:20`) then
fixed 60-byte member headers (`AR_HEADER` `:22`): name in bytes `0..16`
(space-padded, GNU may append `/`), decimal size in `48..58`, members padded to
an even boundary. No `ar` crate — walking it is a dozen lines. Every offset is
bounds-checked; a truncated or lying header returns `None`, never panics.
2. **Pick the codec from the member extension** (`decompress_control_tar`, `:72`) —
a pure extension match, not content sniffing:
`control.tar` → verbatim, `.gz` → `flate2::read::GzDecoder`,
`.xz` → `liblzma::read::XzDecoder`, `.zst` → `zstd::stream::read::Decoder`.
Anything else → `None`.
3. **Untar `./control`** (`control_from_tar`, `:89`) via the `tar` crate; return its
UTF-8 body.
The stanza is then normalized into `Packages` field lines by `clean_control_stanza`
(`lib.rs:308`), and holger appends its own repo-level `Filename` / `Size` / `SHA256`
(computed over the real served bytes) on top — the three fields APT verifies a
`[trusted=yes]` download against.
## RPM — the binary header tag-table parser
An `.rpm` is a **96-byte lead**, a **signature header**, then the **main header**.
Both headers share the RPMv3 structure: magic `8e ad e8 01`, 4 reserved bytes, a
big-endian `nindex` (entry count) and `hsize` (data-store length), then `nindex`
16-byte index entries `(tag, type, offset, count)`, then the `hsize`-byte store the
offsets point into. Module: `server/repository/znippy-rpm/src/header.rs`.
Pipeline (`parse_rpm_header`, `header.rs:174`):
1. **Skip the lead** — validate the 4-byte lead magic (`LEAD_MAGIC` `:22`), step
over 96 bytes (`LEAD_LEN` `:21`).
2. **Step over the signature header** without fully decoding it (`header_end`, `:64`
— reads its `nindex`/`hsize`, returns the byte just past its store), then align
the main header to an 8-byte boundary: `(sig_end + 7) & !7` (`lib.rs`/`:186`).
3. **Decode the main header's tag table** (`parse_header`, `:84`) into `Entry`
records, each validated so its `offset` lands inside the store.
4. **Read the wanted tags** by number into an `RpmMetaRow` (`znippy_common::views`):
`EPOCH` (1003, int32), `SUMMARY` (1004), `VENDOR` (1011), `LICENSE` (1014),
`URL` (1020), `SOURCERPM` (1044), `PROVIDENAME` (1047, string-array),
`REQUIRENAME` (1049, string-array). Tag constants: `header.rs:27`. Strings via
`read_string` (`:118`), arrays via `read_string_array` (`:125`).
`primary.xml` synthesis then emits `epoch="…"`, `<summary>`, `<url>`, and the
`<format>` block (`<rpm:license>`, `<rpm:provides>`/`<rpm:requires>` as
`<rpm:entry>` rows) from that row — `server/repository/znippy-rpm/src/lib.rs:169`
onward. `repomd.xml` references `primary.xml` by SHA256+size; the index is served
uncompressed so `dnf` verifies it directly.
## Bounds, bomb-caps, and hostile input
Both parsers treat the package as adversarial — a mirrored archive may contain
anything. Rules held throughout:
- **No `unsafe`, no panics.** Every slice is `get(..)`-checked or width-validated;
malformed bytes yield `None`, and the caller silently drops to the next fallback
tier. Proven by the `non_*_bytes_yield_none` / `truncated_store_yields_none`
unit tests in each module.
- **Decompression-bomb cap (Deb):** the control tarball is read through
`take(MAX_CONTROL_TAR)` — 16 MiB (`control.rs:25`). A real `control` is a few KiB;
a 4-byte `.zst` that inflates to gigabytes is truncated, not honoured.
- **Resource caps (RPM):** `MAX_NINDEX` = 65 536 tags, `MAX_STORE` = 64 MiB
(`header.rs:45`). A header claiming billions of entries is rejected before any
allocation.
- **Every arithmetic on offsets is `checked_*`** so a crafted size can't wrap and
read out of bounds.
Both modules ship a synthetic-fixture encoder in a `tests_support` submodule (a real
minimal `.deb` / byte-exact `.rpm`) so the crate's `lib.rs` integration tests drive
the *whole* synthesis path — `native_control_parse_enriches_packages` and
`native_header_parse_enriches_primary` prove the emitted index carries the parsed
fields with **no** typed view present.
## Codec dep note — one workspace-wide `links = "lzma"`
**Rule for future contributors: use `liblzma`, never `xz2`.** Both crates wrap the
same native `lzma` library and both declare `links = "lzma"` in their `-sys` crate
(`xz2` → `lzma-sys`; `liblzma` → `liblzma-sys`). Cargo permits only **one** crate
with a given `links` value in a dependency graph, so a workspace that pulls *both*
fails to resolve — a hard error, not a warning.
DataFusion (reached via skade / knut / **nornir**) already pulls `liblzma-sys`.
holger-znippy-deb therefore decodes `control.tar.xz` with `liblzma`
(`control.rs:81`), and the two `zip` dev-deps that defaulted to an `xz` feature now
set `default-features = false, features = ["deflate", "time"]` (they only ever write
Deflated entries):
- `server/repository/znippy-deb/Cargo.toml` — `liblzma = "0.4"` (was `xz2`), with the
rationale in a comment beside it.
- `bench-scenarios/Cargo.toml`, `server/repository/znippy-rust/Cargo.toml` — `zip`
with `default-features = false`.
Reintroducing `xz2` (or any crate that transitively adds `lzma-sys`) re-breaks
workspace resolution for every consumer that also sees DataFusion. If you need xz,
reach for `liblzma`.