# znippy — design (f0, the whole system)
> **Class:** `design` · **Function axis:** `f0` (the system) + `f1` compress · `f2` decompress ·
> `f3` index/seek · `f4` plugins · `f5` zoomies · `f6` iceberg/seal · `f7` signing.
> Strict: this file states *only the shape we run*. Every *why* and every rejected
> alternative lives in [`reasoning.md`](reasoning.md); the per-function detail lives in each
> function's own `*-design.md` (linked below). How to use it: [`guide.md`](guide.md).
## f0 — one archive, three properties
znippy is a content-addressed, seekable compressor: it packs a directory across **every core**,
lets you pull **one file back** without touching the rest, and keeps its index as **plain Arrow
IPC** so an outside SQL engine reads it with no znippy code. One container carries all three:
```text
[ blob_0 … blob_N ] compressed or stored-raw chunk bytes
[ data sub-index_0 … _M ] one Arrow IPC stream per (pkg_type, repo)
[ lookup sub-index ] base columns, sorted by (relative_path, chunk_seq)
[ trie blob ] fst::Map<relative_path → first lookup row>
[ manifest Arrow IPC ] one row per section
[ 8-byte MAGIC "ZNPYMIDX" ][ u64 manifest_offset ]
```
Base index columns (every archive): `relative_path · chunk_seq · fdata_offset · compressed ·
uncompressed_size · blob_offset · blob_size · checksum` (32-byte blake3). Schema:
`znippy-common/src/index.rs`. Each sub-index is a valid Arrow IPC file — DuckDB / Polars /
DataFusion read it directly.
Four laws hold everywhere below:
| law | statement | why it holds |
|---|---|---|
| **L1** | inter-thread messages are slot indices / metadata, **never `Vec<u8>`** | a payload copy at ~7 GB/s NVMe halves throughput (`chunk_revolver`) |
| **L2** | no allocation in the hot path | workers reuse buffers; the codec context is long-lived (`codec.rs`) |
| **L3** | checksums are bound to `chunk_seq` order, not thread identity | any core count reproduces the same blake3 (`index.rs`) |
| **L4** | the writer/finalizer runs concurrently | compressors never wait for the index to be built |
## How the functions wire together (the reuse graph)
| function | crate / source | `uses` | one line |
|---|---|---|---|
| **f1 compress** | `znippy-compress` (`slot_packer.rs`, `stream_packer.rs`) | f5, codec, f3-sink | Gatling no-barrier pack → blobs + index |
| **f2 decompress** | `znippy-common/src/decompress.rs` · `znippy-decompress` | f3, codec | N-worker positioned reads → files back |
| **f3 index/seek** | `znippy-common/src/index.rs`, `archive.rs`, `meta_sink.rs` | — | Arrow index + fst trie + footer; O(key) seeks |
| **f4 plugins** | `znippy-common/src/plugin.rs`, `znippy-plugin-*` | f5 (ljar) | package-aware typed columns |
| **f5 zoomies** | sibling `znippy-zoomies` (linflate, ljar, lgz, lbzip2, lzip-parallel, gatling, stree) | — | pure-Rust parallel decoders + the shared engine |
| **f6 iceberg/seal** | `znippy-iceberg` (`lib.rs`, `seal.rs`) · `meta_sink_append.rs` | f3 | dynamic Iceberg store ⇆ static sealed `.znippy` |
| **f7 signing** | `znippy-common/src/sign.rs` | f3 | detached CMS merkle-fold over the chunk hashes |
The **f5 gatling engine** is the shared substrate: f1 packs through it, and osm-katana's xml→pbf
path runs on the same `1 reader → N workers → 1 writer` shape. **f3** is the spine every reader
touches — f2 restores through it, f6 seals into it, f7 signs the roots it computes. Everything
outside **f6** is tokio-free.
## f1 — compress (the write path)
One no-barrier engine, two entry points. `slot_packer.rs::compress_dir` (io_uring, 128 files/batch
into `ChunkRevolver` coalescing slots) and `stream_packer.rs::compress_stream` (in-memory `Arc<Vec<u8>>`
rounds) share worker/writer/finalizer: every input is split into **slices** (the unit of both
blake3 integrity and parallel work) feeding one global slice queue; N workers blake3 the original
bytes, `compress_into` a reused buffer (or store raw when the frame ≥ input), and `pwrite` at an
`AtomicU64` cursor while the finalizer builds the Arrow index from `ChunkMeta` only. Detail:
[`compress-design.md`](compress-design.md).
## f2 — decompress (the read path)
`decompress.rs`: N workers share one atomic row cursor; each `fetch_add(row) → pread(blob_offset)
→ decompress_into(reused buf) → blake3 verify → pwrite(fdata_offset)`. No reader thread, no
channel — the index has every offset, so parallel `pread` gives NVMe its real queue depth. Detail:
[`decompress-design.md`](decompress-design.md).
## f3 — index & random-access seek
The lookup sub-index (base columns re-sorted by `(relative_path, chunk_seq)`) + an `fst::Map` trie
give **three ways to get a row**: eager full-index `ZnippyArchive::open`, lean `get_file` →
`locate_file` (trie O(key) → binary-search O(log n) → legacy scan), and external Arrow-IPC SQL.
Both live as **reserved** manifest sections (`pkg_type = i8::MIN`), so the 16-byte footer is
unchanged and older archives still open. Detail: [`index-design.md`](index-design.md).
## f4 — plugins (package-aware typed index)
A plugin implements `ZnippyPlugin` (`plugin.rs`): `matches_path`, `schema_fields`,
`extract_metadata` / `extract_batch`, `meta`, `run_command`. The writer composes the schema as
base columns + the union of registered plugins' fields; built-ins register once in
`handlers.rs::builtin_handlers()` and the CLI autodiscovers via `meta()`. Detail:
[`plugins-design.md`](plugins-design.md).
## f5 — zoomies (fast primitives)
Pure-Rust parallel decoders — `ljar`, `lgz`, `lbzip2`, `lzip-parallel` — on one shared SIMD
DEFLATE core (`linflate`), plus the generic `gatling` streaming engine, the `stree` static search
tree, and `osm-katana`. Each decoder borrows a `&[u8]` and never copies its input. Detail:
[`zoomies-design.md`](zoomies-design.md); measured throughput: [`benchmarks.md`](benchmarks.md).
## f6 — iceberg ⇆ seal
`IcebergSink` keeps the sub-index rows in skade/Iceberg tables (durable `catalog.redb`) — a
**dynamic** store that survives restarts, resumes, and is queryable live; the `.znippy` sidecar
holds pure blobs. `seal()` snapshots it into a **static** native container, reusing blobs verbatim
and rewriting only the metadata tail byte-identically to a fresh `compress`. Detail:
[`iceberg-design.md`](iceberg-design.md).
## f7 — provenance signing
Optional detached CMS on top of the per-chunk blake3 integrity: each signature is a merkle-fold of
the chunk hashes already in hand (no content re-read), written as reserved manifest sections via
`ArrowIpcSink::with_signer`. Off by default (feature `sign`); unsigned archives are byte-unchanged.
Detail: [`signing-design.md`](signing-design.md).
## CLI surface
`znippy compress | decompress | list | get | verify` (format-level, type-agnostic), `znippy seal`
(feature `iceberg`), `znippy handlers` (list plugins), `znippy run <handler> <cmd>…`. Entry:
`znippy-cli/src/lib.rs`.
## Dependency graph
<!-- nornir:gen:start:depgraph -->
<!-- nornir:gen:end:depgraph -->
## LAW 6 — the canonical branch, and the guard that makes the trap impossible
**Ask the repository which branch is canonical. Never assume, never hardcode, never
memorise a table.** Several repositories in this constellation carry BOTH a `main` and
a `master` on the remote and they do **not** agree on which is canonical — one
repository's trunk is the next one's abandoned branch. Enumerate them; do not learn
them:
```sh
for r in ~/git/*/; do
both=$(git -C "$r" ls-remote --heads origin main master | wc -l)
[ "$both" = 2 ] && printf '%-20s canonical=%s\n' "$(basename "$r")" \
"$(git -C "$r" symbolic-ref --short refs/remotes/origin/HEAD)"
done
```
On 2026-08-30 that listed exactly three, and the answer was **not** the same for all
three:
| repository | canonical | the trap |
|---|---|---|
| `znippy` | **`master`** | `main` exists and is **153 commits behind** — stale for months |
| `gunnar` | **`master`** | `main` exists; a lane pushed to it at 18:50 on 2026-08-30 |
| `znippy-zoomies` | **`main`** | `master` is the stale one — the opposite way round |
Two lanes pushed to a non-canonical trunk in one week. The gunnar one was harmless
**only by luck** — that tip happened to be an ancestor of `master`, so nothing was
stranded; the znippy one was not.
**What the hook refuses:** a push whose destination is a *trunk* name (`main`,
`master`, `trunk`, `develop`) that is not this repository's canonical trunk. A
deliberate push to a feature branch is untouched — `roadmap/obruten-kedja`,
`lane/whatever`, tags: all pass. The trap is the second default-looking branch, not
branches in general.
**How it knows:** `git symbolic-ref refs/remotes/<remote>/HEAD` — the remote's own
answer — falling back to asking the server with `git ls-remote --symref`. The
canonical name is **never written in the hook**; the same bytes therefore give
opposite answers in `znippy` and `znippy-zoomies`, which is the whole design. A
fourth place for the repositories to disagree would be the bug, not the fix. If
neither source answers, the hook says so and **allows**: a hook that blocked every
push on a repository it could not resolve would be deleted within the day, and then
nothing would guard anything.
The refusal names the branch that *is* right, the command that pushes there
(`git push <remote> HEAD:<canonical>`), the enumeration above, and the deliberate
override (`GIT_ALLOW_NONCANONICAL_TRUNK_PUSH=1`) for the one honest case — maintaining
or retiring the non-canonical trunk on purpose.
**Install** (hooks live outside the working tree, so a clone does not get them):
```sh
cp .githooks/pre-push .git/hooks/pre-push && chmod +x .git/hooks/pre-push
```
The hook chains to `git lfs pre-push` itself when `.gitattributes` tracks LFS paths
(it buffers stdin and replays it), so it *replaces* the LFS hook rather than fighting
it.
**Proven both ways in all three repositories** (LAW 2 — never accept a green you have
not seen red), 2026-08-30:
| repository | refused (exit 1) | allowed (exit 0) |
|---|---|---|
| `gunnar` | `main` | `master`, `roadmap/obruten-kedja` |
| `znippy` | `main` | `master`, `lane/x` |
| `znippy-zoomies` | `master` | `main` |
and end to end: `git push --dry-run origin HEAD:main` is refused in gunnar and in
znippy, while `git push --dry-run origin master` / `origin main` still runs the LFS
half.
## Which recipe reader this repo uses, and why (lane AJ, 2026-08-30)
**Reader:** `nornir-build-thing` **0.1.12**, PATH (`tests/`, `xtask/`).
**Why:** in-constellation; recipe is schema 1. znippy's graph ALSO held a registry copy at
**0.1.3** — eight generations behind the path copy, in one graph — pulled in by
published `modgunn`, whose own floating `"0.1"` let it sit there. Raising
modgunn's floor to `0.1.11` moved it to 0.1.11. znippy keeps NO `[patch.crates-io]`
on purpose (a patch fork is a structural publish blocker), so the last generation
of gap closes on the release cascade.
The rule itself — `[nornir-build-thing] schema = N`, a refusal that names the
version to upgrade to, and a loader that refuses to DROP a key it cannot read —
lives once, in **edda `.nornir/build-design.md`, "THE COMPATIBILITY RULE"**. The
cross-version guard that would have caught the 2026-08-30 skew is
`edda/nornir-build-thing-skew`.