# znippy — reasoning (f0, why this overall shape)
> **Class:** `reasoning`. This file holds the *why*, the trade-offs, and **every design we
> rejected** at the whole-system level. The descriptive shape we actually run is in
> [`design.md`](design.md), which by rule mentions none of this. Per-function reasoning lives in
> [`compress-reasoning.md`](compress-reasoning.md), [`index-reasoning.md`](index-reasoning.md),
> and [`zoomies-reasoning.md`](zoomies-reasoning.md); the deep engine narrative in the classified
> chapters ([gatling](gatling.md), [inflate](inflate.md), [benching](benching.md)).
## Why znippy exists at all
Every existing archiver forces a choice znippy refuses to make. `tar+zstd` compresses all-core but
**decompresses serial** and can't pull one file without streaming the whole thing. `zip` gives
random single-file reads but is single-threaded on both ends. `7z` solid archives trade random
access for ratio. `parquet` is queryable and columnar but isn't a *file archive*. znippy's whole
reason to exist is to be **all-core on both ends, random-access per file, AND directly queryable**
— without pretending to beat `zstd -19` on raw ratio. Ratio was deliberately *not* the goal;
every design call below protects the three properties that were.
## Why the index is Arrow IPC (and nothing bespoke)
The "query it with no znippy library" property is only real if the index is a format outside tools
already speak. **Rejected: a bespoke on-disk B-tree / custom index format** — it would be faster to
seek but would make DuckDB/Polars/DataFusion useless against the archive, killing the third
property. Arrow IPC is the one layout that is simultaneously a valid seekable index *and* a table
an analytics engine reads directly. That single decision is why the lookup sub-index is a sorted
Arrow stream rather than a private structure — see [`index-reasoning.md`](index-reasoning.md) for
why we *also* carry an `fst` trie beside it.
## Why the four laws
The laws are not style; each buys a measured property:
- **L1 zero-copy channels.** On NVMe at ~7 GB/s, copying a `Vec<u8>` payload between the reader and
a compressor thread halves throughput. **Rejected: sending decoded payloads over channels** — we
send slot indices into a pre-allocated ring instead, so bytes never move. The engine that
embodies this (Gatling) has its own rejected-alternatives record in
[`compress-reasoning.md`](compress-reasoning.md) and [`gatling.md`](gatling.md).
- **L2 no hot-path allocation.** A fresh `Vec` per chunk is allocator churn that shows up as
variance in the wall time (the candidate-offset output model measured 7.12 s → 6.70 s once churn
was removed — [`inflate.md`](inflate.md)). Workers reuse buffers; the codec context is long-lived.
- **L3 checksums bound to `chunk_seq`, not thread identity.** **Rejected: hashing in completion
order** — it makes the archive's checksums depend on how many cores ran, so no other machine
could reproduce them. Binding to `chunk_seq` makes integrity a property of the *data*, not the
run, which is also what lets f7 sign a stable merkle root.
- **L4 concurrent writer.** **Rejected: compressors that block on the index writer** — the seal is
serial by nature, so any wait on it caps throughput at the writer's speed. The finalizer runs on
its own thread from `ChunkMeta` only.
## Why one Gatling engine for both entry points
Compress-from-directory and compress-from-memory could each have a bespoke pipeline. **Rejected:
two engines** — they would drift, and the no-barrier invariants (the hard part) would have to be
proven twice. `slot_packer` and `stream_packer` share the same worker/writer/finalizer so the
saturation guarantees are written once. The same engine also powers osm-katana's xml→pbf path,
which is what made it worth extracting into `znippy-zoomies`.
## Why plugins instead of a fixed schema
An airgap mirror wants the *ecosystem* view (maven coordinates, python wheel tags) next to the
bytes. **Rejected: baking every ecosystem's columns into one schema** — it would bloat every
archive and couple the format to a fixed set of package types. Instead a plugin contributes only
its own Arrow columns and the writer unions them; the CLI autodiscovers via `meta()` so there is no
hand-maintained name table. **Rejected: a central handler registry** for the same reason.
## Why "store if incompressible" is unconditional
Already-compressed inputs (`.jar`, `.gz`, `.crate`, `.png`, `.parquet`) are stored verbatim on an
extension skip-list — obvious. Less obvious: **any** slice where the produced frame is ≥ the input
is *also* stored raw, `compressed = false`. **Rejected: trusting the extension list alone** — it
would still pay codec time (and lose bytes) on incompressible content with an innocent extension.
The unconditional size check makes "never bigger than the input, per slice" a guarantee.
The size half of that objection was always covered; the **CPU** half was not, and on git data it
was nearly the whole cost. A caller `ContentHint` and a magic-byte probe now close it
(`znippy-common/src/precompressed.rs`) — 92–95% of compress CPU on real loose objects and a real
packfile, measured. **Rejected: an entropy estimate** as a general fallback — a tunable threshold
that costs a scan and misjudges in both directions, whose wrong answers show up as neither a
failure nor a red test. Every probe is an exact pattern at a fixed offset instead.
## Why a dynamic store *and* a static seal (f6)
A live mirror needs to accept new commits and survive restarts; a distributed artifact needs to be
one immutable, content-addressed file with no warehouse dependency. **Rejected: only a static
format** (can't resume across a reboot) and **rejected: only a dynamic Iceberg store** (every
reader needs the warehouse). Carrying both — `IcebergSink` for the dynamic phase, `seal()` to
freeze it — lets one workflow be resumable while it's being built and immutable once shipped. Seal
reuses the blobs *verbatim* and rewrites only the metadata tail, byte-identical to a fresh
`compress`, so a sealed archive is indistinguishable from a one-shot one.
## Why signing is a merkle-fold, not a re-hash
Provenance (f7) had to be free on top of the integrity we already compute. **Rejected: re-reading
content to sign it** — every chunk already carries a blake3 hash, so the archive root is a
merkle-fold of hashes already in hand, no content re-read. It rides reserved manifest sections, so
an unsigned archive is byte-unchanged and the blob/compress path is untouched — signing is strictly
additive, never a second write path.
## Why nornir bench, not criterion
The whole benchmark story is a deliberate fork away from `cargo bench`/criterion — single-shot
real-workload numbers historized by git SHA and gated for >10% regression, not statistical
micro-sampling. The full argument (dev-deps can't live in a `bin`; criterion's output isn't
machine-historizable the way the release gate needs) is its own chapter: [`benching.md`](benching.md).