# The compress engine — design
> **Class:** `design` · **Function axis:** `f0` (the Gatling engine) + `f1` reader · `f2` slot pool · `f3` workers · `f4` writer/finalizer · `f5` codec · `f6` metadata sink.
> Strict: this file states *only the shape we run*. Every *why*, every alternative, lives in [`compress-reasoning.md`](compress-reasoning.md); the how-to is in [`compress-guide.md`](compress-guide.md).
## f0 — the Gatling engine (the whole)
znippy is a content-addressed, seekable compressor: an Arrow-IPC index over zstd/lz4 chunks,
with blob dedup. Compress is the **"Gatling no-barrier" engine** underneath it. Every input is
split into **slices** — a slice is the single unit of *both* blake3 integrity *and* parallel
work. There is **one global slice queue** across all in-flight slots; there is **no per-core
barrier**, so a slow slice never blocks another slot's slices.
Two entry points feed *one* engine — a directory path and an in-memory stream — and both run
the identical reader → pool → workers → writer/finalizer shape:
| entry | signature | source | feeds |
|---|---|---|---|
| directory | `compress_dir(input_dir, output, no_skip, plugin, …)` | `znippy-compress/src/slot_packer.rs:88` | io_uring batched reader (128 files/batch) → `ChunkRevolver` coalescing slots |
| stream | `compress_stream(output, no_skip)` / `compress_stream_with_sink(output, no_skip, sink_factory)` | `znippy-compress/src/stream_packer.rs:83` / `:90` | in-memory rounds over `Arc<Vec<u8>>`, same worker/writer/finalizer shape |
The non-io_uring fallback reader path is **byte-identical** to the io_uring path — asserted by
`fallback_byte_identical_to_io_uring` (`znippy-compress/src/slot_packer.rs:817`).
Four laws hold the engine together:
| law | statement | enforced by |
|---|---|---|
| **G-1 zero-copy channel** | messages are **slot indices**, never `Vec<u8>` — a payload copy on ~7 GB/s NVMe halves throughput | reader → workers pass slot/slice indices; workers `pwrite` their own payload |
| **G-2 no alloc in hot path** | reused buffers, one long-lived codec ctx per worker, incremental blake3 | `compress_into` reuses `out` (`znippy-common/src/codec.rs:42`) |
| **G-3 deterministic checksums** | checksums are bound to `chunk_seq` order, **not** thread identity — any core count reproduces the same checksum | slices carry `chunk_seq`; finalizer orders by it |
| **G-4 writer/finalizer concurrent** | compressors never wait for the index — the finalizer builds it from `ChunkMeta` only, off the hot path | finalizer consumes `ChunkMeta`, no payload channel |
## f1 — the reader (1-end, read only)
The io_uring reader submits **128 files per batch** and fills `ChunkRevolver` coalescing slots; it does
**no CPU work** — file split is a cheap O(n_workers) candidate-offset division, not compression.
Big files (**> one slice**) are sliced across slots; small files pack into a single slot.
`file_split_block_size` = **10 MB**: a file larger than that spans several index rows, one per
chunk, all sharing `relative_path` and ordered by `chunk_seq`.
## f2 — the slot pool (the Gatling magazine)
The pool itself is **upstream**: `gatling::revolver` in coalescing mode, reached
as `znippy_zoomies::chunk_revolver`. znippy owned a fork of it
(`slotpool.rs::Magazine`) until 2026-08-25 and no longer does — the canonical
implementation beat the replica in 5 of 5 interleaved runs by 5-15 % with 7.9x
fewer allocations, so the collapse cost nothing and removed a second writer.
| type | role | source |
|---|---|---|
| `ChunkRevolver::with_coalescing` | the slot buffers the reader fills | `gatling/src/revolver/coalesce.rs:255` |
| `PackedSlice` | one slice's descriptor pulled by a worker | `coalesce.rs:66` |
| `Filler<'a>` | the reader's writable view, clamped to remaining | `coalesce.rs:351` |
| `Returner` | returns a slot to the pool **only after its last slice** | `coalesce.rs:143` |
What stays znippy's is the **cut policy**, `PoolPlan`
(`znippy-common/src/poolplan.rs`): `slice_size` is the big/small partition
threshold and the big-file cut length, so it lands in `chunk_seq`,
`fdata_offset` and every chunk checksum. `slot_size` and `num_slots` are only
how much buffer is in flight and are planned against the cgroup budget, which is
why a memory-shrunk run is byte-identical to a fat one.
Two behaviours differ from the fork and both are safer. A `Filler` dropped
un-published returns its slot (the fork leaked it), and `claim_slot` blocks for
the pool's whole life instead of reporting shutdown by sender-drop. The first
one bites a drain loop that does not hold what it reclaims — see
`slot_packer.rs`, where the mechanical translation of that loop was a
use-after-free that every unit test passed through.
## f3 — the workers (N-end, carry a slice through ALL heavy stages)
N workers self-dispatch from the one global slice queue. Per slice a worker: **blake3s the
ORIGINAL bytes** (integrity is over the source, not the frame), then `compress_into` a *reused*
buffer (or stores raw), then `pwrite`s its own payload at an `AtomicU64` cursor. No barrier: a
worker that finishes a slice grabs the next, regardless of which slot it belongs to.
## f4 — the writer / finalizer (1-end, stitch/drain in order)
Workers write payload themselves; the **one finalizer** builds the Arrow index from `ChunkMeta`
only — there is no payload channel to the finalizer (**G-4**). It orders rows by `chunk_seq`
(**G-3**) and hands the metadata tail to the sink (`f6`).
## f5 — the codec (store-if-incompressible)
`Codec::compress_into(&mut self, input, out)` (`znippy-common/src/codec.rs:42`) runs OpenZL
zstd/lz4 with per-worker buffer reuse and a long-lived ctx; `decompress_into` (`:66`) mirrors it.
Two store-raw rules set the index `compressed` flag to **false**:
1. **skip path** — resolved by `znippy-common/src/precompressed.rs` (a re-export of
`znippy-zoomies/ldeflate/src/precompressed.rs`, which owns the probe; only the
extension leg is znippy's) in a fixed order:
an explicit caller `ContentHint`, then the extension table (`.jar` / `.gz` /
`.crate` / `.png` / `.parquet` / `.pack` / `.idx`), then a magic-byte probe over
the first 16 bytes (zlib, `PACK`, gzip, zstd, xz, bzip2, zip, png, jpeg …). Stored
raw without a compress attempt. The probe is the only tier that can reach an entry
with **no extension**, which is what a git loose object is.
2. **any slice** whose produced frame is **≥ its input** is stored raw.
`no_skip` forces compression of every slice (disables rule 1); it is exactly
`SkipPolicy::compress_everything()`. `compress_dir_with_policy` /
`compress_stream_with_policy` take the full `SkipPolicy`, so a caller that already
knows its input is compressed says so once for the batch rather than being sniffed.
Rule 1 is a **CPU** optimisation, not a size one — rule 2 already guaranteed the bytes.
Measured 2026-08-03 on oden, 5388 real git loose objects (35.8 MB) and a real 22.4 MB
packfile: compress CPU 3.70 s → 0.30 s and 4.06 s → 0.21 s (92% / 95% saved). The
archive grows 0.17% / 1.2%, because rule 2 had been recovering that much from
already-deflated data at full codec cost; `--no-skip` buys it back.
## f6 — the metadata sink
`trait ArchiveMetaSink` (`znippy-common/src/meta_sink.rs:43`); default impl `ArrowIpcSink`
(`:73`) writes sub-indexes + a lookup + an fst trie + a manifest + the `ZNPYMIDX` footer. The
sink is chosen by a `MetaSinkFactory` so the container backend (inline Arrow IPC, or an Iceberg
warehouse) is swappable without `znippy-compress` taking the heavy async dependency.
## f7 — the tree's shape (`__znippy_fsmeta__`)
The data index is a table of **chunks**, and a chunk belongs to a regular file.
Two facts of a real tree therefore had no row shape at all and were dropped:
* an **empty directory** owns no file, so it owns no chunk. Extract only ran
`create_dir_all` on a *file's* parent, so a directory containing nothing did
not come back.
* a **unix mode**. Nothing recorded `metadata().mode()` and nothing restored it,
so an extracted file got `0666 & ~umask` — a `0755` script came back `0664`.
Both were silent: `verify` green, every file byte-exact, tree still wrong.
Fixed with one reserved section, `ZNIPPY_FSMETA_MODULE` (`znippy-common/src/fsmeta.rs`),
holding Arrow IPC `(relative_path, kind, mode)`. Reserved means `is_reserved_module`
already makes every data reader skip it, so it is additive both ways — no
`ZNIPPY_FORMAT_VERSION` bump, no new column for the iceberg/append/gunnar writers,
and an older znippy opens the archive unchanged. A `mode` column on the base schema
would have been all three of those; a zero-byte data row per directory would make
`list`/`get`/DuckDB see a *file* where a directory is.
Rows are one per directory (empty or not), plus **one archive-scoped
`FileDefaultMode` row** carrying the tree's modal file mode and one row per file
that deviates. The first cut compared against a hardcoded `0o644` and its own test
caught it: that is `0o666 & ~umask` for `umask 022`, and on a `umask 002` box every
plain file "deviated". A row per file is exact but pays a second copy of every path
the lookup sub-index already holds — ~4 MB on a 50 000-blob mirror to say one thing
50 000 times. The modal row is exact, umask-independent, and O(distinct modes).
Restore runs in three phases and each ordering is forced: **directories first**
(before extraction — nothing else will ever create an empty one), **file modes
after extraction** (a `0444` file must be written before it is sealed),
**directory modes last, deepest first** (a `0555` directory cannot have children
created inside it). Applied only on a full extract; a filtered extract restores a
subset and would be misrepresented by the whole tree's directory set.
## f8 — a container-registry mirror, end to end
The use case the shape section and `--format generic` were added for: pack what
`talosctl image cache-create` writes into ONE archive and restore it byte-exactly
on a fresh host.
```text
<root>/blob/<sha256-hex> 1 KB … 500 MB, ALREADY gzip/zstd, NO extension
<root>/manifests/<host>/<repo>/reference/<tag> small JSON
<root>/manifests/<host>/<repo>/digest/<sha256> small JSON
```
```bash
znippy compress --format generic -i /var/cache/mirror -o mirror
znippy decompress -i mirror.znippy -o /var/cache/mirror # byte-exact, dirs and modes included
```
`--format generic` (aliases `none`/`raw`/`plain`/`blob`, `type_id 26`) is the
handler that claims **nothing**: no extensions, no columns, `matches_path` always
false. The default `rust` also extracts nothing from such a tree — `matches_path`
is `ends_with(".crate")` — but it records `pkg_type = 1`, so `--type rust` and
every `RustView` consumer would report a registry mirror as crates.io content. A
`pkg_type` is a promise; the honest one here is none.
The blob filenames are bare hex, so the extension table (f5 rule 1) cannot reach
one and the **magic-byte probe** is the only leg that can. MEASURED 2026-08-24,
3975WX, 1.2 GB tree (12 × 100 MB gzip blobs + 2 000 manifests, output to NVMe):
| run | wall | user CPU | archive |
|---|---|---|---|
| probe active (default) | **0.54 s** | **0.43 s** | 1 200 803 631 B |
| `--no-skip` (zstd forced over already-gzip bytes) | 43.27 s | 322.40 s | 1 200 803 631 B |
80× the wall clock and **750× the CPU** for a **byte-identical** archive. Blobs are
content-addressed, so this is also the path where a single wrong byte breaks an
image pull with a digest mismatch and no other symptom — `diff -r` over the
restored 1.2 GB tree reports nothing, empty directories included.