# The compress engine — reasoning
> **Class:** `reasoning`. This file holds the *why*, the trade-offs, and **every design we
> rejected**. The descriptive shape we actually run is in [`compress-design.md`](compress-design.md),
> which by rule mentions none of this. If you want to know "why not X?" — X is here.
## Why "Gatling" at all — the 1→N→1 rule
The whole engine exists to keep NVMe saturated while N cores compress. The shape that does it is
**1 → N → 1**:
- **1 reader, read only.** It does *no* CPU work: splitting an input into slices is a cheap
O(n_workers) candidate-offset division, not a compression pass. The instant the reader starts
doing heavy work, it becomes the serial ceiling and every worker starves behind it.
- **N workers carry one unit through ALL heavy stages.** A single slice goes through blake3 *and*
compress on the same worker, with **no barrier**, and workers **self-dispatch** the next slice.
- **1 writer, stitch/drain in order.** The finalizer only orders and drains; it does no
compression.
The law behind it: **heavy work on either 1-end becomes the serial ceiling.** Both 1-ends must
stay trivial; all weight lives on the N-end. This is the single rule to remember.
## The "blame Gatling" trap
There was a moment when a Gatling path ran **slower** than plain rayon, and the tempting
conclusion was "Gatling is the wrong model, rayon is faster." That was wrong. The real cause was
**heavy work sitting on a 1-end** — a stage that belonged on the N workers had crept onto the
reader/writer, so the serial ceiling dropped. Moving the weight back to the N-end fixed it. The
lesson is the 1→N→1 rule, **not** "rayon is faster." Never blame the model for a misplaced stage.
## Rejected: rayon fork-join with a per-slot-completion BARRIER (the "lbzip2-v1 bug")
The obvious parallel shape is `rayon`: fan a batch of slices out, `join`, then release the batch.
We **rejected** it because the join is a **barrier**: the slot cannot recycle until its *slowest*
slice finishes, so one slow slice stalls the whole batch and the other cores sit idle waiting.
This is the classic lbzip2-v1 failure. The fix is the **no-barrier slot ring**: one global slice
queue, workers self-dispatch, and a slot returns (`Returner`) only after its own last slice — a
slow slice in slot A never blocks slot B's slices. The barrier is exactly the thing the ring
removes.
## Rejected: copying `Vec<u8>` payloads across channels
The tidy dataflow is "worker compresses, sends the `Vec<u8>` payload down a channel to a single
writer." We **rejected** it: on ~7 GB/s NVMe, that extra payload copy **halves throughput** —
the memcpy bandwidth becomes the bottleneck before the disk does. Instead the **channel carries
slot indices only** (law G-1), and each worker `pwrite`s its *own* payload at an `AtomicU64`
cursor. Zero payload copies between compress and disk. The finalizer likewise gets `ChunkMeta`
only, never bytes.
## Rejected: per-file open/read syscalls
Reading each input file with its own `open`/`read` pair means a syscall storm on a tree of many
small files — the reader (a 1-end) spends all its time in the kernel and starves the workers. We
**rejected** it for the io_uring **128-files-per-batch** submission: one batched submission
amortizes the per-file syscall overhead away, keeping the reader trivial as the 1→N→1 rule
demands. The non-io_uring fallback exists for portability and is held **byte-identical** by
`fallback_byte_identical_to_io_uring` — the reader implementation must never change the bytes.
## Why blake3 over the ORIGINAL bytes, ordered by chunk_seq
Integrity is hashed over the **source** bytes, not the compressed frame: a checksum must verify
"is this the file the user gave us," independent of which codec or level produced the frame.
And checksums are bound to **`chunk_seq` order, not thread identity** (law G-3) — so a 4-core run
and a 64-core run of the same input produce the *same* checksums and the *same* index. If the
hash depended on which worker happened to grab a slice, archives would be non-reproducible across
machines. Determinism is the price of admission for content-addressing and dedup.
## Why store-if-incompressible, and why `no_skip` exists
Compressing an already-compressed input (`.jar`/`.gz`/`.crate`/`.png`/`.parquet`) burns CPU to
produce a frame that is **larger** than the input. So the skip path stores those raw, and — as a
backstop for anything the extension list misses — **any** slice whose frame ends up ≥ its input
is stored raw too, with the index `compressed` flag set false. `no_skip` exists for the cases
where you want a guaranteed-uniform pipeline (benchmarks, forcing the codec, verifying the store
path is never taken): it disables the extension skip and pushes every slice through the codec.