# lgz — reasoning
> **Class:** `reasoning` · **Function:** `lgz`. Why the engine is shaped this way and
> where gzip is honestly *not* saturable. The implemented shape is in
> [`lgz-design.md`](lgz-design.md).
## Why gzip is only partly a parallel problem
DEFLATE is a single LZ77 back-reference window that flows the whole stream. A byte in
the middle can reference any of the previous 32 KiB, which itself references earlier
bytes, transitively back to byte zero. So a *plain* single-member gzip (`gzip foo`)
has **no interior point you can start decoding from** without first decoding
everything before it. That is a property of the format, not a defect in lgz. The
honest verdict: single-member gzip decode is **serial-by-format** and lgz's Path 3
fallback (`lgz/src/bin/lgz.rs:666`) accepts it — one core, `flate2`, correct output.
Two escape hatches exist, and lgz takes both when the input offers them:
1. **Concatenated members.** `cat a.gz b.gz`, log rotation, `pigz --independent`, and
member-per-file `.tar.gz` produce *independent* members — each its own window,
each its own CRC/ISIZE trailer. That is genuinely embarrassingly parallel, and it
is the common shape in practice, so it is Path 1 (the fast path).
2. **Full-flush points.** `pigz` inserts `Z_FULL_FLUSH` markers that emit an empty
stored block, resetting the window. Segments between them are independent. lgz
detects full-flush vs sync-flush by the post-boundary bytes (`lgz.rs:562`) and only
splits on full-flush — splitting a sync-flush stream would decode garbage because
the back-references cross the boundary.
## Why the writer had to change (the ~8.5 → ~11.5 story)
The middle "decode" move was already parallel; what pinned lgz below its rivals was
the **tail**. The old multi-member path built a `Vec<Vec<u8>>` of every decoded member
(O(output) resident — a peak-RSS regression) and drained it through one serial writer.
Two fixes landed:
- **`-o` files** get `write_segments_positional` (`gatling/src/parwrite.rs:191`): each
worker `pwrite`s its own member at a prefix-sum offset into a pre-sized file. No
collector on the critical path. This is why `-o` beats stdout.
- **stdout** can't be offset-written, so it uses `run_ordered_sink` plus a **recycled
warm-buffer pool**. The subtle win was not the ordering — it was killing the
per-member `Vec::with_capacity`→drop→`munmap` churn that serialised every worker on
the kernel `mmap_lock` and capped saturation at ~8.5/12 regardless of thread count
(`lgz.rs:340`). Recycling ISIZE-presized buffers gives a warm, already-faulted-in
buffer per member, so after warm-up there are no output allocations in the hot loop.
Result: `-o` multi-member ~7.8/12 (offset-pwrite); stdout held via the ordered path.
The remaining gap to a pure `pwrite` path is stdout's inability to seek — an OS
constraint, not a scheduling one.
## Why not a bespoke thread pool
Earlier lgz carried its own scoped-thread clone. The constellation's no-rayon law
(`tests/rayon_free_law.rs`) folds every codec onto the one shared `gatling` engine, so
the ordering, backpressure, and permit-bounded memory are proven once and reused. See
[`gatling-reasoning.md`](gatling-reasoning.md).
## Honest numbers
No `cores_busy` is hard-coded here. The real figures land in the
[`README.md`](README.md) A/B/C mashup on the quiet-box heavy-bench wave.