# lgz — design
> **Class:** `design` · **Function:** `lgz` (multi-member gzip decode). Strict: the
> shape we ship, every claim carrying a real `file:symbol`. How to run it in
> [`lgz-guide.md`](lgz-guide.md); why gzip is only *partly* saturable in
> [`lgz-reasoning.md`](lgz-reasoning.md). The family view is
> [`decompressors-design.md`](decompressors-design.md); the engine is
> [`gatling-design.md`](gatling-design.md).
## What it is
A pure-Rust parallel decoder for the *stock* `.gz` format — anything `gzip`/`pigz`
wrote, decoded across every core. The CLI is `lgz/src/bin/lgz.rs`; the library twin
is `lgz::decompress_tar_gz_filter` (`lgz/src/lib.rs:63`) / `_capped`
(`lgz/src/lib.rs:76`) and the in-memory `decompress_gz` (`lgz/src/parallel.rs:108`).
gzip has **two** kinds of parallelism, and lgz picks the one the input actually
offers, per invocation:
## Path 1 — concatenated members (the fast, fully-parallel path)
`try_multimember_fastpath` (`lgz/src/bin/lgz.rs:200`). The common `.gz` from log
rotation, `cat *.gz`, `pigz --independent`, or a member-per-file `.tar.gz` is a run
of **fully independent** gzip members. lgz `mmap`s the file and scans for every
member start with `lgz::speculative::find_gzip_members_parallel`
(`lgz/src/speculative.rs:471`) — itself a `gatling_for_each` fan-out over the box
(header-shape + inflate probe per candidate). `< 2` members ⇒ not this path.
Each member's uncompressed length is its own 4-byte **ISIZE** trailer, so the exact
output byte-layout is known *without decoding*. From there the two output shapes
diverge:
- **`-o file`** → pre-size the output (`set_len(total)`), then
`gatling::parwrite::write_segments_positional` (`gatling/src/parwrite.rs:191`) fans
the members across the pool: each worker decodes one member into a **reused**
scratch buffer and immediately `pwrite`s it at its prefix-sum offset. No collector,
no serial concat — in-flight memory ≈ `workers × member_size`, never O(output).
The decode validates its length against ISIZE and fails the segment on mismatch.
- **stdout / pipe** (cannot seek) → `gatling::gatling::ordered::run_ordered_sink`
(`gatling/src/gatling_ordered.rs:113`) streams members to a worker pool; the
ordered sink writes each the instant it is next-in-order **while the other workers
keep decoding**. A `Mutex<Vec<Vec<u8>>>` **recycled buffer pool** (`lgz.rs:350`)
hands each drained, warm, ISIZE-presized buffer back to the next member — zero
steady-state allocation and no per-member page-fault storm (the old
`Vec::with_capacity`+drop churned 130 K+ minor faults on the kernel `mmap_lock`,
capping saturation at ~8.5/12).
## Path 2 — single member with flush boundaries (streaming)
`main`'s streaming pipeline (`lgz/src/bin/lgz.rs:420`). A single-member stream can
still be split *iff* the encoder inserted `Z_FULL_FLUSH` points (pigz does). A single
**dispatch thread** is the producer: it reads 200 MB chunks (`CHUNK_SIZE`), prepends
the undecoded **carry** into headroom, strips the gzip header (first chunk) and 8-byte
trailer (last chunk), and calls `split_chunk` (`lgz/src/chunk.rs:29`) to cut at flush
boundaries. It distinguishes the two flush kinds by the bytes after the boundary
(`lgz.rs:562`):
- **full-flush** (`00 00 00 FF FF` — empty stored block resets the LZ77 window) ⇒ the
segments are independent, queued one at a time to `run_ordered_sink`; N map workers
run `decode_segment_into_hint` (`lgz/src/chunk.rs:79`); the `WriteSink`
(`lgz.rs:62`) writes in strict stream order.
- **sync-flush** (back-references cross the boundary) ⇒ segments are *not*
independent, so the whole chunk decodes as **one** segment (still overlapping I/O
with the producer).
## Path 3 — no boundary at all (serial fallback, honest)
A plain single-member gzip (`gzip foo`) has **no** interior split point — one
continuous LZ77 window. When the producer finds no boundary in the whole file it sets
`fallback_flag` and `main` (`lgz.rs:666`) tries, in order:
`decode_concatenated_members` → `speculative_decode` (DEFLATE block-boundary guess,
**trailer-verified before a byte is emitted**, `lgz.rs:161`) → single-threaded
`flate2`. This path is **serial-by-format**, not a saturation bug (see
[`lgz-reasoning.md`](lgz-reasoning.md)).
## Correctness
Every emitted byte is checksum-verified: the multi-member `-o` path checks each
member's decoded length against ISIZE; the in-memory `decompress_gz` re-checks the
whole-output CRC32 + ISIZE (`lgz/src/parallel.rs:131`); the speculative path is
discarded unless `gzip_trailer_ok` passes.
## Whereabouts
| Thing | file:symbol |
|---|---|
| CLI entry / arg routing | `lgz/src/bin/lgz.rs:420` (`main`) |
| multi-member fast path | `lgz/src/bin/lgz.rs:200` (`try_multimember_fastpath`) |
| member scan (parallel) | `lgz/src/speculative.rs:471` (`find_gzip_members_parallel`) |
| flush split | `lgz/src/chunk.rs:29` (`split_chunk`) |
| per-segment decode | `lgz/src/chunk.rs:79` (`decode_segment_into_hint`) |
| ordered streaming sink | `gatling/src/gatling_ordered.rs:113` (`run_ordered_sink`) |
| positional pwrite | `gatling/src/parwrite.rs:191` (`write_segments_positional`) |
| speculative + trailer check | `lgz/src/speculative.rs:256`; `lgz/src/bin/lgz.rs:161` |
| library API | `lgz/src/lib.rs:63` / `:76`; `lgz/src/parallel.rs:108` |
## Saturation state (just-landed)
Multi-member `-o` decode reaches **~7.8/12 cores** on the quiet box (stdout can't be
offset-written, so it uses the ordered-sink path); single-member gzip is
serial-by-format and exempt. Honest bench numbers land in the
[`README.md`](README.md) A/B/C mashup on the next heavy-bench wave — this doc records
no fabricated `cores_busy`.