# Decompressors — design
> **Class:** `design` · **Function:** `decompressors` (the parallel decode family).
> Strict: chosen shape only. The bake-off numbers are in [`benchmarks.md`](benchmarks.md);
> how to run them in [`decompressors-guide.md`](decompressors-guide.md); the "why not
> keep four hand-rolled pipelines" road-not-taken in [`reasoning.md`](reasoning.md) and
> [`chunk_revolver-unsorted.md`](chunk_revolver-unsorted.md).
## Per-decompressor deep-dives (exact dataflow + skade-style SVG each)
Each decoder is documented on its own, with a hand-authored flow diagram of its
**exact** dataflow (input → boundary scan → self-dispatch workers → ordered/positional
sink → output):
- **lgz** — [`lgz-design.md`](lgz-design.md) · [guide](lgz-guide.md) · [reasoning](lgz-reasoning.md)
- **lbunzip2** (CLI) — [`lbunzip2-design.md`](lbunzip2-design.md) · [guide](lbunzip2-guide.md) · [reasoning](lbunzip2-reasoning.md)
- **lbzip2 library readers** — [`lbzip2-readers-design.md`](lbzip2-readers-design.md) · [guide](lbzip2-readers-guide.md) · [reasoning](lbzip2-readers-reasoning.md)
- **ljar** — [`ljar-design.md`](ljar-design.md) · [guide](ljar-guide.md) · [reasoning](ljar-reasoning.md)
- **lzip-parallel** — [`lzip-parallel-design.md`](lzip-parallel-design.md) · [guide](lzip-parallel-guide.md) · [reasoning](lzip-parallel-reasoning.md)
- shared **gatling** engines — [`gatling-design.md`](gatling-design.md); **scanners** (json/xml/vtd) — [`scanners-design.md`](scanners-design.md); **osm-katana** convert — [`osm-katana-design.md`](osm-katana-design.md).
## What it is
Five workspace member crates (`Cargo.toml:17`), each a **pure-Rust parallel decompressor**
for a *stock* archive format — the format every existing tool already reads, decoded across
**every core** where the legacy serial tools leave the box idle. These are the subjects of
the bake-off.
| Crate | Version | Format | Core idea |
|---|---|---|---|
| `lgz` | 0.3.6 | multi-member gzip (`.gz`) | split at member boundaries, DEFLATE-decode members in parallel |
| `lbzip2` | 0.5.4 | bzip2 (`.bz2`) | scan block-start bit patterns, decode independent BWT blocks in parallel |
| `linflate` | 0.1.4 | raw DEFLATE | SIMD match-copy, branchless refill, segment-aware — the shared decode core |
| `ljar` | 0.2.3 | JAR (`.jar`) | Central-Directory scan → parallel per-entry DEFLATE |
| `lzip-parallel` | 0.2.4 | ZIP (`.zip`) | near-verbatim twin of `ljar` (JAR ⊂ ZIP) |
## The shared decode shape
Each format is a set of **independently decodable segments** (gzip members, bzip2 blocks,
ZIP/JAR entries). The decode is the same three-move gatling pipeline:
1. **Scan** the compressed bytes for segment boundaries (`deflate_scan.rs` /
`block_scan.rs` / `central_dir.rs`) — cheaply, up front.
2. **Fan** the segments across N workers, each decoding one segment to completion (the
`linflate` fast loop for DEFLATE formats; the BWT/MTF/Huffman path for bzip2).
3. **Collect** the outputs in order and write them out.
`linflate` is the reused DEFLATE core underneath `lgz`, `ljar`, and `lzip-parallel`:
`inflate_segment` / `inflate_segment_with_prefix` / `inflate_to_vec` (`linflate/src/lib.rs`)
decode one segment given its (optional) back-reference prefix window.
## Public library surfaces
Each crate ships a library twin of its CLI (so a decode is callable, not just a binary):
- `lgz::decompress_tar_gz_filter(data, needle)` / `_capped(…)` — decode + filter members.
- `ljar::decompress_jar(data)` / `decompress_jar_filter(data, needle)` /
`decompress_jar_stream(reader)` → `Vec<JarEntry>`.
- `lzip-parallel::decompress_zip(data)` and the `_filter` / `_filter_set` /
`_filter_suffixes` / `_stream` family → `Vec<ZipEntry>`.
- `linflate::inflate_into` / `inflate_segment*` / `inflate_to_vec` — the raw DEFLATE core.
All five also expose the crate-local `functional_status(component, check, ok, detail)`
introspection shim (mirrors `lib.rs:35`), so `nornir test --features testmatrix` sees each
decode.
## Parallelism substrate (the no-rayon law, workspace-wide)
All five crates are now **rayon-free** and fan out through the one shared engine:
- `lgz`, `lbzip2`, `ljar`, `lzip-parallel` route their per-segment decode across
`gatling::gatling_forkjoin::gatling_for_each` (e.g. `ljar/src/reader.rs:122`,
`lzip-parallel/src/reader.rs:172`); `linflate` is a pure single-buffer decode library
with no fan-out of its own.
- `ljar` and `lzip-parallel` were **ported off** their old `OnceLock<rayon::ThreadPool>`
singletons (rayon deps dropped); `lgz`/`lbzip2` folded their private scoped-thread clones
onto the shared engine. `tests/rayon_free_law.rs` now walks every workspace member and
bans private pools, so no crate can quietly reintroduce one. The history of how the codecs
were rayon-from-birth and how the law got fixed is in [`reasoning.md`](reasoning.md) and
[`gatling-reasoning.md`](gatling-reasoning.md).
**The writer tail (landed).** Each format's decode (the middle "Fan" move) is
already parallel; what left the DEFLATE codecs writer-bound (lgz ~3.1, ljar ~3.5, lzip ~4.0
of 12 cores, vs lbz2 ~10.4) was the **single ordered collector + writer** at the tail — not
slot starvation (widening `ring_slots` is refuted). The fix, now on `master`, is the `gatling::parwrite`
layer (`gatling/src/parwrite.rs`): each worker `pread`s and
writes its **own** output — independent files (`extract_entries_unordered`, ZIP/JAR) or
offset-`pwrite`d regions of one pre-sized stream (`write_segments_positional` /
`write_slices_positional`). Measured cores_busy: ljar/lzip **→ ~11.5/12**, lgz **→ 7.8/12**
(offset-pwrite on `-o` files; stdout can't be offset-written), lbz2 held ~10.4. See
[`gatling-design.md`](gatling-design.md) and [`gatling-reasoning.md`](gatling-reasoning.md).