# Workspace perf audit — zero-copy deviants & speed blunders (2026-06-11)
<!-- nornir:class: history -->
Six parallel source audits across the decompressors (`linflate`, `lgz`,
`lbzip2`, `ljar`, `lzip-parallel`) and the core (de)compress pipeline
(`znippy-compress`, `znippy-common`, `gatling`). Triggered after the stree fix;
each agent specifically hunted the stree antipattern (per-call
`is_x86_feature_detected!` + non-inlinable `#[target_feature]`). Findings are
ranked; every item has file:line in the agent transcripts.
## P0 — biggest levers (cross-cutting)
1. **The whole build runs at SSE2 — AVX2 code paths are dead.**
`.cargo/config.toml` sets only `target-dir`; no `target-cpu`/`target-feature`.
`linflate/src/copy.rs:117` `copy_chunks::<32>` is a plain `#[inline(always)]`
generic that inlines into the non-feature `inflate_fast`, so LLVM lowers the
32-byte match copy to **2× `movdqu` (SSE2)**, not `vmovdqu ymm` — the module
doc's "N=32 → ymm" claim is false for the shipped build. This is the stree
bug's whole *class*, workspace-wide. Fix: build local binaries with
`target-cpu=x86-64-v3` (free ~1.5–2× as proven for stree), **or** add AVX2
`#[target_feature]` variants for the hot copy (mirroring the existing
`copy_chunks_avx512`). Same reproducibility caveat as stree — don't commit a
global flag; set it for the local planet/release build.
2. **stree antipattern replicated in the flush scanners.**
- `lgz/src/deflate_scan.rs:30-46` (`find_next_flush`) — `is_x86_feature_detected!`
runs **per call**, and `find_next_valid_flush` (`:165`) calls it in a
`while let` skip loop once per false-positive candidate (~thousands per
200 MB chunk). Genuine hot loop. Fix: fold the candidate-skip loop *inside*
one `#[target_feature]` AVX2 context (resolve detection once), exactly the
stree fix.
- `ljar/src/deflate_scan.rs:37-48` — per-call in `find_all_flushes` (per
segment, benchmarked in `inflate_bench.rs`). Same fix.
- `lbzip2`, `lzip-parallel`, and the compress pipeline are **clean** — their
SIMD detection is already hoisted (or compile-time `_mm_prefetch`).
## P1 — Law-1 violations on common paths
3. **STORE entries do a full `.to_vec()`** — `ljar/src/entry.rs:73` and
`lzip-parallel/src/entry.rs:73`. Already-compressed entries (`.png`, nested
`.jar`, `.so`) are memcpy'd from a buffer that outlives the call. Should
return `Cow::Borrowed`. This is GB/s of needless copy on store-heavy archives.
4. **slot_packer skip-path memcpy** — `znippy-compress/src/slot_packer.rs:557-590`.
On the `skip`/incompressible branch (the *full-I/O-speed* common case for
`.jar/.gz/.parquet`) the worker copies the whole slot into an owned `Vec` just
to release the slot early. `stream_packer.rs::Payload::Skip` already shows the
zero-copy fix (carry slot id, `pwrite` from source, release after). Biggest
win in the compress path.
5. **Streaming readers double-clone names + per-entry `vec![0u8; n]`** —
`ljar/src/reader.rs:96-126`, `lzip-parallel/src/reader.rs:153-164`. Name
cloned twice (`std::mem::take` instead); read buffer zero-filled then
overwritten (uninit `set_len` like `chunk.rs` already does); not reused across
batches.
## P2 — allocation churn / double copies
6. **lgz `decode_chunk` double-copy** — `lgz/src/chunk.rs:159-167`: segments into
`Vec<Vec<u8>>`, then every byte memcpy'd again into an un-presized `result`.
7. **lbzip2 `decompress_parallel` per-block `Vec` + ~1 GB final concat** —
`lbzip2/src/parallel.rs:67-70`; route through the segment path (`chunk.rs`
already decodes into a pre-sized output). NB the bench measures *this* slower
path, possibly not the production `chunk` path.
8. **lbzip2 `par_map` per-slot `Mutex`** — `lbzip2/src/par.rs:42`: `n` Mutex
allocs + `n` locks for a disjoint-index scatter; use `UnsafeCell`/raw-ptr
scatter (zero sync).
9. **stream_packer unbounded channel** — `znippy-compress/src/stream_packer.rs:90`:
`unbounded()` of `ArchiveEntry{data: Vec<u8>}` → no backpressure, unbounded
in-flight payloads. Make it `bounded(workers·k)`.
10. **io_uring per-batch allocations** — `slot_packer.rs` `run_small_pass`:
`cstrings`/`fds`/`offsets`/`read_results` reallocated per 128-file batch, plus
`(0..n).collect::<Vec<_>>().chunks(256)` (lines 411/447) allocates a throwaway
index Vec just to chunk a range. Hoist + reuse; use `.step_by(256)`.
11. **Big-file metadata re-reads whole file** — `slot_packer.rs:289`
`std::fs::read(path)` per plugin-matched big file, after it already streamed
through the slot pool. Extract from header bytes in-slot, or reuse one buffer.
12. **Per-segment `vec![0u8; estimate]` zero-fill + `.to_vec()` result** —
`lgz/src/speculative.rs:199-214`, `lzip-parallel/src/chunk.rs` `decode_segment`.
Reuse the uninit `with_capacity`+`set_len` pattern already in `lgz/chunk.rs`.
## P3 — algorithmic / micro
13. **lbzip2 MTF byte-at-a-time** — `lbzip2/src/mtf.rs:39` `copy_within(..idx,1)`
shifts up to ~255 bytes per symbol (~900K symbols/block): the dominant bzip2
decode cost. Blocked/SIMD MTF (libbzip2 MTFA/MTFV, or register-shift for
`idx<16`) is the biggest single decode win.
14. **lbzip2 `tt.push`/`resize` per symbol** — `block.rs:204/219`: raw writes into
the pre-sized buffer + one hoisted bound check (RLE2 stage already does this).
15. **linflate micro:** 14 KB fixed-table memcpy per fixed block
(`fixed.rs:55`); dist 8..31 scalar (could AVX2-tile, `copy.rs:55`); dead code
`extract_var` (BMI2, unused) + `compute_code_for_sym`.
16. **ljar central-dir `String` per entry even when filtered out** —
`central_dir.rs:140`: allocate-then-filter; borrow `&str` from `cd_data`.
## Verified CLEAN (don't touch)
- Core zero-copy is sound: the coalescing `ChunkRevolver` (`znippy-zoomies`) sends
slot ids/ptrs not payloads; Gatling typed mode; `decompress.rs` (reused
buffers, positioned I/O, lock-free cursor). Laws 1/3/4 hold there.
- blake3 one-shot over a contiguous slice (SIMD dispatched once) — incremental
`.update()` would be no faster.
- lbzip2 BWT inverse is O(n) counting sort (not O(n log n)); RLE2 raw-ptr +
prefetch; bitreaders are 64-bit branchless refill, `#[inline(always)]`.
- No stree antipattern in lbzip2 / lzip-parallel / compress / codec.
## Benchmark coverage
| crate | nornir release gate | per-crate criterion + history |
|---|---|---|
| linflate | ✓ guava + scala (InflateCmp) | **✗ none** |
| ljar | ✓ guava + scala | ✓ inflate/jar_throughput/chunk_into |
| lzip-parallel | ✓ guava + scala | ✓ zip_throughput (A/B vs `zip`) |
| lbzip2 | **guava only** | ✓ decompress_bench (but tests the slower parallel path) |
| lgz | **guava only** | **✗ none** |
Gaps to close:
- **nornir:** add `lbzip2_scala`, `lgz_scala` (done in this commit).
- **per-crate:** `lgz` and `linflate` have **no** local bench/history — the two
crates with the worst SIMD findings (P0.1, P0.2) have nothing to A/B against.
Per the project's benchmark-first law, add these *before* the P0/P1 fixes.
- No bench exercises STORE-heavy archives (P1.3) or the streaming readers (P1.5).
## Results (2026-06-11) — measured, not assumed
Benchmark-first paid off and overturned the audit's top theory.
- **P0.1 (linflate AVX2) — REJECTED by benchmark.** Added a criterion bench
(`benches/inflate.rs`: text/binary/rle corpora) and tried the obvious fix —
multiversion `inflate_fast` so the whole decode loop runs in a
`#[target_feature(enable="avx2")]` context (making the inlined `copy_chunks::<32>`
emit `ymm`). Controlled quiet A/B: it **regressed** every corpus (reverting it
was +5.3% text, +4.0% rle, +1.1% binary). Reason: real DEFLATE decode is
**Huffman-bit-decode-bound, not match-copy-bound** — forcing the scalar
bit-decode into an AVX2 codegen context costs more (register pressure /
instruction selection) than the wider copy saves. The audit's "biggest lever"
was theoretical; the workload disagrees. **Reverted.** linflate keeps the new
bench + dead-code cleanup; its real win is a `target-cpu=native` build (general
codegen), documented, not forced. Lesson: SIMD-everything ≠ faster; measure.
- **P0.2 (lgz/ljar scanner dispatch)** — different shape (removing per-call
*overhead* in a hot loop, like the stree fix, not wrapping a loop in AVX2) —
implemented, benched separately.
- **lbzip2** (P3.13/14, P2.7/8): MTF small-index fast path (u128 shift, with an
exhaustive equivalence test), `decompress_parallel` routed through the segment
path (no per-block `Vec`, concat N→~128), lock-free scatter (no per-item
Mutex), raw `tt` writes. 26 tests + byte-exact-vs-libbz2 green; no regression
on the Huffman-bound test file, wins scale with block count / MTF-heavy data.
- **znippy-compress** (P1.4, P2.9/10/11): zero-copy `Payload::Slot` skip path
(writer pwrites from the slot, releases after — kills the stored-raw memcpy),
bounded entry channel, hoisted/reused io_uring batch buffers + `step_by`
instead of `(0..n).collect().chunks()`, reused big-file metadata buffer. 35
integration tests green; compress→decompress→verify roundtrip byte-exact.
- **lgz** (P0.2, P2.6/12): scanner dispatch hoisted to a `OnceLock` fn-pointer
with the candidate-skip loop folded into the `target_feature` variants (no
per-call detection); presized `decode_chunk`; uninit speculative buffers. 22
tests green, decoded output byte-identical.
- **ljar** (P0.2, P1.3/5): scanner dispatch hoisted; STORE entries borrow via
`Cow` on the batch-write path; reader name moved once (`into_par_iter`) +
uninit read buffer. 57 tests green. The scanner change A/B'd **neutral**
(±0.5%, within noise — the bench corpora aren't flush-dense) — kept because it
removes the antipattern with no regression and helps flush-dense streams.
- **lzip-parallel** (P1.3/5): STORE `Cow` borrow + direct `write_all` on the
batch path; uninit read buffer + single name move. 56 tests green.
- **linflate micro:** dead `compute_code_for_sym` made `#[cfg(test)]` (was a
test-only helper triggering a dead-code warning).
**Build lever (P0.1, the real linflate/SIMD win):** since in-code AVX2 dispatch
regressed, the win is a native-ISA build — proven ~1.5–2× for stree, and it lets
LLVM widen `copy_chunks` etc. for free. **Now committed** in `.cargo/config.toml`
as `rustflags = ["-C","target-cpu=native"]`. Safe because a from-source workspace
build runs on the machine it was built on (build==run), and `.cargo/config.toml`
is not part of any published crate package — crates.io consumers build with their
own flags, unaffected. Override per-invocation with `RUSTFLAGS=` for a portable
binary.
All six crates + the `znippy-tests` integration suite + `znippy-cli` build green.
No version bumps yet (publishing the changed decompressor crates is a separate
gated step — the nornir release bench can't run while the sibling tree is broken).