# znippy perf — batch query + ordered bulk decompress (future)
<!-- nornir:class: reasoning -->
> Status: **IDEAS / future perf work, not scheduled.** Two API-shape wins on top
> of the existing engine (already among the fastest); both are measurable with the
> holger bench harness before any commit. Grounded in the v0.7 multi-index +
> content-addressed blobs + parallel zstd + Arrow-IPC index structure.
## A. Batch query — `get_many(&[coord]) -> Vec<Option<Package>>`
One-at-a-time `extract_file` / `view.get` structurally forgoes four wins a batch
unlocks (it's not just call-overhead amortization):
1. **Random I/O → sequential.** Resolve all N coords → `FileLoc`s, **sort by
`blob_offset`**, issue coalesced/sequential reads (or `readv`) instead of N
scattered `pread`s. Dominant cost on **cold** archives (USB/optical in airgap)
— random→sequential is the big multiplier there, before decompress.
2. **Decompress-once on shared blobs.** Content-addressed ⇒ identical bytes = one
blob. A dep tree repeating a transitive crate, or `pom`+`jar` sharing content,
today decompresses the shared blob once *per caller*; a batch dedups by blob id
and decompresses **once**, fanning out.
3. **Cross-file parallelism.** Within-file chunk decompress is already parallel;
*across* files the per-call API serializes (caller awaits each). A batch feeds
the whole work-list to rayon → saturates cores even for many-small-files.
4. **One index pass.** Project coord columns once for the batch, not N borrows.
**Use case:** dependency-tree resolution / holger serving a client's whole
resolution set (N-at-once). API: `get_many` on `ZnippyReader` + each typed view,
keeping the lazy split (batch-resolve handles cheap; batch-materialize bytes = the
win). **Keep single `get`** — batch is the cold/many path; N=1 or page-cached is a
wash.
## B. Ordered bulk decompress (export / seal / full extract)
Single-file is already optimal; this is for **whole-archive** decompression.
1. **Columnar SIMD index scan.** Index is Arrow IPC → build the work-list (total
output size, offset sort, pkg_type filter) as a vectorized pass over contiguous
`offset`/`size` columns, not row-by-row. (See `znippy-arrowipc-analysis.md`.)
2. **Read in `blob_offset` order; partition contiguously per core.** Avoid
*striping* work across cores (→ random I/O per core). Give each core a
**contiguous offset range** → sequential reads + per-core L2/L3 stays warm.
Sequential-per-core beats striped at the same core count.
3. **Warm zstd window + buffer reuse.** Decompress consecutive blobs (and a file's
chunks in `chunk_seq` order) back-to-back → reuse output buffers, keep the
decode working set hot; interleaving files thrashes.
**Rule:** let the Arrow columns drive a sorted, contiguously-partitioned,
sequential plan — not directory/coord order.
## De-risk before committing
Both measurable with the existing bench harness (`holger/bench-scenarios` +
`xtask/examples/nornir-bench.rs`). Add two scenarios on a `huge`-scale archive:
- `get_many` batch-read vs N× single, **cold + warm** (drop page cache between).
- full-extract `ordered` vs `naive` mode.
First check what `archive.rs:154`'s loop already does (within-file parallelism
likely exists; cross-file batching + bulk-ordering likely don't). Numbers first.