# znippy-zoomies — guide
> **Class:** `guide` · audience: a techie decoding through, or building on, zoomies.
> The *what* is in [`zoomies-design.md`](zoomies-design.md); the *why* in [`zoomies-reasoning.md`](zoomies-reasoning.md); the numbers in [`benchmarks.md`](benchmarks.md).
> This file is the workflow and the worked code.
## f0 — decode a stream, end to end (the whole workflow)
The zoomies rule you never break: **hand the decoder a borrowed `&[u8]`; never copy the input.**
The path from bytes to output:
1. **Read once into a slot** (or `mmap` the file). Do not clone.
2. **Split** at safe boundaries — `Z_FULL_FLUSH` marks for DEFLATE, BWT blocks for bzip2.
3. **Decode segments in parallel** through `linflate`, each into its region of one buffer.
4. **Stitch in order** and `set_len(total)`.
For a single file you call `linflate` directly; for a `Read` stream at scale you let `gatling` do
steps 1–4 and you only supply a `Codec` + `Sink`.
## linflate — decode into your own buffer
```rust
// Simple: one shot into a fresh Vec.
let out: Vec<u8> = linflate::inflate_to_vec(&compressed, expected_size)?;
// Zero-alloc: decode into a caller buffer. It MUST carry the headroom.
let mut buf = vec![0u8; expected_size + linflate::OVERWRITE_HEADROOM];
let n = linflate::inflate_into(&compressed, &mut buf)?; // n == bytes written
buf.truncate(n);
```
`OVERWRITE_HEADROOM` is not optional slack — SIMD match-copy writes past the last real byte, so an
output region that omits it returns `InflateError::OutputOverflow` (a clean error, never memory
corruption). Every region you decode into — including each segment's slot — must include it.
Mid-stream segment (does **not** need to end on `BFINAL=1`):
```rust
// prefix = the already-decoded tail that back-references may reach into.
let n = linflate::inflate_segment_with_prefix(&segment_bytes, &prefix, &mut region)?;
```
## the candidate-offset output model (B6) — parallel in one buffer
Use `ljar::chunk::decode_chunk_into` when you have many segments to inflate into a single output:
```rust
// One buffer, generous per-segment caps → candidate offsets (prefix sum of caps).
// All segments decode IN PLACE in parallel (rayon fork-join), then a forward-seek
// compaction slides each down to its true offset. actual <= cap ⇒ every move is
// leftward ⇒ nothing un-moved is clobbered. A segment that overflows its cap spills
// to an owned Vec and is rebuilt sequentially — output stays byte-identical.
ljar::chunk::decode_chunk_into(&segments, &mut output)?; // ljar/src/chunk.rs:173
```
Rule of thumb: make caps **generous** (over-estimate). The cost of a too-small cap is a rare spill +
sequential rebuild; the cost of a too-large cap is only transient address space in the one buffer.
## gatling — stream a `Read` source through N workers
Byte mode: workers emit `Vec<u8>`, the collector concatenates:
```rust
use zoomies::gatling::{self, Config, SlotFill, Codec, Sink};
struct Inflate; // your transform
impl Codec for Inflate { /* fn transform(&self, seg: &[u8]) -> Vec<u8> { … } */ }
let cfg = Config {
chunk_size: 64 << 20,
carry_headroom: 1 << 20, // room for a prepended carry across a split
ring_slots: n_workers + 2,
initial_carry: Vec::new(),
slot_fill: SlotFill::Big, // planet-scale: size once, reuse, no re-zero
};
gatling::run(reader, Inflate, my_sink, cfg)?;
```
Typed mode when workers produce records, not bytes (no reassembly — the split pre-cuts logical
segments and the collector just forwards each in order):
```rust
gatling::run_typed(reader, my_typed_codec, my_typed_sink, cfg)?; // src/gatling.rs:99
```
`SlotFill` choice: **`Incremental`** for small/partial inputs (grows a slot to bytes read);
**`Big`** for planet-scale (sizes each slot once to `carry_headroom + chunk_size`, reuses, never
re-zeroes). Keep heavy work *inside* the worker — boundary-find, decode, parse, encode, sort all ride
one unit with no barrier (the 1→32→1 principle). A reader or writer that does CPU work is your ceiling.
Async / I/O-bound (S3 multi-PUT and friends):
```rust
use zoomies::gatling::io; // src/gatling_io.rs — tokio, bytes::Bytes moved never cloned
io::run_ordered(jobs, /* max_in_flight = */ 32).await?;
```
## stree — id → position lookup
```rust
use zoomies::stree::STree64;
let tree = STree64::build(&sorted_keys); // keys: sorted i64
let pos = tree.find_exact(id); // serial, Option<usize>
// Batched + software-pipelined + prefetched — this is where the speedup lives:
let hits = tree.batch_stream::<16>(&query_ids); // src/stree.rs:773
// mmap-backed store (leaf layer is the mmap):
let hits = mmap_tree.lookup_batch_pipeline::<16>(&query_ids, &mmap); // :387
```
Build with `RUSTFLAGS="-C target-cpu=native"` — the AVX2 `count_lt` path roughly doubles batch
throughput over the portable default. Call `advise_hugepage` on large trees for the RAM-bound regime.
## running the benchmark suite
One suite, via nornir's `Bencher` trait — **not** `cargo bench` / criterion:
```bash
cargo run -p xtask --release --example nornir-bench # prints one BenchRun JSON line
NORNIR_BENCH_SKIP_HEAVY=1 cargo run -p xtask --release --example nornir-bench # skip heavy scales
NORNIR_BENCH_SKIP_PLANET=1 cargo run -p xtask --release --example nornir-bench # skip planet scale
```
Results are historized in the Iceberg warehouse by git SHA, gated at **>10% regression**, and
rendered into the README. See [`benchmarks.md`](benchmarks.md) for the current numbers.