Gunnar

nordisk/znippy

.nornir/decompress-design.md

# znippy decompress — design

> **Class:** `design` · **Function axis:** `f0` (the restore) + `f1` worker engine · `f2` single-file read · `f3` safety + CLI.
> Strict: this file states *only the shape we run*. Every *why*, every alternative, lives in [`decompress-reasoning.md`](decompress-reasoning.md); the how-to is in [`decompress-guide.md`](decompress-guide.md). For the index this reads against, see [`index-design.md`](index-design.md).

![The archive restore pipeline](assets/archive_pipeline.svg)

## f0 — the restore (the whole)

A znippy archive is a **content-addressed, seekable** blob: every chunk's byte position
(`blob_offset`, `blob_size`), its output position (`fdata_offset`), its codec flag
(`compressed`) and its `blake3` checksum already live in the index. Restore therefore is not
a *stream* — it is **pure random access**. There is **no reader thread, no channel, no slot
pool**: `N` workers share one atomic row cursor and each pulls its own bytes.

Full restore: `znippy-common/src/decompress.rs:65 decompress_archive(...)`; selective restore
(by `pkg_type` / `repo` sub-index) `:76 decompress_archive_filtered(...)`. Both re-exported at
the top level as `znippy-decompress/src/unpacker.rs:5 decompress_archive` / `:11
decompress_archive_filtered`.

Three laws govern the restore; a run is conformant iff all hold:

| law | statement | enforced by |
|---|---|---|
| **L1 — random-access input** | input is never streamed; each chunk is `pread` from its `blob_offset`. Parallel `pread` gives the NVMe real queue depth | positioned `read_exact_at` (`decompress.rs:225`) |
| **L2 — no hot-path alloc** | every worker reuses one read buffer + one output buffer across all its rows | per-worker `read_buf` / `out_buf` reuse (`decompress.rs`) |
| **L3 — deterministic, core-count-independent** | every chunk's checksum is bound to its `chunk_seq`; verification never depends on worker count or completion order | `blake3::hash(out)` vs per-chunk checksum (`decompress.rs:250`) |

**L3 is the load-bearing one:** because a checksum is bound to `chunk_seq` (not to a worker or
an arrival slot), the result is byte-identical whether you decompress on 1 core or 128, and a
corrupt archive is *detected* rather than silently rewritten.

**Reuse map:** decompress depends only on the index reader (`index.rs`), the codec
(`codec.rs`), and `blake3`. It is used by the `znippy` CLI (`decompress` / `get` / `verify`)
and by any caller that links `znippy-decompress`.

## f1 — the worker engine

`decompress_archive_filtered` (`decompress.rs:76`) opens the archive once as a shared
`Arc<File>`, builds the sorted work list from the index, and spawns `N` workers over one
`Arc<AtomicUsize>` row cursor (`decompress.rs:162`). Each worker loops:

```text
row = cursor.fetch_add(1, Relaxed)          // decompress.rs:194  claim next row, lock-free
if row >= total { break }
blob_offset, blob_size, fdata_offset,
compressed, uncompressed_size, checksum  ←  index columns for `row`
pread(blob_offset, blob_size) → read_buf     // decompress.rs:225  positioned, no shared seek
out = compressed ? decompress_into(read_buf, out_buf)   // codec.rs:66  OpenZL, buffer reuse
              : read_buf                                //          raw copy when not compressed
assert blake3::hash(out) == checksum         // decompress.rs:250  L3
pwrite(out, fdata_offset)                    // decompress.rs:269  positioned write, write_all_at
```

Rows that share a `relative_path` share one `Arc<File>` output handle; `pwrite` is positioned,
so ordering between workers is irrelevant. No worker ever waits on another — the only shared
mutable state is the atomic cursor. Codec: `znippy-common/src/codec.rs:66 decompress_into`
decompresses into a caller-owned `Vec` (buffer reuse, no per-chunk allocation).

## f2 — single-file random read

To pull *one* file without touching the rest of the archive:
`znippy-common/src/decompress.rs:318 get_file(archive_path, target)`. It resolves the target's
contiguous chunk run through `znippy-common/src/index.rs:736 locate_file` (see
[`index-design.md`](index-design.md) `f2` for the three-tier resolver), then for each
`ChunkLoc`: `pread(blob_offset)` → `decompress_into` (or raw copy when `compressed` is false)
→ `blake3`-verify against the per-chunk checksum → place the bytes at `fdata_offset`. Returns
the reassembled file bytes. Same L1/L3 discipline as f1, single-threaded.

## f3 — safety and the CLI surface

Every output path is passed through `safe_output_path(out_dir, rel_path)` (`decompress.rs:27`)
**before** any write: it rejects zip-slip (`../`) and absolute paths, so a hostile index can
never escape `out_dir`. Guaranteed by `safe_output_path_rejects_zip_slip_and_absolute_paths`
and `decompress_corrupt_archive_errors_not_panics` (a corrupt archive errors, never panics).

| command | entry | does |
|---|---|---|
| `znippy decompress -i <archive> -o <dir> [--type maven\|rust\|python] [--repo NAME]` | `znippy-cli/src/lib.rs:379 Commands::Decompress` | full or filtered restore (f0/f1) |
| `znippy get -i <archive> -p <relative/path> [-o out]` (omit `-o` ⇒ stdout) | `:420 Commands::Get` | single-file read (f2) |
| `znippy verify -i <archive>` | `:434 Commands::Verify` | decompress-and-verify every chunk, write nothing |

`--type` / `--repo` narrow the restore to a data sub-index; an empty filter equals full
restore. `verify` is exactly f1 with the `pwrite` suppressed — it proves every checksum without
producing files.