> **RULE: Never add Co-Authored-By or any AI attribution to commits. Ever.**
# znippy
Arrow IPC archive format with parallel compress/decompress, package-aware metadata extraction, and random-access reads.
Repository: https://git.gunnar.rs/nordisk/znippy
---
## Workspace layout
```
znippy-common/ trait definitions, plugin API, shared types, index schema
znippy-compress/ write path: read → plugin extract → chunk → compress → write
znippy-decompress/ read path: decompress → reconstruct files
znippy/ THE binary `znippy` — a 3-line wrapper over znippy_cli::run()
znippy-cli/ LIBRARY holding all the CLI code (handlers.rs = handler register).
Its duplicate `znippy-cli` binary was removed 2026-08-03: it was a
byte-identical copy of znippy/src/main.rs. Versions move in lockstep
with znippy/ (tests/tests/build_variants.rs enforces it).
plugins/ EVERY handler crate, split by the target it is built for.
A plugin crate lives in one of these two directories and
nowhere else; see plugins/README.md.
plugins/native/znippy-plugin-python/ wheel/sdist metadata — rlib+cdylib [type_id 2]
plugins/native/znippy-plugin-maven/ JAR/pom.xml metadata — rlib+cdylib [type_id 3]
plugins/native/znippy-plugin-media/ image/audio/video metadata — rlib [type_id 25]
plugins/native/znippy-plugin-skidbladnir/ airgap deploy-bundle members [type_id 40]
plugins/native/znippy-plugin-rust-toolchain/ airgap rust-dev bundle members[type_id 41]
plugins/native/znippy-plugin-git/ git objects keyed by oid hex — rlib [type_id 42]
carries the reserved __gunnar_oid__ / __gunnar_graph__ /
__gunnar_reach__ / __gunnar_refs__ / __gunnar_secrets__ sections
plugins/wasm/znippy-plugin-maven-wasm/ THE wasm example: the maven handler
compiled to wasm32-unknown-unknown and loaded through
wasm_loader.rs. Holds no maven logic — it depends on
plugins/native/znippy-plugin-maven and is only the ABI shim.
znippy-iceberg/ Apache Iceberg integration
znippy-zoomies/ fast primitives: OSM index, gatling engine, psort
znippy-zoomies/{linflate,lgz,lbzip2,ljar,lzip-parallel,zstd-sys-rs}
znippy-zoomies/osm-katana OSM → GeoParquet/PBF (CLI: convert/xml2pbf/pbf2geo/geo2arrow/verify)
tests/ integration + benchmark suite
xtask/ cargo xtask helpers
```
See `WORKSPACE.md` for full dependency graph and release gate protocol.
---
## Design
### File format
```
[ blob_0 ][ blob_1 ] ... [ blob_N ]
[ Arrow IPC sub-index ] ← rows: path, pkg_type, chunk_seq, offset, len, checksum, metadata cols
[ Arrow IPC sub-index ] ← one sub-index per plugin type (DenseUnion layout)
[ global footer ] ← offsets to each sub-index, version, flags
```
- Blobs are zstd or lz4 compressed chunks (or stored as-is for already-compressed files)
- Each sub-index is a valid Arrow IPC file — readable by DuckDB/Polars/DataFusion directly
- Already-compressed input is stored raw at full I/O speed — it never touches the codec.
`znippy-common/src/precompressed.rs` resolves that in a fixed order: an explicit
caller `ContentHint` (free and exact — `SkipPolicy::already_compressed()` covers a
whole batch), then the extension table (`.jar`, `.gz`, `.crate`, `.png`, `.parquet`,
`.pack`, `.idx` …), then a magic-byte probe over the first 16 bytes (zlib, `PACK`,
gzip, zstd, xz, …), then compress. The probe is what reaches an entry carrying **no
extension** — a git loose object is named for its oid. No entropy estimate: every
probe is an exact pattern at a fixed offset, with no threshold to tune.
The **probe itself lives in `znippy-zoomies/ldeflate/src/precompressed.rs`** and
`znippy-common` re-exports it; only the extension leg (`skip_by_path`) is znippy's,
because ldeflate is handed `&[u8]` and never a `Path`. The arrow cannot point the
other way — znippy-common already depends on znippy-zoomies, so it would be a cycle.
- The **tree's shape** — empty directories and unix modes — rides in the reserved
`__znippy_fsmeta__` section (`znippy-common/src/fsmeta.rs`), Arrow IPC
`(relative_path, kind, mode)`. The data index is a table of *chunks*, so a
directory owning no file owned no row: an empty directory did not survive a
round trip and a `0755` script came back `0664`, both silently, with `verify`
green. Reserved means every data reader skips it, so it is additive both ways —
no format-version bump, and an older znippy opens the archive unchanged.
One row per directory + ONE archive-scoped `FileDefaultMode` row (the tree's
modal mode) + one row per deviating file, so the section is O(distinct modes)
rather than a second copy of every path. Restore order is load-bearing: dirs
before extraction, file modes after, directory modes last and deepest-first.
See `.nornir/compress-design.md` §f7-f8.
### Pipeline (compress path)
```
WalkDir
│
├── BIG files (> 200 MB slice)
│ └── sequential chunked reads → Gatling pipeline → compress → write
│
└── SMALL files (≤ 200 MB)
└── read into slot (ChunkRevolver, coalescing) → batch plugin extract → chunk → compress → write
│
└── Arrow IPC index written incrementally per pass (no accumulation)
```
### Design laws
**Law 1 — Zero-copy channel communication.**
All inter-thread messages are slot indices (`usize`) into a pre-allocated ring buffer (`ChunkRevolver`). No `Vec<u8>` clones cross thread boundaries. On 32 cores saturating NVMe (~7 GB/s), copying payloads would halve throughput.
**Law 2 — No allocation in hot path.**
Compressor threads reuse buffers across chunks. OpenZL context is long-lived. blake3 `Hasher` is `.update()`'d incrementally.
**Law 3 — Deterministic checksums independent of core count.**
Checksums are bound to `chunk_seq` order, not thread identity. Any machine reproduces the same checksums by sorting decompressed chunks by `chunk_seq` before hashing.
**Law 4 — Writer is concurrent, compressors never wait for index.**
The Arrow IPC writer runs on its own thread. Compressors send completed chunks and continue immediately. Index rows are buffered and flushed when a sub-index batch is full.
### Gatling pipeline
No-barrier streaming engine in `znippy-zoomies`. Replaces the original rayon-based pipeline:
- Fixed-size slot pool — producer fills slots, sends slot index, consumer drains
- No barrier between read and compress stages
- Backpressure from pool exhaustion (producer blocks when all slots are in-flight)
- Used in both compress and osm-katana xml→pbf paths
### io_uring (slot packer)
`znippy-compress/src/slot_packer.rs` uses io_uring batched opens+reads (128 files/batch) for the small-file path. Zero per-file syscall overhead — the kernel processes the entire batch asynchronously.
### Plugin system
A plugin (package handler) targets one packaging ecosystem. It implements
`ArchiveTypePlugin` (`znippy-common/src/plugin.rs`): ONE static `HandlerSpec`
(name, type_id, aliases, ecosystem, extensions, columns, extra subcommands — each
stated once) plus `extract_metadata`.
| Method | Purpose |
|---|---|
| `spec()` | the static `HandlerSpec` |
| `extract_metadata(path, &data)` | per-file metadata (the only required logic) |
| `run_command(cmd, args)` | optional; default is the shared `coords` command |
| `matches_path(path)` | derived: ends with a spec extension (override for magic/shape matching) |
| `schema_fields()` | derived from `spec.fields` |
| `name()` / `type_id()` / `meta()` | derived from the spec (`meta()` is the owned discovery record) |
**Discovery**: `znippy-cli/src/handlers.rs::builtin_handlers()` is the single registration point. The CLI autodiscovers handlers via `meta()` — no central name table maintained by hand.
```
znippy handlers # list register
znippy compress --format rust … # select handler by name/alias
znippy run rust coords foo.crate # dispatch to handler's run_command()
```
**Where a handler's source lives — rule P-5** (`.nornir/plugins-design.md` §f3):
| tier | where | condition |
|---|---|---|
| A | `znippy-common/src/plugins/native/*.rs` | needs **only** `std`, `arrow` and `znippy-common` |
| B | its own crate under `plugins/native/` | needs a third-party parser, a decompressor, a network fetch, its own crate-type or its own release cadence |
| W | its own crate under `plugins/wasm/` | is an ABI shim over a tier-B crate, built for `wasm32-unknown-unknown` and loaded by `wasm_loader.rs` |
`znippy-common` sits under every other crate, so a dependency added there is paid by all
of them on every target including `wasm32` — that is the whole reason for the A/B split.
Neither tier is a staging area for the other, and a tier-B crate with its modules at
`src/` root is the normal Rust layout, not a misplacement. Tier A currently holds
cargo/npm/rpm/deb/gem/conda plus the `skeletons.rs` stubs; tier B holds the six crates
under `plugins/native/`; tier W holds `znippy-plugin-maven-wasm`.
**Tier A does not move, and the reason is a consumer, not a preference.** The six
in-common handlers are reached as `znippy_common::plugins::<x>_native::<X>Plugin` from
*outside* this workspace — `holger/traits/src/lib.rs` names six of them in non-test code,
and four holger repository crates name them in tests. Lifting them into
`plugins/native/znippy-plugin-*` crates would need those crates to depend on
`znippy-common` (for `ArchiveTypePlugin`) while `znippy-common` re-exported them back —
a package cycle cargo rejects — so the public path would simply disappear from a
**published** crate. Measured 2026-08-05: the six cost gunnar's `cargo xtask guard`
**zero** crates (they import only `std` + `arrow`, and `gunnar-coldtier` is `exclude`d
from gunnar's workspace, so znippy is not in that graph at all), so there is no
dependency-weight argument for paying that price.
**Tier W is where the wasm ABI lives, and it is a separate crate on purpose.**
`plugins/wasm/znippy-plugin-maven-wasm` holds no maven logic: it depends on
`plugins/native/znippy-plugin-maven` and exports only `alloc` / `result_len` /
`extract` / `plugin_extensions` / `plugin_schema`. Both surfaces read the same
`row.rs` — one `MAVEN_SCHEMA`, one field writer — and
`plugins/wasm/znippy-plugin-maven-wasm/tests/maven_native_wasm_agree.rs` loads the real
`.wasm` and asserts the two produce the same `ExtensionRow` (it fails, never skips, when
the module is missing or stale).
```bash
cargo build --target wasm32-unknown-unknown --release -p znippy-plugin-maven-wasm
cargo test -p znippy-plugin-maven-wasm # the agreement guard; needs the build above
```
**Compile targets are per crate, not uniform**: `znippy-plugin-maven` and
`znippy-plugin-python` still declare `cdylib + rlib`, but nothing loads that host
`.so` — the WASM artifact comes from tier W. media, skidbladnir, rust-toolchain and git
are `rlib`-only compiled-in host handlers.
### CLI surface
```
znippy compress <dir> # compress directory → .znippy archive
znippy compress --format generic <dir> # a tree that is NOT a package registry
# (aliases none/raw/plain/blob, type_id 26): no
# metadata, no pkg_type claimed. The default
# `rust` extracts nothing from such a tree either,
# but records pkg_type = 1 — so `--type rust` and
# every RustView consumer would report a
# container-registry mirror as crates.io content.
znippy append -i <archive> -a <dir> # append a dir into a sealed archive (reuses old blobs, re-seals)
znippy decompress <archive> # reconstruct files
znippy list <archive> # list contents (Arrow query)
znippy verify <archive> # verify checksums
znippy handlers # list registered plugins
znippy run <handler> <cmd> … # handler-specific subcommand
```
---
## Build & test
```bash
# THE build door — reads .nornir/build-variants.toml, the one declaration.
cargo xtask build --suite ORIGINAL --list # what the delivery contains
cargo xtask build --suite ORIGINAL # both atoms
cargo xtask build --suite ORIGINAL --atom znippy # the lean archiver
cargo xtask build --suite ORIGINAL --atom znippy-full # + iceberg, sign
cargo build --workspace
cargo test --workspace
cargo test -p znippy-tests --test build_variants # the build-matrix guard
cargo test -p tests # integration tests
cargo bench -p tests # full benchmark suite
cargo bench -p <crate> # per-crate bench
```
### A `git worktree` of znippy cannot resolve the full workspace
Measured 2026-08-03, cargo 1.97.1. In any checkout that is **not**
`/home/rickard/git/znippy`, every cargo command dies before it builds anything:
```
error: package collision in the lockfile: packages znippy-common v0.9.12
(<the worktree>/znippy-common) and znippy-common v0.9.12
(/home/rickard/git/znippy/znippy-common) are different, but only one can be
written to lockfile unambiguously
```
The cause is one edge: `xtask` dev-depends on `nornir` at `../../nornir`, and
nornir path-depends on `../znippy/znippy-common` — an **absolute destination**, so
it re-enters the canonical checkout while the workspace member resolves inside the
worktree. Two different directories, same name and version.
It is a resolve-time failure, so it is workspace-wide: `-p <crate>` and
`--exclude xtask` do **not** avoid it. Three plausible dodges were tried and all
fail:
| attempt | result |
|---|---|
| sibling symlink farm so nornir's `../znippy` points back at the worktree | **WORKS — see below. This row was wrong.** |
| `paths = [...]` override in `.cargo/config.toml` | path overrides only replace *registry* sources, never a path dependency |
| bump `znippy-common`'s version so the two copies differ | works for that one crate, then collides on `znippy-compress`, and on down the list |
**The sibling symlink farm DOES work** (re-measured 2026-08-30 on oden, lane AU,
cargo 1.96). The row above said cargo canonicalises package roots so the symlink
resolves back to the real checkout. It does not: cargo joins a path dependency
against the directory of the manifest **as it was opened**, so a `nornir` opened
through a symlink resolves `../znippy` inside the farm.
The shape that works, exactly:
```text
~/git/au-lane/znippy ← the worktree, named `znippy` (git worktree add)
~/git/au-lane/nornir -> ~/git/nornir (symlink)
~/git/au-lane/znippy-zoomies -> ~/git/znippy-zoomies
~/git/au-lane/<every other sibling a manifest reaches> -> ~/git/<same>
```
Two conditions, and both matter. The worktree directory must be **named
`znippy`**, because that is the name nornir's `../znippy/znippy-common` asks for
— a worktree at `~/git/au-znippy` collides exactly as documented above. And the
farm needs every sibling any manifest in the graph reaches, not just nornir:
`draupnir` was the second one missing here and cargo names it plainly
(`failed to read .../au-lane/draupnir/Cargo.toml`), so add them until it stops.
MEASURED in that layout: `cargo metadata` resolves, `cargo test --workspace`
builds **including `xtask`** (`Compiling xtask v0.1.0
(~/git/au-lane/znippy/xtask)`), and 636 tests ran. So `xtask/examples/nornir-bench.rs`
is runnable from a worktree too, and none of `[workspace] members` has to be edited.
**The old workaround** — drop `"xtask"` from `[workspace] members` for the length
of the run — still works and is still the answer if you are already in a
badly-named worktree and do not want to move it. It is not a change to commit,
because `-p xtask` is how the release gate invokes the bench suite.
**The consequence for benchmarks:** `xtask/examples/nornir-bench.rs` can only be
*run* from `/home/rickard/git/znippy`, never from a worktree. A real fix means
giving nornir a published `znippy-common` dependency instead of a path one; that
is a nornir-side call, not a znippy one.
---
## Release procedure
See `WORKSPACE.md §Release gate protocol` for the full gate.
**Short form:**
1. `cargo bench -p <crate>` — run benchmarks
2. Compare against `bench_history.json` — no metric drops >10% (same machine)
3. Append result to `bench_history.json`
4. Update `design.md` / `README.md` if architecture or benchmark numbers changed
5. `git tag vX.Y.Z && git push --tags`
6. `cargo publish -p <crate>`
**Publish order when znippy-common changes:**
```
znippy-common
→ znippy-compress, znippy-decompress
→ znippy-plugin-maven, znippy-plugin-python
→ znippy-cli → znippy (znippy DEPENDS on znippy-cli, so cli goes first)
```
**Decompressor publish order (when linflate changes):**
```
linflate → lgz, ljar, lzip-parallel → znippy-zoomies → osm-katana
```
Intra-workspace dependencies are declared with **both** `version` and `path`
(e.g. `znippy-common = { version = "0.9.1", path = "../znippy-common" }`). Cargo
builds against the local path and publishes the version, so local dev and
`cargo publish` both work with **no** `[patch.crates-io]` block. Publish
bottom-up (per the order above) so each crate's deps are already on crates.io
when the next is verified.
---
## Bench history
`bench_history.json` at workspace root — one JSON line per run.
Decompressor crates each have their own `bench_history.json`.
See `WORKSPACE.md §Benchmark column schema` for required fields.