znippy
A parallel, random-access archive: pack a directory at all-core speed, pull any single file back without unpacking the rest — and query the index in DuckDB / Polars / DataFusion, because it's just Arrow IPC.
⚡ GATLING — one all-core engine · zero rayon · zero-copy · ~11/12 cores
Mashup — one archive vs tar/zip/7z/parquet
znippy is the only archive that's all-core on both ends, random-access per file, and directly queryable — without pretending to beat zstd/7z on raw ratio:
| capability | znippy | tar + zstd | zip | 7z | parquet |
|---|---|---|---|---|---|
| All-core compress | ✅ | ~ (-T0) | ◐ | ✗ | ✅ |
| All-core decompress | ✅ | ✗ serial | ✗ | ✗ | ✅ |
| Random single-file read | ✅ | ✗ | ✅ | ✗ (solid) | — (columnar) |
| Query without the tool (DuckDB / Polars) | ✅ | ✗ | ✗ | ✗ | ✅ |
| Best pure ratio | ◐ | ✅ (-19) | ✗ | ✅ | — |
The full capability matrix and every measured per-backend throughput table (JAR / ZIP / gzip / bzip2 decode vs the legacy C tools) are in the full overview.
Measured — znippy vs Arrow-IPC / zstd
One byte-identical mixed corpus (compressible text + incompressible blobs) packed
into a single compressed, queryable Arrow-IPC container two ways — znippy's
all-core pipeline vs a stock arrow-rs FileWriter with CompressionType::ZSTD —
then fully read back. Same box, same run, warmup + iterations:
v0.9.6 · oden · 32 cores · 2026-07-22
| workload | compress_mbs | cores | decompress_mbs | files | in_mb | peak_rss_mb | ratio_x |
|---|---|---|---|---|---|---|---|
| arrowipc_mashup | 416.22 | 32 | 908.10 | 8000 | 1000 | 5208 | 1.44 |
| znippy_mashup | 111.89 | 32 | 8466 | 8000 | 1000 | 5208 | 1.47 |
Not the same job: znippy stores every file as its own random-access, blake3-verified chunk behind a queryable FST index and an immutable seal; the Arrow-IPC/zstd rival concatenates ~1000 files per record-batch into one zstd stream — smaller granularity traded away, no per-file addressing. So arrow's simpler batching wins raw compress throughput while znippy wins decompress and keeps capabilities arrow can't offer; both hit the same ratio on identical distinct data. Each side uses its library's default codec settings.
Read the real docs
- 📖 The complete manual —
docs/manual.pdf— every subsystem chapter (format, Gatling pipeline, random-access, Iceberg/seal, decompressors). - 📄 Full overview —
.nornir/README-full.md— the detailed pitch, quickstart, what's-new, the full competitive matrix, and all measured benchmarks. - 📝
docs/book.md— the manual, as Markdown.