<p align="center">
<img src=".nornir/assets/znippys.jpeg" alt="znippy-zoomies" width="320" />
</p>
# znippy-zoomies
Single-purpose parallel Rust primitives built on one all-core engine, not rayon's fork-join model.
- **`gatling`** — multicore streaming engine; zero copy, allocation, I/O-wait, and CPU-pause.
- **`lgz` / `lbunzip2`** — pure-Rust parallel gzip / bzip2 decoders on gatling.
- **`ldeflate`** — pure-Rust parallel DEFLATE/zlib **compressor**; gzip / ZIP / JAR on one planner.
- **`stree` / `stree32`** — AVX2 static search trees, ~40 ns/query.
- **`vtd`** — parallel OSM-XML scanner → compact byte-offset index; bounded RAM.
- **`psort`** — parallel sample sort over 16-byte `i64`-keyed records.
## Decompressor benchmarks
Each of our parallel decoders against the standard tools **for its own format**, grouped by
format — every one of ours vs every one of theirs, in two modes. One deterministic 1 GB
corpus, min-of-3. Columns are hand-picked (MB/s · cores · wall s · peak RSS). A 🏆 marks a
decoder that beat a long-reigning record — see [`records.toml`](records.toml).
### Multi core variants — all cores
**gzip**
<!-- nornir:gen:start:benches compare=znippy-gz,pigz winner=row self=znippy-gz cols=decompress_mbs,cores_busy,decompress_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
**bzip2**
<!-- nornir:gen:start:benches compare=znippy-bz2,lbzip2 winner=row self=znippy-bz2 cols=decompress_mbs,cores_busy,decompress_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
**zip**
<!-- nornir:gen:start:benches compare=znippy-zip,unzip winner=row self=znippy-zip cols=decompress_mbs,cores_busy,decompress_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
**jar**
<!-- nornir:gen:start:benches compare=znippy-jar,jar winner=row self=znippy-jar cols=decompress_mbs,cores_busy,decompress_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
### Single core variants — one core each
**gzip**
<!-- nornir:gen:start:benches compare=znippy-gz_st,gzip,pigz winner=row self=znippy-gz_st cols=decompress_mbs,cores_busy,decompress_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
**bzip2**
<!-- nornir:gen:start:benches compare=znippy-bz2_st,bzip2,lbzip2_st,pbzip2 winner=row self=znippy-bz2_st cols=decompress_mbs,cores_busy,decompress_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
**zip**
<!-- nornir:gen:start:benches compare=znippy-zip_st,unzip winner=row self=znippy-zip_st cols=decompress_mbs,cores_busy,decompress_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
**jar**
<!-- nornir:gen:start:benches compare=znippy-jar_st,jar winner=row self=znippy-jar_st cols=decompress_mbs,cores_busy,decompress_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
<sub>`cores_busy` = avg CPU cores busy during decode (12 = full saturation). `znippy-*` run all
cores; `_st` rows are pinned to one worker. 🏆 = beat a long-reigning record ([`records.toml`](records.toml)).</sub>
## Latest run — 32 cores, 2026-07-23: the harness refused to publish it
<!-- nornir:gen:start:bench_status -->
<!-- nornir:gen:end:bench_status -->
The published tables above are **untouched** by this run, and that is deliberate: a
preview may never overwrite a measured, published table. `nornir docs render` hides a
preview run from every marker that has not opted in with `preview=yes`, so the blocks
below are the only ones it can reach.
Nothing here failed. The decoders did exactly what they were asked and the roundtrip
tests passed — but the OSM-XML input was **24 MB against the 1024 MB heavy floor**, so
the harness declined to emit a publishable `BenchRun` for the whole run rather than let
an under-floor pass become a number somebody quotes. Read every row below as a
**preview**: honest measurements of a workload too small to publish.
**gzip — preview**
<!-- nornir:gen:start:benches preview=yes compare=znippy-gz,pigz,gzip winner=row self=znippy-gz cols=decompress_mbs,cores_busy,decompress_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
**bzip2 — preview**
<!-- nornir:gen:start:benches preview=yes compare=znippy-bz2,lbzip2,bzip2,pbzip2 winner=row self=znippy-bz2 cols=decompress_mbs,cores_busy,decompress_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
**one core each — preview**
<!-- nornir:gen:start:benches preview=yes compare=znippy-gz_st,gzip,znippy-bz2_st,bzip2,lbzip2_st,pbzip2 winner=row self=znippy-gz_st,znippy-bz2_st cols=decompress_mbs,cores_busy,decompress_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
**OSM convert — preview, and the arm that put the whole run under the floor**
<!-- nornir:gen:start:benches preview=yes compare=osm-katana,osm-katana_st,osmium winner=row self=osm-katana cols=convert_mbs,cores_busy,convert_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
<sub>No `zip` / `jar` rows: those arms did not run in this pass. The host string above is
the harness's own — it composes `machine` from the CPU/RAM/corpus it measured and ignores
`NORNIR_MACHINE`, so this run reads as its Threadripper rather than as `oden`. To promote
any of this to a published table: `NORNIR_ZOOMIES_CORPUS_MB=1024` **and** an osm-katana
input ≥ 1024 MB (or `NORNIR_OSM_KATANA_CLIP=<region>` against a planet file).</sub>
## Compressor benchmarks
Everything above is decode. [`ldeflate`](https://crates.io/crates/ldeflate) (published
2026-08-03) is the workspace's one **write**-side crate — the parallel DEFLATE/zlib
engine plus the gzip, ZIP and JAR deflators built on it. It gets its own section rather
than a column above, because a compressor cannot be raced the way a decoder can: the
bake-off reconstructs *one fixed archive*, while `gzip -6` and this crate emit different
bytes, so MB/s alone ranks nothing. The contested number is what the **schedule** costs
against the *same crate's* serial path over the same objects at the same level — so the
baseline row is `compress_objects_serial`, not `gzip`.
`plan::compress_objects` is the only scheduler in the crate. gzip, ZIP and JAR route
through it and differ **only in framing** — `Container::wrap` puts a header and trailer
around a body the planner already produced, and the three containers added no scheduling
code at all. The planner picks a unit kind per object by size: `Unit::Batch` below
`coalesce_below = 64 KiB` (one task drains many, reusing one encoder), `Unit::Whole`, or
`Unit::Segment` above 4 MiB (cut at `Z_FULL_FLUSH`, so one big blob cannot serialise a
worker). **`Unit::Batch` covers 93.8% of real git objects** — measured over 48 917 of
them — which is why one `Config` serves both git and LFS. `limits.rs` refuses an
amplifying request past `MAX_PIECES = 262 144` rather than attempting it.
### Scaling — the same crate against its own serial path
<!-- nornir:gen:start:benches best=mb_per_sec:high cols=mb_per_sec,cores_busy,speedup_vs_serial,efficiency_pct compare=deflate.repack_git_objects_st,deflate.repack_git_objects_mt1,deflate.repack_git_objects_mt2,deflate.repack_git_objects_mt4,deflate.repack_git_objects_mt8,deflate.repack_git_objects_mt16,deflate.repack_git_objects_mt self=deflate.repack_git_objects_mt -->
**oden · AMD Ryzen Threadripper PRO 3975WX 32-Cores · 499 GiB · 30 770 real git objects,
715.4 MB, median 986 B · level 6 · min-of-3 · load 40–47 · 2026-08-04**
| workload | mb_per_sec | cores_busy | speedup_vs_serial | speedup ÷ cores held |
|---|---|---|---|---|
| deflate.repack_git_objects_st | 72.5 | — | 1.00 | — |
| deflate.repack_git_objects_mt1 | 69.2 | 0.93 | 0.95 | 1.02 |
| deflate.repack_git_objects_mt2 | 137.1 | 1.84 | 1.89 | 1.03 |
| deflate.repack_git_objects_mt4 | 258.5 | 3.55 | 3.57 | 1.01 |
| deflate.repack_git_objects_mt8 | 424.4 | 5.86 | 5.86 | 1.00 |
| deflate.repack_git_objects_mt16 | 621.1 | 8.48 | 8.57 | 1.01 |
| **deflate.repack_git_objects_mt** | **842.7** | **12.15** | **11.63** | **0.96** |
<!-- nornir:gen:end:benches -->
<sub>**Read the last column.** Above 8 threads the `speedup_vs_serial` column measures
**the box, not the code** — oden is shared and this crate takes every core it can get, so
a run that asks for 32 threads and is given 12.15 cores returns 11.63×. Across a 7-run
batch on 2026-08-03 the 32-thread speedup spanned **10.38×–23.20× (123.5% spread)** while
the **CPU cost of the schedule held to 1.025×–1.054× of serial**, and efficiency against
cores *actually held* stayed in a 94.9–97.6% band at every occupancy. An earlier headline
of **28.30× / 88% / 2266 MB/s** is not refuted — carried forward it predicts 29.2–31.2× at
that occupancy — but it was a statement about machine occupancy dressed as one about the
crate, it cannot be reproduced on a shared box, and it is retired as a crate claim.
Full tables, the negative controls and both falsified hypotheses: [`benchmarks.md`](benchmarks.md).</sub>
### Never compress what is already compressed
`precompressed.rs` gates the entropy coder on **hint → extension → magic-byte probe** — no
entropy estimate, no tunable threshold, at most 16 byte compares. Acting on a `store`
verdict is free because `stored_deflate` emits RFC 1951 §3.2.4 stored blocks: a
length-prefixed `memcpy`, *not* `flate2` at level 0, which would still build, reset and
walk the encoder.
**oden · 24 real git packfiles + pack indexes · 113.8 MiB · level 6 · min-of-3 · decided on
CPU seconds · load 66 · 2026-08-04**
| probe | cpu-s per GiB | wall MB/s | ratio |
|---|---|---|---|
| off — deflate everything | 22.85 | 184.8 | 1.0041 |
| **on — store the already-compressed** | **0.99** | **724.6** | 0.9999 |
**→ 23.1× less CPU, 95.7% saved, output +0.41% larger.**
<sub>Decided on **CPU seconds**, not wall time: at load 66 the wall column is noise (the
same arm read 267.5 / 2075.5 MB/s on a quieter box) while `cpu-s per GiB` moved under 2%.
**Quote the saving, not the multiplier** — across four runs on 2026-08-03/04 the multiplier
wandered 19.4×–23.1× because its denominator is ~1 cpu-s/GiB, while the saving held at
94.8–95.7% and the size penalty came out **byte-identical (+0.41%) every time**. It
independently reproduces znippy's own 92–95% figure. Negative controls, same harness:
with the probe given nothing to find (git loose objects) it collapses to **1.03×**, and
with the probe vetoed by `ContentHint::Compressible` over the *same* packfiles to
**1.04×** at byte-identical output — so the 23× is a property of the probe.</sub>
## OSM converter
`osm-katana` (in-repo sub-crate) converts OSM-XML/PBF → GeoParquet all-core on a no-barrier
gatling pipeline, head-to-head with the legacy reference tools `osmium` / `osmconvert` for the
same input. One convert pass, min-of-3. Columns: MB/s, cores busy, wall seconds, peak RSS (MB).
<!-- nornir:gen:start:benches compare=osm-katana,osmium,osmconvert self=osm-katana cols=convert_mbs,cores_busy,convert_s,peak_mem_mb -->
<!-- nornir:gen:end:benches -->
<sub>`convert_mbs` = input MB / wall; `cores_busy` = average CPU cores busy (12 = full saturation
on this box). `osm-katana` runs all cores; `osmium` / `osmconvert` are the single-stream C++
references, benched only when installed. By default a small synthetic OSM-XML corpus exercises the
emit path on any box — the HEADLINE number comes from the off-box planet run
(`NORNIR_OSM_KATANA_INPUT=<planet.osm> nornir bench run znippy-zoomies`). Region **clip** headline
(cut Sweden out of the planet with our tool): add `NORNIR_OSM_KATANA_CLIP=10.9,55.3,24.2,69.1`.</sub>
### Staged pipeline
The classic planet chain — each stage eats the **previous stage's output file**:
`xml2-pbf` (OSM XML/bz2 → PBF) → `pbf2-geo` (PBF → GeoParquet) → `geo2-arrow` (GeoParquet → Arrow
IPC). `NORNIR_OSM_KATANA_CHAIN=1 nornir bench run znippy-zoomies` warehouses one row per stage
(`stage_mbs` = that stage's input MB / wall); point `NORNIR_OSM_KATANA_INPUT` at a planet for the
heavy chain. Off by default — it adds a full three-stage pass — so the table lands after that run.
## Docs
- **[Manual](docs/manual.pdf)** · **[Idiot's guide](docs/idiot_guide.pdf)**
## License
MIT. The `stree` module retains Ragnar Groot Koerkamp's copyright notice (see `LICENSE`).