# plan — znippy
<!-- nornir:class: unsorted -->
The **planning spine**: the single source of truth for *what's next*, the locked
decisions behind *what is*, and the working order. The docs describe what exists
(`.nornir/design.md` = the map; `.nornir/{benching,random-access}.md` = the
territory); this file is the live worklist. Supersedes the frozen
`legacy/TODO_NOW.md` + `legacy/backlog.md` (kept only as history — do not add to
them).
Convention: this is the nornir `plan.md` "spine" (see
`nornir/.nornir/docs-generation.md §Doc taxonomy`). One source per fact —
decisions live here; the docs *link* to them.
---
## Laws (locked)
1. **Zero-copy channel communication.** Inter-thread messages are slot
indices / `ChunkMeta`, never payloads. (Design law 1.)
2. **No allocation in the hot path.** Reused compress/decompress buffers,
long-lived codec context. (Design law 2.)
3. **Deterministic checksums independent of core count.** blake3 bound to
`chunk_seq` order, over the *original* (pre-compression) bytes. (Design law 3.)
4. **Writer is concurrent; compressors never wait on the index.** Workers pwrite
their own payload at an `AtomicU64` cursor; the finalizer only builds the
Arrow index from `ChunkMeta`. (Design law 4.)
5. **No-barrier Gatling pipeline — ever.** One global slice queue, no
slot-completion barrier (the lbzip2-v1 bug). Decided 2026-05-24.
6. **Benchmark-first for any performance change.** Baseline the OLD code, change
it, re-run the SAME bench, compare on the same machine; proceed only if not
slower (>10% same-machine = release-gate fail). The bench suite is nornir's
(`xtask/examples/nornir-bench.rs`), not criterion — see `.nornir/benching.md`.
Applies squarely to all of #7.
7. **Public/private hygiene.** `znippy` is public; `nornir` is private. Never
copy nornir's internal docs into `znippy/.nornir`. znippy's own docs follow
nornir's doc conventions but carry only znippy content.
---
## Done (0.9.x)
- **#7 B6 — ljar zero-copy chunk output** (`ljar::chunk::decode_chunk_into`).
The lbzip2 pattern done right: one pre-allocated buffer, **candidate offsets**
per segment, all segments decoded in parallel *in place* (no per-segment
`Vec`), then **forward-seek compaction** to the true cumulative offsets — caller
reuses the buffer across chunks. Single-segment hits a true zero-copy fast
path; a slot overflow (rare) falls back to owned buffers, never corrupts. Test
`decode_chunk_into_matches_decode_chunk`; A/B bench `benches/chunk_into.rs`
(`decode_chunk_flatten` vs `decode_chunk_into_reused`).
- **Compress** uses the gatling no-barrier ordered-sink
(`gatling::ordered::run_ordered_sink`, off the old per-thread ChunkRevolver;
`int_ring` deleted). **Decompress** is a different design: positioned `pread`
(`FileExt::read_exact_at`) driven by a shared atomic row cursor over
`thread::spawn` — no gatling, no slots, no channel. Each worker grabs the next
row via the lock-free cursor, preads its blob, decompresses/verifies, and
`pwrite`s to `fdata_offset` (`znippy-common/src/decompress.rs`).
- **Random-access reads** — sorted lookup sub-index + `fst` trie (`.nornir/random-access.md`).
- **Iceberg metadata sink** — `ArchiveMetaSink` + `ArrowIpcSink` (byte-identical
default) + `znippy-iceberg::IcebergSink`.
- **Store-if-incompressible** (commit `8bca9aa`) — write path stores raw when the
codec frame ≥ input; never inflates a blob, skips the decode cost. Test
`test_incompressible_data_stored_raw`; bench metrics `raw_chunks` /
`compressed_chunks`.
- **nornir bench suite + warehouse + `workspace_znippy/` control plane**
(registered, monitored). Bench cores/file-count metrics; region geo benches
(`geo_liechtenstein` always-on, `geo_europe` skip-until-present).
- **osm-katana region-clip + synthetic generator** (item B / J-perf) — see
`.nornir/region-clip.md`. `bbox-at-decode, None-free, vtd_bench-gated`:
clip is one branchless point-in-box test in `reader::parse_nodes` *before*
tag-JSON, behind the default-OFF `region-clip` feature (lean build
byte-identical). Presets `europe`/`nordics`, CLI `--clip`, lib
`extract_region`. `write_synthetic_geoparquet` drives the real `NodeWriter`.
Inject-assert tests (synthetic round-trip + liechtenstein None/europe/tight/empty
+ `KATANA_FIXTURE` opt-in real-data arm). vtd_bench decode arm proves the
None-path matches the no-branch baseline (≈62.7 ms vs 64.36 ms, 353 167 nodes,
3975WX/32c) — no regression; tight box is faster (skips 91 % tag-JSON).
- **gatling::io async-sibling circuit SVG** (J2-old) — `assets/gatling_io_flow.svg`
in the house style (dark gradient, cyan/ice, monospace, left→right), embedded
in `gatling.md` alongside the sync `gatling_flow.svg`.
---
## Open work
Priority order. Each perf item is gated by Law 6 (baseline first).
### Bounded / ready
- **#5b — `IcebergZnippyReader: ZnippyReader`** (the remaining half of #5).
**✅ UNBLOCKED — `skade` shipped 2026-06-09** (`skade/skade`, 0.1.0)
and verified against znippy's wishlist (`nornir/znippy_agents_whishes.md §4`):
every item delivered — `skade::open(dir)` one-call warehouse,
`Table::{append,ingest,read,count,sql}`, bridges incl. **`unwiden` with
bit-reinterpret** for `u64 > i64::MAX` (exact `blob_offset` round-trip),
windowed parquet reads, SQL via datafusion 52.2 (shares arrow 57 — no
bridging). 8/8 skade tests green locally. **Implementation now bounded:**
1. `znippy-iceberg`: swap `MemoryCatalogBuilder` → `skade::open(warehouse)`
in `IcebergSink` (durable `catalog.redb`, fixes the ephemeral-catalog flaw);
write sub-indexes via `Table::append`.
2. `IcebergZnippyReader`: `skade::open` → `table(name).read()` →
`skade::unwiden(batches, znippy_schema)` → file→chunks map → pread +
`codec::decompress_into` from the `.znippy` sidecar.
3. Roundtrip test: compress `--meta-format iceberg` → read back → byte-compare.
**Gate CLEARED 2026-06-09:** skade 0.1.0 published to crates.io (by us,
user-authorized; dry-run verified against stock deps). znippy-iceberg can
depend on `skade = { version = "0.1.0", path = "../../skade/skade" }`
— wait, skade lives OUTSIDE the znippy workspace (sibling repo), so use
version-only `skade = "0.1.0"` in znippy-iceberg. Implementation may start.
Original analysis (kept for context):
- **Root problem with the current sink:** `IcebergSink::finish` uses
`MemoryCatalogBuilder` — the catalog (table pointers) is **in-process and
dies with the process**; only warehouse files survive. Read-back would mean
crawling `metadata/*.metadata.json` + `StaticTable` and *guessing* which
metadata file is current. Fragile by construction.
- **The fix: swap `MemoryCatalog` → `skade_katalog::RedbCatalog`** (crates.io
0.1.x, pure-Rust, redb-backed, implements `iceberg::Catalog`). A
`catalog.redb` beside the warehouse becomes the durable catalog. Versions
align exactly (iceberg 0.9.1, arrow 57.x both sides); dependency direction is
acyclic (`znippy-iceberg → skade-katalog → znippy-zoomies`).
- **Write side bonus:** `RedbCatalog::atomic_release` = multi-table atomic
commit — znippy writes one table per `(pkg_type, repo)` sub-index, so a whole
archive's index can flip in ONE transaction (today each table commits
separately).
- **Read side becomes trivial** (the skade-katalog pattern):
`RedbCatalogBuilder::default().db_path(…).warehouse_location(…).load(…)` →
`catalog.load_table(&ident)` → `table.scan().select_all().build()?.to_arrow()`
→ RecordBatch stream → **un-widen** the sink's signed cast (`Int32→UInt32`,
`Int64→UInt64`; don't reuse `ZnippyArchive::build_file_index`, it expects
`UInt64`) → file→chunks map → pread + `codec::decompress_into` from the
`.znippy` blob sidecar. Roundtrip test: compress `--meta-format iceberg` →
read back → byte-compare.
- **SQL for free:** `iceberg_datafusion::IcebergCatalogProvider::try_new(catalog)`
+ DataFusion `SessionContext` → `SELECT … FROM nornir.<ns>.<subindex>` —
proven in skade-katalog's `datafusion_sql.rs` (they vendor a patched
iceberg-datafusion 0.9.1 with a parallel per-file partitioned scan). Gives
`znippy` SQL over an iceberg-format archive index without DuckDB. Optional
feature, after the reader.
- **F6 cross-pollination:** skade-katalog's `parquet_layout` +
`rowgroup_windows` + `read_row_groups` (bounded-memory windowed row-group
ingest, parallel across threads) is exactly the shape the spatial archive
(F6) needs for billion-row coordinate ingest.
### Large / standalone (still benchmark-first)
- **#7 — ljar decode perf.**
- **B6** — ✅ done, library *and* binary (see Done). Library:
`decode_chunk_into`. Binary (wired 2026-06-09): the candidate-offset pattern
woven into the pread + shared atomic row cursor decompress workers
(`thread::spawn`, no gatling/slots/channel) — per chunk ONE pooled output
buffer with per-segment candidate offsets (exact cap when the entry's
uncompressed size is known; ratio-hint cap `seg_cap()` otherwise), each
worker `inflate_segment`s straight into its region (overflow → owned spill,
rare), forward-seek compacts in place, then `pwrite`s straight to
`fdata_offset` and reuses the buffer for its next cursor row. Zero
per-segment allocation in steady state. **A/B (law 6), 163 840-entry 10 GB corpus, best-of-5:
7.12 s → 6.70 s (~6 %), variance 7.1–11.3 → 6.7–7.4 s.** Output
byte-identical. The library `decompress_jar` path was already pre-sized via
`linflate::inflate_to_vec(uncompressed_size)`.
- **B7** — ✅ **was already shipped as `linflate`** (discovered 2026-06-09;
this backlog entry was stale). The custom SIMD DEFLATE decoder exists as
the `linflate` crate — all five design modules (bitreader/tables/fixed/
copy/fastloop per `linflate/DESIGN.md`), shared by ljar+lgz+lzip-parallel.
The leftover pre-extraction fork `ljar/src/inflate/` (diverged; still
carried the refill_slow corruption bug) was **deleted**; `ljar::inflate`
is now a re-export of linflate, so the criterion `inflate` benches measure
the live core. New nornir scorecard: `linflate_guava`/`linflate_scala`
(vs miniz_oxide). Chapter: `.nornir/inflate.md`. **#7 is closed.**
- **BUG — FIXED 2026-06-09** — linflate `refill_slow` looped on `bits <= 56`,
so entering with exactly 56 bits (what a fast refill leaves) pushed `bits`
to 64: invariant breach (debug assert), and in release the next fast
refill's `<< bits` masked to `<< 0` on x86 → 8 raw bytes OR'd over the whole
register, silent corruption. Fix: loop bound `< 56` (max exit 63, ≥56
post-condition kept). Regression test
`refill_slow_at_56_bits_keeps_invariant`; ljar `batch_timing` green again;
lgz + lzip-parallel suites pass.
- **F6 — spatial archive.** znippy as a geo-indexed coordinate store for
osm-katana (tiled coords, per-tile random decompress). Roadmap / research.
- **F7 — VTD → zoomies.** Move the generic parallel-XML scanner out of
osm-katana into znippy-zoomies. Roadmap.
### Blocked on upstream
- **Arrow IPC scatter-gather write** (apache/arrow-rs#9835). The metadata-write
copy costs ~3.6× on metadata-heavy archives. Our code is already structured for
it; `cargo update` recovers the speed when it lands. No action.
---
## Working order / release gate
Publish bottom-up per `CLAUDE.md §Release procedure`; gates per
`workspace_znippy/release/SPEC.md` (bench → no >10% same-machine regression →
history append → docs fresh → tag → publish). The README's `<!-- nornir:gen -->`
tables regenerate from the warehouse — never hand-edited.
---
## Session checkpoint 2026-06-10 — in-flight work LANDED + RELEASED
**RELEASED 2026-06-10** (master `5c017bb`): `znippy-zoomies 0.1.9` (xml+json
scanners, additive) and `znippy-iceberg 0.1.1` (#5b reader + durable skade
catalog, additive — znippy-cli's `^0.1` dep unaffected) published to crates.io;
tags `zoomies-v0.1.9` + `iceberg-v0.1.1` pushed to codeberg. Other workspace
crates unchanged at 0.9.1 (not republished). Sibling releases the same day:
`skade-katalog 0.1.4`, `nornir 0.4.4`. Full bench-gate/docs-render (SPEC gates
2–5) NOT run — these are additive feature/scanner releases with no perf change;
the bench harness (`xtask`→`nornir` dev-dep) is unaffected. Original pre-release
checkpoint below.
---
## Session checkpoint 2026-06-10 — in-flight work LANDED, release pending
All three in-flight streams from the 2026-06-09 checkpoint are **finished,
green, and on master** (`5983058` merge + `fix(xtask)` doctest on top):
- **F7a — generic XML scanner.** `znippy_zoomies::xml` extracted from `vtd`;
vtd is now the OSM specialization on top (public API preserved; osm-katana
builds+tests green). `pub mod xml` + `benches/xml_scan.rs` wired. Fixed the
doc example (`TagSet::new` needed `&[u8]` slice coercion). Commit `5a3fe82`.
- **F7b — parallel NDJSON scanner.** `znippy_zoomies::json` (the worktree
commit `24db989`) integrated by copying `json.rs` + `json_scan.rs` and
wiring `pub mod json` + serde_json dev-dep + bench. 14 json tests green.
Commit `66fa30e`. Agent worktree retired.
- **#5b — IcebergZnippyReader.** Built on the `feat/iceberg-reader` branch
(was `znippy-5b/` worktree), merged no-ff into master. skade `0.1.0`
version-only dep, `default-features = false`. Roundtrip test green.
Commit `c538252`, merge `5983058`. Worktree retired.
**Cargo.lock note (resolved):** the big crypto/server tree (russh/tonic/axum/
quinn/aws-lc) is legit — it is `nornir`'s transitive tree, pulled as a **dev**
dep of `xtask`/`ljar`/`lzip-parallel` (the bench harness). Not in any
published crate's runtime closure. The 2026-06-09 "investigate origin" caution
is closed.
**Full `cargo test --workspace` is GREEN** (lib + integration + doctests).
Only remaining work: the **release** itself (bench-gate → version bump →
tag → `cargo publish` bottom-up → push to codeberg). Release order per
`CLAUDE.md §Release procedure` / `WORKSPACE.md`.
**Open question for the human (auto-authorized "YES on all" this session):**
versions chosen by the agent — see the release checkpoint appended at run time.
---
## Session checkpoint 2026-06-09 (late) — in-flight work
**Two background agents were launched on F7 and may have left UNCOMMITTED
work** (they were instructed not to commit; review before committing):
- **F7a — generic XML scanner.** Working in the MAIN tree
(`/run/media/rickard/T9/git/znippy`). Goal: extract tag-generic
`znippy-zoomies/src/xml.rs` out of `vtd.rs` (vtd = OSM specialization on
top; osm-katana's `xml_vtd::*` paths must keep working), + criterion bench
`benches/xml_scan.rs`, benchmark-first A/B (no regression vs current vtd).
Check `git status` in the main tree for its output; verify
`cargo test -p znippy-zoomies -p osm-katana` before committing.
- **F7b — parallel NDJSON scanner.** Working in an ISOLATED WORKTREE
(check `git worktree list`). Goal: `znippy-zoomies/src/json.rs` mirroring
the vtd pattern — RFC 8259 raw-`\n`-illegal-in-strings ⇒ memchr newline =
safe split; zero-copy `RecordSpan` callback API; string-aware `find_key`
(find_attr analog); criterion bench `benches/json_scan.rs` vs serde_json
per-line baseline (serde_json dev-dep only). Single-array JSON = documented
v2. Merge the worktree into master after review.
After both land: write the `.nornir` chapter for the scanners (xml+json),
pointer in `design.md`, close F7 here.
**#5b is READY TO IMPLEMENT** (deliberately not started — agent F7a owns the
main tree's build): skade 0.1.0 is on crates.io. Steps are in the #5b entry
above. Use `skade = "0.1.0"` (version-only; sibling repo) in znippy-iceberg.
**Done this session (all pushed, master `7158b85`):** store-if-incompressible;
`.nornir/` doc model (design map + plan spine + inflate chapter); iceberg
write backend (`--meta-format/--warehouse`) + `compress_stream_with_sink`;
selective extract (`--type/--repo`); B6 (library `decode_chunk_into` + binary
candidate-offset pipeline, A/B 7.12→6.70 s); linflate `refill_slow` corruption
fix; B7 reality (linflate IS the SIMD DEFLATE; deleted ljar's diverged fork;
`linflate_*` nornir scorecard 1.33–1.39× vs miniz); skade analyzed, verified,
**published to crates.io by us**; `workspace_znippy` created+registered+pushed;
nornir wishes/mcp_todo coordination files.
**Still open elsewhere:** `_st` bench refresh + docs-render-all (nornir agent);
F6 spatial archive (roadmap); pre-existing dirty `Cargo.lock` at session start
(left uncommitted on purpose — investigate origin before committing).
---
## 2026-06-11 — stree (static search tree) benched + perf-fixed
Worried about the Ragnar-derived id-lookup tree (`znippy-zoomies/src/stree.rs`).
Found the article ("Static search trees: 40× faster than binary search",
curiouscoding.nl) and reproduced his comparison.
- **Benches added to nornir bencher** (NOT criterion): `znippy.stree_8mb /
_128mb / _1gb` in `xtask/examples/nornir-bench.rs` — single `find_exact` +
batched `batch_stream` vs `binary_search` baseline, swept cache→RAM.
`stree_1gb` gated behind `NORNIR_BENCH_SKIP_HEAVY`.
- **Gap found:** original ~110 ns/q @ 1 GB, ~55 ns @ 8 MB batched — 2–4× off
Ragnar even in cache. `binary_search` baseline matched his (~1050 vs ~1150 ns),
so harness sound.
- **Two fixes (both keep API + all 32 stree tests green):**
1. AVX2 multiversioning — `count_lt` was dispatching `is_x86_feature_detected`
per node + calling a non-inlinable `#[target_feature]` fn. Hot traversals
now generic over the counter, AVX2 check hoisted once per query/batch into a
`#[target_feature]` wrapper so the intrinsic inlines.
2. `advise_hugepage()` (`madvise(MADV_HUGEPAGE)`) on the tree Vec in both
constructors — host is `transparent_hugepage=madvise`, so the tree got 4 KB
pages → dTLB thrash at 1 GB.
- **Result:** default build 1 GB batch 110→96 ns (11.2×); `target-cpu=native`
1 GB 66 ns (16×) and 8 MB 19 ns — in Ragnar's single-thread range.
- **Doc:** `.nornir/stree.md` (full write-up + remaining levers: B=16 two-line
node, native planet build, interleave/left-max, multi-thread → his ~7 ns),
pointer added in `benching.md`.
- **Not committed** — left for review. Remaining levers need format/layout
sign-off (B=16 changes internal-node on-disk layout).
**Follow-up (same day):** tried `const B=16` (two-cache-line node) per request.
All tests green but it **regresses the batched path** (best ns/q B=8→B=16: 8 MB
40→47, 128 MB 68→82, 1 GB 104→120 default) — P-way batching already hides the
miss latency fewer levels would save, leaving only B=16's doubled node cost.
Helps serial-only (1 GB serial 580→457) but production is batched. Reverted to
B=8; `count_lt_avx2` left B-agnostic so it's a one-line flip. Ragnar credit
strengthened in the module header + `stree.md`. NOTE: couldn't re-run the full
nornir suite — the sibling `../../nornir` working tree is mid-edit by another
agent and currently fails to compile (`viz/model.rs` gix mismatch); benched via a
throwaway `znippy-zoomies` example instead (no nornir dep). Cargo.lock shows only
a skade 0.4.6→0.4.8 path-dep bump, unrelated.