# znippy-zoomies — Årsstädning audit (2026-06-28)
> **Reconciliation (2026-07-17):** the dated snapshot below is preserved as
> authored on 2026-06-28. Findings that no longer reflect the tree have since
> been closed in place — marked `[x]` with a dated **RESOLVED** note — rather
> than deleted, so the original audit stays honest. The tallies under **Counts**
> are the original 2026-06-28 figures and are left unchanged.
Overall health is good: the low-level core (gatling sync/forkjoin/io engines, psort sample sort, stree/stree32 SIMD search, linflate fast loop, chunk_revolver slot pool, vtd/xml/json scanners) is genuinely state-of-the-art and the resolved-mode pass-2 Parquet path is clean and parallel. The recurring themes are (1) the four codec sub-crates (lgz, ljar, lzip-parallel, lbzip2) hand-roll the gatling + chunk_revolver pipeline four times and duplicate chunk-split/deflate-scan/zip-core code instead of driving the in-repo helpers — with ljar/lzip-parallel being near-verbatim twins (JAR == ZIP); (2) the osm-katana way/node hot paths force per-way coord clones and throwaway HashMaps and carry two superseded node-store implementations (~700 dead lines); (3) the flagship sync gatling engine and chunk_revolver (unsafe pointer code) have zero tests and no perf bench; (4) capabilities (verify, streaming decompress) live inside binaries with no library/Serialize twin; and (5) docs/README drift around stree numbers and an absent xtask harness. Two test suites also shell out to `curl`/`unzip`, the only no-shelling violations in the repo.
## Tier 1 — Hygiene & principles
### zero-copy-alloc
- [ ] **[HIGH]** encode_wkb forces a per-way coords Vec clone at every call site — `osm-katana/src/shared/gpu_backend.rs:9-26` (callers `osm-katana/src/reader.rs:106`, `:169`, `osm-katana/src/xml_reader.rs:2439`) — `GpuBackend::encode_wkb` takes `&[RawWay]` where `RawWay{coords: Vec<(f64,f64)>}` owns coords it only reads, so every way is deep-cloned (`coords: collected[i].coords.clone()`) into a throwaway RawWay; on planet that is hundreds of millions of coord-Vec allocs+memcpys across all three way pipelines for zero benefit. **Fix:** make WKB encode borrow — change `RawWay.coords` to `&'a [(f64,f64)]` (or add `encode_wkb_borrowed`/take `&[CollectedWay]`), build the list with `coords: &collected[i].coords`; CpuBackend already only reads the slice. _(effort: M)_
- [ ] **[HIGH]** Per-way throwaway `HashMap<String,String>` built just for is_closed_area when a zero-alloc slice helper exists — `osm-katana/src/xml_reader.rs:680-684`, `osm-katana/src/reader.rs:374-395` — `parse_way_meta` does `tag_pairs.iter().cloned().collect()` then `is_closed_area(&refs,&tags_map)`; reader.rs is worse, building the map with `(*key).to_string(),(*val).to_string()` (two heap allocs/tag) only to area-test and serialize — all discarded per way. `geometry::is_closed_area_slice` already takes the tag-pair slice directly. **Fix:** call `is_closed_area_slice(&refs,&tag_pairs)`, delete the HashMap, and serialize tags straight off the borrowed `&str` pairs. _(effort: S)_
- [ ] **[MEDIUM]** Raw bz2/gz path serializes way/rel Parquet zstd encode on a single writer thread while nodes are parallel-encoded — `osm-katana/src/xml_reader.rs:1003-1083` (RawParallelParquetSink), wiring `:1136-1145` — in geometry=raw single-pass, node row groups encode on workers but way/rel batches ship to one `writer::ParquetSink` thread doing serial zstd for ways AND rels; the resolved pass-2 path already solved this (`parallel_way_writer`/`parallel_rel_writer`, `:2034-2051`,`:1989-2012`). Raw mode is the asymmetric laggard with N-1 cores idle. **Fix:** switch RawParallelParquetSink to the Pass2TypedSink pattern — workers call `enc.encode(&batch)`, collector only `w.append(rg)`; drop the single ParquetSink. _(effort: M)_
- [ ] **[MEDIUM]** Arrow IPC writer re-copies the entire sorted column arrays with per-batch `to_vec()` — `osm-katana/src/node_store_arrow.rs:223-234` — after sorting, the writer loops batches doing `Int64Array::from(sorted_ids[start..end].to_vec())` (+lats/lons), copying every element of the already-materialized sorted Vec a second time into fresh per-batch Vecs — tens of GB of needless memcpy on the single finalize thread at planet scale. **Fix:** build each full column array once (consume the sorted Vec), then `array.slice(start,len)` per batch (zero-copy, shares Buffer); removes all three `to_vec()`s. _(effort: S)_
- [ ] **[MEDIUM]** optimize() copies every WKB geometry out of the zero-copy Arrow BinaryArray into an owned Feature.geometry Vec — `osm-katana/src/optimize.rs:293-297` — `geom.value(i)` is a borrowed `&[u8]` into the decoded column but the worker stores `geometry: g.to_vec()` into Feature, deep-copying every kept feature's WKB (100M+ features = a full second copy of the geometry column). **Fix:** carry a zero-copy slice handle — keep the source `Arc<BinaryArray>` alive and store `(array,row_index)`, or store `bytes::Bytes` sliced from the Arrow Buffer. _(effort: M)_
- [ ] **[MEDIUM]** Decode workers materialize owned-String NodeRecords (incl. `String::from("{}")` per untagged node) as an intermediate before Arrow build — `osm-katana/src/reader.rs:295-309`, consumed `osm-katana/src/writer.rs:471-479` — `parse_nodes` builds `Vec<NodeRecord>` with an owned `tags_json` String per node (and a fresh 2-byte heap alloc for every untagged node — the vast majority), then NodeAccumulator copies each into the Arrow StringBuilder: every tag allocated once, copied again. **Fix:** append directly into the Arrow builders (serialized tag bytes; `append_value("{}")` as a borrowed literal for untagged), eliminating the Vec+owned-String hop; at minimum kill the per-untagged-node `String::from`. _(effort: L)_
- [ ] **[LOW]** Gatling collector reallocates the carry buffer every chunk with `buf[safe..].to_vec()` — `src/gatling.rs:412-426` — the no-barrier collector flush allocates a fresh Vec for the boundary remainder each flushed chunk, a steady per-chunk allocation in the core helper used by every consumer. **Fix:** keep a reusable carry Vec — `carry.clear(); carry.extend_from_slice(&buf[safe..]);` (or `copy_within`+truncate+swap). _(effort: S)_
### no-shelling
- [ ] **[HIGH]** Tests shell out to `curl` to download Maven JARs instead of a pure-Rust HTTP client — `ljar/tests/maven_artifacts.rs:81` and `lzip-parallel/tests/maven_artifacts.rs:81` (`fn ensure_jar`) — spawns `Command::new("curl").args(["-sSfL","-o"])` (with `.expect("curl must be available")`) to fetch from repo1.maven.org: forbidden shelling, opaque failures, non-portable, duplicated across both files. **Fix:** replace with a pure-Rust blocking GET (`ureq` dev-dep, or `reqwest` blocking) writing the body to `path`, factored into one shared test-support helper. _(effort: S)_
- [ ] **[HIGH]** Benchmark baseline shells out to system `unzip` instead of the in-process `zip` crate (already a dev-dependency) — `ljar/tests/maven_artifacts.rs:400` & `:599`; same lines in `lzip-parallel/tests/maven_artifacts.rs` — `bench_unzip` + the sequential compare spawn `Command::new("unzip")` (`.expect("unzip must be available")`); redundant since both crates already use the `zip` crate next door in `reference_entries`. **Fix:** replace with an in-process extraction loop over `zip::ZipArchive` writing each entry to the tempdir and timing that (apples-to-apples Rust vs Rust), shared between both files. _(effort: S)_
- [ ] **[LOW]** CLI end-to-end test spawns its own built binary via Command instead of calling the library entry point (symmetry) — `ljar/tests/maven_artifacts.rs:259` (`maven_cli_extract_guava`) and `lzip-parallel/tests/maven_artifacts.rs:259` — runs `Command::new(env!("CARGO_BIN_EXE_ljar"))...status()`, asserting only an exit code + on-disk files, not structured output. **Fix:** move extraction into a public lib fn (e.g. `ljar::extract_jar_to_dir`), have the bin and the test both call it; keep the subprocess test as a thin smoke check only. _(effort: M)_
### helper-reuse
- [ ] **[HIGH]** Four codec CLIs hand-roll the gatling streaming pipeline + chunk_revolver slot pool — `lgz/src/bin/lgz.rs`, `ljar/src/bin/ljar.rs`, `lzip-parallel/src/bin/lzip.rs`, `lbzip2/src/bin/lbunzip2.rs` — each builds the identical Reader→Main(carry+split)→N Workers→ordered Collector→Writer engine by hand with raw `mpsc::sync_channel` wiring plus a manual `Vec<u8>` ring-slot recycle (FilledSlot/WorkItem/CollectorMsg/WriteMsg, is_ring_slot, done_count lifetime) — ~500-700 lines each — i.e. `gatling::run` + chunk_revolver reimplemented four times in the same repo. osm-katana already proves these codecs plug into gatling via `TypedCodec`/`TypedSink`. **Fix:** implement `gatling::Codec`/`Sink` per format and drive each CLI through `gatling::run`/`run_typed`, allocating ring slots from `chunk_revolver::ChunkRevolver`; delete the bespoke mpsc plumbing and Vec slot recycling in all four bins. _(effort: L)_
- [x] **[HIGH]** `lbzip2/src/par.rs` is a local no-barrier work-stealing pool replacing gatling/rayon — `lbzip2/src/par.rs` — a hand-rolled scoped-thread work-stealing parallel-map (AtomicUsize claim of `0..n`, UnsafeCell/MaybeUninit result slots, LBZIP2_THREADS sizing) whose own doc says "Replaces lbzip2's old dedicated rayon::ThreadPool" — exactly the role `gatling`/`gatling_forkjoin` own, a private copy one directory below the helper. **Fix:** drive lbzip2 block decode through gatling/gatling_forkjoin like osm-katana; if the dynamic claim-balancing is a genuinely new variant, move `par_map` INTO znippy-zoomies as a shared helper rather than a private copy. _(effort: M)_ — **RESOLVED 2026-07-17:** `lbzip2/src/par.rs` no longer exists (deleted); lbzip2 block decode now fans out through the shared gatling fork-join engine. Finding no longer reflects the tree.
- [ ] **[HIGH]** ljar and lzip-parallel are near-identical twin crates (JAR == ZIP container) — `ljar/src/` vs `lzip-parallel/src/` — central_dir.rs differs by 96/350 lines; batch.rs/parallel.rs/reader.rs are parallel copies; the CLI bins share identical type layouts (FilledSlot, WorkItem, CollectorMsg, WriteMsg, is_ring_slot) — whole-crate duplication of central-dir parsing, entry decode, batching, and the streaming pipeline. **Fix:** collapse into one `zip-core` crate (one set of central_dir/entry/chunk/reader/batch) parameterized for the JAR↔ZIP deltas with thin front-ends, or make ljar depend on lzip-parallel for the shared machinery. _(effort: L)_ — **TODO / deferred (evaluated 2026-07-17):** still open, intentionally NOT consolidated. Twin pair is `ljar/src/{central_dir,deflate_scan,reader,batch,chunk,entry}.rs` ↔ `lzip-parallel/src/{central_dir,deflate_scan,reader,batch,chunk,entry}.rs`. Full extraction into a shared `zip-core` leaf is an L-effort cross-crate refactor: the two are separately-publishable crates with distinct error types (`JarError` vs `ZipError`), CLIs, and bench suites, and lzip-parallel currently carries a pre-existing red unit test (`chunk::tests::decode_chunk_two_segments`, fails on a clean tree independent of this consolidation). No low-risk shared-helper slice lands green in isolation, so this stays a tracked follow-up rather than being shipped red.
- [x] **[MEDIUM]** Sibling codecs diverge into four different parallel-map primitives instead of one helper — `lgz/src/lib.rs` (thread::scope par_map), `ljar/src/lib.rs:39-56` (rayon ThreadPool singleton), `lzip-parallel/src/lib.rs:32-49` (rayon ThreadPool singleton), `lbzip2/src/par.rs` (scoped-thread pool) — the same job (ordered parallel decode of N segments) done four ways; the two `OnceLock<rayon::ThreadPool>` singletons risk oversubscription when composed (osm-katana pulls in lgz + lbzip2 alongside gatling). **Fix:** expose one shared ordered `par_map`/fork-join helper from znippy-zoomies (extend gatling_forkjoin) and have all four crates call it; delete the per-crate `thread_pool()` singletons and lgz's local par_map. _(effort: M)_ — **RESOLVED 2026-07-17:** the two `OnceLock<rayon::ThreadPool>` singletons (ljar, lzip-parallel) and `lbzip2/src/par.rs` are gone; all four codecs now fan out through the one shared `gatling::gatling_forkjoin` engine and rayon is no longer a dependency of any of them. Finding no longer reflects the tree.
- [ ] **[MEDIUM]** Per-codec chunk split/decode logic duplicates the gatling Split/Codec contract — `lgz/src/chunk.rs`, `ljar/src/chunk.rs`, `lzip-parallel/src/chunk.rs`, `lbzip2/src/chunk.rs` — each defines its own `ChunkSplit{segment_starts,decode_segments,consumed}` with split_chunk/decode_chunk + manual carry; lgz's header even says "Mirrors ljar-rs/lbzip2-rs chunk.rs" — exactly `gatling::Split` + `Codec::split/decode` reinvented per codec. **Fix:** re-express each `split_chunk` as `Codec::split → gatling::Split` and `decode_chunk` as `Codec::decode`; remove the bespoke ChunkSplit structs once the CLIs run on gatling. _(effort: M)_
- [ ] **[MEDIUM]** deflate_scan.rs SIMD flush-boundary scanner copied across lgz/ljar/lzip — `lgz/src/deflate_scan.rs`, `ljar/src/deflate_scan.rs`, `lzip-parallel/src/deflate_scan.rs` — the SIMD DEFLATE full-flush boundary scanner (`00 00 FF FF`) is duplicated three times (ljar/lzip differ by ~20 lines, lgz a third variant); all three already depend on the shared DEFLATE crate `linflate`, its natural home. **Fix:** move deflate_scan into linflate, reconcile the lgz variant's extra logic, delete the three local copies. _(effort: M)_
> Cross-ref: the helper-reuse audit also flagged the `curl`/`unzip` test shelling (`ljar`/`lzip-parallel` `tests/maven_artifacts.rs:81,400,599`) — covered above under **no-shelling**.
### cli-ui-symmetry
- [ ] **[HIGH]** `verify` capability has no library twin and exists as two divergent CLI impls — `osm-katana/src/bin/main.rs` (Cmd::Verify + verify_one/count_node_refs/count_geometry/count_refs_for_row_groups, ~119-end) and `osm-katana/src/bin/verify_parquet.rs` (full 61-line standalone reimpl) — GeoParquet verification lives entirely in binaries, never in `osm_katana` lib, so no programmatic/gRPC caller can verify output; worse, it is implemented twice with DIFFERENT behavior (main.rs: footer-stats fast-path + row-group-parallel projected decode; verify_parquet.rs: single-threaded full-column decode) that can report different numbers, and results are only `println!`'d (no Serialize/state_json twin). **Fix:** add `pub fn verify(dir:&Path)->Result<VerifyReport>` (+ `#[derive(Serialize)] VerifyReport` exposing state_json) in a new `osm-katana/src/verify.rs` (the parallel footer-stats impl); make both Cmd::Verify and the bin thin wrappers (or delete verify_parquet). _(effort: M)_
- [ ] **[HIGH]** Codec streaming file→file decompress pipelines exist only inside the CLI binaries, no library API twin — `lgz/src/bin/lgz.rs` (~830), `ljar/src/bin/ljar.rs` (~830), `lzip-parallel/src/bin/lzip.rs` (~674), `lbzip2/src/bin/lbunzip2.rs` (~473); libs expose only in-memory decompress (`lgz/src/lib.rs:73`, `lbzip2/src/parallel.rs:15`, `lzip-parallel/src/lib.rs:63`, `ljar/src/lib.rs:70`) — each CLI inlines the whole 1-reader→N-worker→collector→writer ring-slot streaming pipeline in main(), while the libraries offer only `decompress_*(&[u8])->Vec<u8>`; the bounded-memory streaming capability the CLI provides has no programmatic entry point (and duplicates gatling/chunk_revolver four times). **Fix:** lift the pipeline into each crate's `stream` module as e.g. `pub fn decompress_stream<R:Read,W:Write>(r,w,workers)->Result<u64>`, built on gatling + chunk_revolver; reduce each bin to arg-parse + one call. _(effort: L)_
- [ ] **[MEDIUM]** parquet2arrow bin reimplements Parquet→Arrow inline, diverging from `osm_katana::geo2arrow` — `osm-katana/src/bin/parquet2arrow.rs` vs `osm-katana/src/geo2arrow.rs:36` + `osm-katana/src/bin/main.rs` Cmd::Geo2Arrow — two CLI ways into the same capability: the `geo2arrow` subcommand calls the lib fn, while the auto-discovered `parquet2arrow` bin hand-rolls its own ParquetRecordBatchReader→ArrowFileWriter loop, bypassing the library and able to drift (schema/compression). **Fix:** delete parquet2arrow.rs and keep the subcommand, or generalize the lib fn (`pub fn parquet_to_arrow(src,dst)`) and have both call it. _(effort: S)_
- [ ] **[MEDIUM]** Redundant standalone auto-bins duplicate unified `osm-katana` subcommands — `osm-katana/src/bin/{xml2pbf.rs, pbf_to_parquet.rs, parquet2arrow.rs, verify_parquet.rs}` duplicate main.rs subcommands `{Xml2Pbf, Pbf2Geo, Geo2Arrow, Verify}` — Cargo auto-discovers every `src/bin/*.rs`, shipping four extra CLIs as second entries into capabilities that already have subcommands; a flag/fix added to one path silently misses the other. **Fix:** collapse the benign duplicates into the subcommands (delete xml2pbf.rs/pbf_to_parquet.rs, or mark `required-features`/examples) so each capability has one CLI surface backed by the lib fn. _(effort: S)_
- [ ] **[LOW]** Phase-log telemetry is emit-only: no CLI/programmatic read or facett viz twin, no queryable state_json — `osm-katana/src/phase_log.rs` (PhaseSink writes JSONL to stderr/--log); no log-view subcommand in `osm-katana/src/bin/main.rs` — observability (`wall_s`, `cpu_cores`, `busy_pct`, per-phase counters) is one-way JSONL with no CLI to read/summarize a prior run, no `PhaseEvent` parse twin, no facett/state_json. **Fix:** add `osm-katana log-view <path>` + `pub fn parse_phase_log(path)->Vec<PhaseEvent>` (Serialize) and render the summary through a facett component whose state_json is the parsed events. _(effort: M)_
### test-bench-coverage
- [x] **[HIGH]** Sync gatling core engine (run/run_typed) has zero tests — `src/gatling.rs:278` (run), `src/gatling.rs:559` (run_typed) — the flagship no-barrier worker-pool the whole standard centers on (788 loc, Codec/Sink traits, slot ring, ordered collector, backpressure) has ZERO `#[test]` while its async (gatling_io) and fork-join (gatling_forkjoin) siblings both ship inject-and-assert tests, and its only caller (osm-katana `raw_gatling_run_typed`) is itself untested. **Fix:** add `#[cfg(test)]` with a fake Read + test Codec/Sink asserting REAL output: chunk_id-ordered emission, exactly-once decode, slot recycling (ring_slots < n_chunks completes without deadlock), backpressure, Err short-circuit propagation, observed peak concurrency reaching Config workers; plus a run_typed typed-output test. _(effort: M)_ — **RESOLVED 2026-07-17:** the sync engine (now `gatling/src/gatling.rs`, re-exported as `znippy_zoomies::gatling`) is covered by `tests/gatling.rs`, which drives both `gatling::run` (byte mode) and `gatling::run_typed` (typed mode) over multi-chunk / multi-segment inputs and asserts the assembled output equals a serial baseline byte-for-byte (ordered emission), plus a carry-exceeds-headroom case. Finding no longer reflects the tree.
- [ ] **[HIGH]** chunk_revolver helper (ring + unsafe slot pool) has zero tests — `src/chunk_revolver/ring.rs`, `src/chunk_revolver/revolver.rs` — a named shared helper and the zero-copy slot pool gatling relies on, full of unsafe pointer work (SendPtr unsafe Send+Sync, unsafe `get_chunk_slice`, const-N RingInner variants behind `dispatch!`, try_get_chunk/return_chunk recycling) with ZERO tests — the highest-risk gap on this dimension. **Fix:** unit-test RingBuffer FIFO order/wraparound/is_full/is_empty/capacity at boundaries across each const-N size; ChunkRevolver acquire-until-exhausted (None), return_chunk→re-acquire (recycle), base_ptrs disjoint, and get_chunk_slice round-trips a known injected pattern. _(effort: M)_
- [ ] **[HIGH]** osm-katana XML reader and gatling pipeline driver entirely untested — `osm-katana/src/xml_reader.rs` (2468 loc, 0 tests), `osm-katana/src/xml_to_pbf.rs:413` (raw_gatling_run_typed, 1034 loc, 0 tests); also reader.rs(509), coords_ipc.rs(301), phase_log.rs(305), pbf_enc.rs(267), pbf_decomp.rs(169) all 0 tests — the two largest modules of the flagship gatling consumer (the OSM-XML parse engine and the only place the SYNC gatling engine is wired up), plus the zero-copy coords IPC and the gatling-mandated phase telemetry, are unverified. **Fix:** add an integration test injecting a small known OSM-XML buffer through raw_gatling_run_typed asserting exact decoded node/way records (ids/coords/tags); coords_ipc round-trip (write→mmap-read identical i64/f32); phase_log assertion of recorded phase/cores-busy values. _(effort: L)_
- [ ] **[MEDIUM]** gatling::io tested only with fake sleep jobs, never a live/remote sink — `src/gatling_io.rs:196` (mod tests) — gatling_io exists to be the SAVE-stage pool for network/S3 PUT/GET fan-out, but all three tests drive in-memory tokio-sleep+arithmetic jobs; no test moves a Bytes payload over a socket, so the real S3/HTTP path is untested. **Fix:** add a test spinning up a localhost HTTP server (or in-process mock object store) driving run/run_ordered with real async PUT/GET, asserting bytes received server-side and returned descriptors match. _(effort: M)_
- [ ] **[MEDIUM]** psort parallel sample sort has no bench despite explicit perf claims — `src/psort.rs:62` (samplesort_aos_by_i64_key), `src/psort.rs:257` (radix_sort_aos_by_i64_key) — correctness is well tested but the doc's "sample sort beats fixed-radix LSD" / "below the floor single-threaded sort_unstable wins" claims have NO bench guarding them. **Fix:** add a criterion bench over realistic OSM-id distributions and several N (below/above the parallel floor) comparing samplesort_aos vs radix_sort_aos vs `slice::sort_unstable_by_key`, plus a release-only perf-floor assertion. _(effort: M)_
- [ ] **[MEDIUM]** Sync gatling engine has no throughput/overlap bench — `src/gatling.rs` (run/run_typed), `src/gatling_forkjoin.rs` — the only root benches are xml_scan/json_scan; nothing measures the engine's whole value proposition (no-barrier read/decode/sink overlap). **Fix:** add a gatling bench (synthetic Read + cheap CPU codec + counting sink) measuring `gatling::run` vs single-threaded baseline across worker counts (release-asserting it scales past 1 thread), plus a gatling_forkjoin map/scanline bench. _(effort: M)_
- [ ] **[MEDIUM]** lgz parallel gzip decompressor ships no bench while every sibling codec does — `lgz/` (no benches/ dir); cf. `lbzip2/benches`(3), `linflate/benches`(1), `ljar/benches`(2), `lzip-parallel/benches`(1) — lgz is a "pure Rust parallel gzip decompressor" whose entire point is parallel speed, and the only codec crate with zero benches. **Fix:** add `lgz/benches/inflate_bench.rs` mirroring siblings, with a pigz/Z_FULL_FLUSH corpus (parallel path) and a standard gzip corpus (flate2 fallback), measuring throughput vs single-threaded flate2 across worker counts. _(effort: M)_
- [ ] **[MEDIUM]** osm-katana (12k loc parallel pipeline) has no criterion benches — `osm-katana/` (no benches/ dir); ad-hoc `bin/planet_bench.rs`, `bin/vtd_bench.rs` — the heavy gatling consumer (XML scan, PBF decode/encode, parallel coord sort, Parquet write, optimize) has zero tracked benches; the two "bench" files are hand-rolled bins producing no comparable numbers. **Fix:** add `osm-katana/benches` with criterion per-stage hot paths (xml_reader scan rate, pbf_decomp/pbf_enc, optimize gatling end-to-end on a fixture) reusing the fixture builder; keep the bins as manual planet-scale drivers. _(effort: M)_
- [ ] **[MEDIUM]** .nornir docs reference a non-existent xtask bench harness — `.nornir/stree32.md:105-135`, `.nornir/README.md:13` — documents `xtask/examples/nornir-bench.rs` registering Stree32Bench and `cargo run --release -p xtask --example nornir-bench`, but there is NO xtask crate or nornir-bench example in the repo (members: lbzip2, lgz, linflate, ljar, lzip-parallel, osm-katana, zstd-sys-rs). **Fix:** either add the xtask crate with the documented example, or correct stree32.md/README to point only at the real in-test commands (`cargo test --release stree32_bench -- --nocapture`). _(effort: S)_
- [ ] **[LOW]** stree (i64) perf benches are print-only `#[ignore]` tests with no perf assertion — `src/stree.rs:1517` (bench_compare_all_methods), `:1574` (bench_stree64_batch_variants), `:1609` (bench_compare_1m_records_osm_ids) — they time variants but only `eprintln` throughput, so they cannot fail on regression; the u32 sibling (`stree32.rs:919/937`) already release-asserts `speedup > 1.0/10.0`, but the i64 OSM workload has no such guard. **Fix:** have the i64 bench tests return the measured speedup and add a release-only `assert!(speedup > floor)`, or port to a tracked criterion bench. _(effort: S)_
### docs-vs-code
- [ ] **[MEDIUM]** Dangling `.nornir/stree.md` reference in three places — the file does not exist — `.nornir/README.md:12`, `README.md:14` (generated), `src/stree.rs:42` — all three point readers to `.nornir/stree.md` "for the full benchmark and the comparison to Ragnar's published numbers", but only stree32.md exists; src/stree.rs:42 even attributes its B=8-vs-B=16 measurement to the missing file. **Fix:** recreate `.nornir/stree.md` capturing the STree64 benchmark, or remove/redirect the dead reference at all three sites (point at the in-code bench comment `src/stree.rs:41-47`); edit the `.nornir/README.md` source and regenerate `README.md`. _(effort: S, easy-fix)_
- [ ] **[MEDIUM]** README stree benchmark numbers contradict the figures in src/stree.rs — `.nornir/README.md:11-12` (and generated `README.md:13-14`) vs `src/stree.rs:41-44` — README claims batched `find_exact` ~19 ns/query in cache and ~66-96 ns/query at 1 GB (vs ~1050 ns binary search), but the committed B=8 numbers are ~40 ns/q at 8 MB and ~104 ns/q at 1 GB — roughly 2× the in-cache figure and outside the 1 GB range, and now unsourced since stree.md is gone. **Fix:** reconcile to one canonical set — update `.nornir/README.md` to the actual src/stree.rs figures (or record a newer measurement in code/test and cite it), then regenerate README.md. _(effort: S)_
- [ ] **[MEDIUM]** README omits the flagship gatling family and 6 other public modules — `.nornir/README.md:5-16` (source of generated README.md) — README lists only `vtd`, `stree`, `chunk_revolver`, but lib.rs exports 10 public modules and Cargo.toml's description headlines "generic no-barrier streaming engine (gatling) + parallel sample sort (psort)"; gatling/gatling_forkjoin/gatling_io/psort/json/xml/stree32 are all absent — the marquee feature is undocumented at the front door. **Fix:** add README bullets for gatling (+forkjoin/io), psort, json, xml, stree32 matching `src/lib.rs:3-22` and the Cargo.toml description; edit `.nornir/README.md` and regenerate. _(effort: S)_
- [ ] **[MEDIUM]** chunk_revolver documented as the slot pool "behind the streaming engine" but gatling.rs never uses it — `.nornir/design.md:162-163` (and `:49-56`) — design.md describes chunk_revolver as "the zero-allocation slot pool behind the streaming engine … slot indices cross threads, never Vec<u8> clones", but gatling.rs implements its own slot management (`InFlight.slot: Vec<u8>`, raw `data_ptr`, `src/gatling.rs:163-189`, fill_slot `:211`) and chunk_revolver has zero in-crate uses — it is a standalone primitive consumed by external crates (znippy). **Fix:** reword design.md to state chunk_revolver is a standalone slot-pool primitive used by external consumers while gatling manages its own Vec slots — OR, if intended, make gatling.rs actually consume chunk_revolver. _(effort: S)_
- [ ] **[LOW]** stree32.md "warehouse harness" section references an xtask that does not exist in this repo — `.nornir/stree32.md:107-110` & `:135` — says "`xtask/examples/nornir-bench.rs` registers `Stree32Bench`" with `NORNIR_RUN_HEAVY=1 cargo run --release -p xtask --example nornir-bench -- --only znippy.stree32_512mb`, but this leaf repo has no xtask crate; the harness lives in the consumer (nornir) repo. **Fix:** mark the warehouse-harness paragraph as cross-repo (consumer/nornir), consistent with how design.md flags sibling-repo docs; the in-repo `cargo test stree32_bench` commands are correct and can stay. _(effort: S, easy-fix)_
- [ ] **[LOW]** gatling-guide TypedSink trait block omits the `finish()` method — `.nornir/gatling-guide.md:208-211` vs `src/gatling.rs:119-126` — the guide's `TypedSink<T>` shows only `fn process(&mut self, out: T, is_last: bool)`, but the real trait also defines `fn finish(&mut self) -> Result<()> { Ok(()) }` (called once after all segments), so a reader won't know the post-stream flush hook exists. **Fix:** add the `fn finish(&mut self) -> Result<()> { Ok(()) }` line (with its comment) to the TypedSink block to match `src/gatling.rs:125`. _(effort: S, easy-fix)_
### dead-lost-code
- [ ] **[HIGH]** Entire node_store_arrow module is orphaned (superseded Arrow-IPC node store) — `osm-katana/src/node_store_arrow.rs` (539 lines, declared `osm-katana/src/lib.rs:6 pub mod node_store_arrow;`) — a complete Arrow-IPC node store (ArrowNodeStore, ArrowCoordWriter, convert_soa_to_arrow/convert_flat_to_arrow, open/open_mmap/lookup/lookup_batch + its own 6 tests) that NOTHING in production references (only the `pub mod` decl + its own test block); the live path is `node_store.rs CoordFileWriter::finalize → external_merge_sort_inner → coords_ipc ArrowMmap`, which hand-rolls the same single-RecordBatch IPC layout. dead_code missed it because the fns are `pub` and reachable from its own tests. **Fix:** confirm coords_ipc fully replaces it (it does), then delete the module and remove the `pub mod` from lib.rs; if its pyarrow/DuckDB-readable + per-batch LZ4/ZSTD advantages are wanted instead, wire ArrowCoordWriter into CoordFileWriter and delete the coords_ipc path — but do not keep both. _(effort: M)_
- [ ] **[HIGH]** Dead disk-merge node-store path (MmapTree / from_mmap / merge_sorted_chunks / CoordFileWriter::push) — `osm-katana/src/node_store.rs:31` (Inner::MmapTree), `:55` (from_mmap), `:200` (push), `:745-905` (merge_sorted_chunks) — a whole pre-ArrowMmap K-way disk-merge path is dead: `Inner::MmapTree` never constructed, `from_mmap` only called from inside `merge_sorted_chunks` (`:771/:901`) which itself has no external caller, and `CoordFileWriter::push` superseded by the bulk `write_raw` sink; the live finalize uses `external_merge_sort_inner` (Inner::ArrowMmap). ~170 lines kept alive only by mutual reference. **Fix:** delete `merge_sorted_chunks` + its private helpers, `NodeStore::from_mmap`, the `Inner::MmapTree{tree,mmap}` variant, and `CoordFileWriter::push`; re-run cargo to clean up newly-dead STree64Mmap/merge helpers. _(effort: M)_
- [ ] **[MEDIUM]** reader.rs blob_coords and blob_node_records superseded by fused blob_nodes_and_coords — `osm-katana/src/reader.rs:59` (blob_coords), `:71` (blob_node_records) — both `pub fn` flagged never-used; they were the decoupled per-blob decoders now replaced by `blob_nodes_and_coords` (~`:82`), which fuses coord extraction + node-record encoding into a single protobuf parse for the Gatling PBF pass-1 codec. **Fix:** delete `blob_coords` and `blob_node_records`; confirm `parse_node_coords`/`parse_nodes` still have other callers and remove `parse_node_coords` if it was only used by blob_coords. _(effort: S)_
- [ ] **[LOW]** geometry.rs encode_point and is_closed_area_slice are dead (live variants are *_inline / non-slice) — `osm-katana/src/geometry.rs:21` (encode_point), `:43` (is_closed_area_slice) — cargo flags both never-used; the pipeline uses `encode_point_inline` (`[u8;21]`, called from writer.rs ×3) and `is_closed_area` (the tag-map variant). **Fix:** delete `encode_point` and `is_closed_area_slice`. _(effort: S, easy-fix)_ _(note: conflicts with the zero-copy-alloc fix that wants to adopt `is_closed_area_slice` — resolve by either keeping+using the slice variant or porting its logic; do not blindly delete if adopting it.)_
- [ ] **[LOW]** Never-read struct fields: chunk is_last, coords_ipc count, Pass2TypedSink include_rels — `lbzip2/src/chunk.rs:39` (is_last), `osm-katana/src/coords_ipc.rs:84` (count), `osm-katana/src/xml_reader.rs:1951` (include_rels) — three fields constructed but never read (logic uses the local `is_last` param / passes count separately / gates via `rel_writer.is_some()`), misleading readers about live state. **Fix:** remove each unused field + its assignment; if intended for future read, add a doc note + `#[allow(dead_code, reason=...)]` instead of a silent warning. _(effort: S, easy-fix)_
- [ ] **[LOW]** Unused imports left in source (Write, NamedTempFile, lbzip2::block) — `osm-katana/src/node_store.rs:949` (std::io::Write), `osm-katana/src/node_store_arrow.rs:396` (tempfile::NamedTempFile), `lbzip2/benches/stage_breakdown.rs:6` & `lbzip2/benches/profile_stages.rs:7` (use lbzip2::block) — pure compiler-warning clutter that hides real warnings. **Fix:** delete the unused imports (or `cargo fix`); node_store_arrow.rs:396 disappears anyway if that module is deleted. _(effort: S, easy-fix)_
- [ ] **[LOW]** lbzip2 bench files registered as multiple build targets — `lbzip2/Cargo.toml` (decompress_bench.rs, profile_stages.rs, stage_breakdown.rs) — each is both explicitly listed under `[[bench]]` AND auto-discovered from benches/, so each compiles twice with warnings on every build. **Fix:** set `autobenches = false` (or remove the redundant explicit `[[bench]]` entries) so each bench is a single target. _(effort: S, easy-fix)_
> Cross-ref: the dead-code audit independently flagged ljar as a near-verbatim copy of lzip-parallel (`ljar/src/*` vs `lzip-parallel/src/*`) — covered above under **helper-reuse** (JAR == ZIP twin crates).
## Tier 2 — Excellence (algorithms/datatypes)
- [ ] **[MEDIUM]** Concatenated-gzip parallel decode uses foreign flate2 instead of the in-house linflate fast loop — `lgz/src/speculative.rs:407-432` (line 426) — `decode_concatenated_members` is the parallel fast path for multi-member .gz, yet decodes each member with `flate2::read::GzDecoder` (zlib-ng/miniz) — but the crate exists for linflate's SIMD preload/burst loop, which its own bench (`ljar/src/bin/bench_inflate_cmp.rs`) reports beats zlib-ng; parallel-but-foreign leaves per-core speedup on the table. **Fix:** decode each member through linflate (strip the 10-byte gzip header + optional extras, feed raw DEFLATE to linflate's segment decoder, as `lgz/src/chunk.rs` already does), keeping flate2 only as the genuine error fallback. _(effort: M)_
- [ ] **[MEDIUM]** Inverse-BWT test asserts only output length, never the reconstructed bytes — `lbzip2/src/bwt.rs:48-68` — `inverse_bwt` is the single most correctness-critical bzip2 step, but its only test feeds a contrived array and asserts `output.len()==6` ("we just verify it doesn't panic") — the forbidden "didn't panic" anti-pattern; a wrong T-vector permutation still yields 6 bytes and passes. **Fix:** inject a real BWT output + orig_ptr for a known string (e.g. 'banana' → 'annb$aa' with its documented orig_ptr) and assert `next_byte()` reproduces the exact original bytes; add a property test that bwt-encode→inverse round-trips random inputs. _(effort: S, easy-fix)_
- [ ] **[LOW]** Streaming gatling fans work through Arc<Mutex<mpsc::Receiver>> instead of a lock-free MPMC channel — `src/gatling.rs:342-357` (and run_typed ~`:627-640`) — units are distributed to N workers via `mpsc::sync_channel` whose single Receiver is shared as `Arc::new(Mutex::new(work_rx))`, so every worker takes a global lock to `recv()`; std mpsc is the slowest channel and the Mutex serializes dispatch, contradicting design.md's (`:28,:35`) "no central scheduler / self-dispatch" story that the forkjoin/psort variant honors with an atomic fetch_add. Impact is muted because decode units are coarse. **Fix:** replace the Arc<Mutex<mpsc>> work queue with a lock-free MPMC channel (crossbeam-channel or flume) — Sync multi-consumer receivers with no mutex. _(effort: M)_
- [ ] **[LOW]** Byte-mode gatling allocates a fresh Vec<u8> per decode unit, bypassing the slot-pool philosophy — `src/gatling.rs:70` (Codec::decode → Vec<u8>), consumed `:364-369` — every worker returns a newly heap-allocated `Vec<u8>` per segment and the collector concatenates them, so the output side allocates+frees one Vec per unit in the steady-state hot loop of an engine that ships chunk_revolver specifically to avoid this (codecs already use thread-local reusable buffers internally, then copy into the returned Vec). **Fix:** offer a `decode_into(&mut Vec<u8>)` / pooled-output Codec variant so workers write into a recycled buffer leased from a small per-worker pool (or chunk_revolver ring) returned after the collector consumes it. _(effort: M)_
- [ ] **[LOW]** Speculative decode materializes every segment as its own Vec then copies all into a combined buffer — `lgz/src/speculative.rs:286-322` — decodes each segment into a separate `Vec<u8>` (pass 1), rewrites prefixes in place (pass 2), then allocates a fresh `output` and `extend_from_slice`s every segment (`:316-321`) — a second full pass over the entire decompressed output plus peak RAM of (all segments + combined copy), ~2× peak memory for multi-GB decompressions. **Fix:** after pass 1 the per-segment lengths are known — allocate the final buffer once and have each worker decode/fix-up directly into its computed offset range via `inflate_segment_with_prefix` into a shared output slice, eliminating the concatenation copy. _(effort: M)_
- [ ] **[LOW]** AVX2 node compare uses movemask_epi8 + popcount/8 instead of the 4-bit movemask_pd — `src/stree.rs:612-626` — `count_lt_avx2` reduces each 256-bit `_mm256_cmpgt_epi64` with `_mm256_movemask_epi8` (32 bits, 8 duplicated per lane) then `count_ones()/8`, wasting the redundant-byte expansion and a divide on the hottest inner primitive of the search tree. **Fix:** cast with `_mm256_castsi256_pd` and use `_mm256_movemask_pd` (exactly 1 bit per i64 lane); OR the (B/4) masks and a single `count_ones()` gives the count directly — no `/8`, same latency. _(effort: S, easy-fix)_
- [ ] **[LOW]** run_ordered reorder buffer is a heap-allocating BTreeMap when a fixed ring would be O(1) alloc-free — `src/gatling_io.rs:164-187` — out-of-order completions buffer in `BTreeMap<usize,R>` (node alloc + O(log n) per insert/remove), even though the buffer is provably bounded by `max_in_flight` and indices are contiguous from `next_emit`. **Fix:** replace with a `Vec<Option<R>>`/VecDeque of length cap indexed `idx % cap`; insert at `idx % cap`, drain while `slot[next_emit % cap]` is Some — O(1), zero per-item alloc. _(effort: S, easy-fix)_
- [ ] **[LOW]** OSM integer/coord parsing is a scalar per-byte digit loop, missing SWAR fast parse — `src/vtd.rs:153-177` (parse_coord_e7) and `src/xml.rs:169-181` (parse_i64) — id/lat/lon parsing walks one ASCII byte at a time with a per-byte `is_ascii_digit` branch + mul-add; at planet scale over hundreds of millions of matched elements this is the per-element decode cost, while simdjson/fast_float parse 8 digits at once with a couple of SWAR multiplies. **Fix:** add a SWAR 8-digit parser (load 8 bytes, subtract '0', masked-compare validate, fold with the 1+10+100… trick) for the common fixed-width fast path, falling back to scalar for short/odd lengths. _(effort: M)_
- [ ] **[LOW]** bzip2 Huffman long-code path bit-walks a binary tree instead of a second-level table — `lbzip2/src/huffman.rs:131-157` — codes longer than FAST_BITS (10) fall to `decode_slow`, a bit-by-bit pointer-chase down `nodes[node_idx][bit]` (one dependent load + one bit read per bit, up to 20 levels) — the least cache-friendly option, though a rare path. **Fix:** replace the node-array fallback with a second-level lookup table (offset+sub-bits, same shape linflate already uses via HUFFDEC_SUBTABLE) so long codes resolve in one extra table read. _(effort: M)_
- [ ] **[LOW]** Library decode paths print progress with eprintln! instead of a log facade — `lgz/src/speculative.rs:413`; `lgz/src/parallel.rs` and `lgz/src/bin/lgz.rs` (multiple) — reusable library decode fns (not just the bin) emit `eprintln!` diagnostics (e.g. "lgz: found N concatenated gzip members"), giving embedding consumers unsolicited stderr spam they cannot route/silence, plus a lock+syscall on the decode path. **Fix:** route diagnostics through the `log`/`tracing` facade (or return structured stats), keeping raw `eprintln!` confined to the CLI binary. _(effort: S, easy-fix)_
## Easy fixes (safe to apply now)
- [ ] `.nornir/README.md:12`, `README.md:14`, `src/stree.rs:42` — remove/redirect the dead `.nornir/stree.md` reference (edit the .nornir source, regenerate README). _(docs-vs-code)_
- [ ] `.nornir/stree32.md:107-110,135` — mark the xtask "warehouse harness" paragraph as cross-repo (consumer/nornir). _(docs-vs-code)_
- [ ] `.nornir/gatling-guide.md:208-211` — add the `fn finish(&mut self) -> Result<()> { Ok(()) }` line to the TypedSink block. _(docs-vs-code)_
- [ ] `osm-katana/src/geometry.rs:21,:43` — delete dead `encode_point` and `is_closed_area_slice` (but reconcile with the zero-copy fix that wants to adopt `is_closed_area_slice`). _(dead-lost-code)_
- [ ] `lbzip2/src/chunk.rs:39`, `osm-katana/src/coords_ipc.rs:84`, `osm-katana/src/xml_reader.rs:1951` — remove the never-read struct fields (is_last / count / include_rels). _(dead-lost-code)_
- [ ] `osm-katana/src/node_store.rs:949`, `osm-katana/src/node_store_arrow.rs:396`, `lbzip2/benches/stage_breakdown.rs:6`, `lbzip2/benches/profile_stages.rs:7` — delete unused imports. _(dead-lost-code)_
- [ ] `lbzip2/Cargo.toml` — set `autobenches = false` (or drop the redundant `[[bench]]` entries) so benches build once. _(dead-lost-code)_
- [ ] `lbzip2/src/bwt.rs:48-68` — make the inverse-BWT test assert exact reconstructed bytes for a known string + add a round-trip property test. _(tier2)_
- [ ] `src/stree.rs:612-626` — switch `count_lt_avx2` to `_mm256_movemask_pd` (no `/8`). _(tier2)_
- [ ] `src/gatling_io.rs:164-187` — replace the run_ordered `BTreeMap` reorder buffer with a fixed-capacity ring. _(tier2)_
## Counts
- **High:** 14
- **Medium:** 22
- **Low:** 18
- **Total:** 54 (after de-duping the `curl`/`unzip` test-shelling finding and the ljar==lzip twin-crate finding across dimensions)