<!-- nornir:class: design -->
# Single-pass VTD — discard-until-boundary-then-store
**Rickard's idea.** The planet is **bz2 (non-seekable)** — the changesets + whole node prologue
(~86 GB) MUST be decompressed sequentially anyway. The two-pass VTD ([`crate::vtd`], and
osm-katana's `seek_past_nodes`) assumes a *seekable* mmap and re-seeks past the nodes, wasting
that mandatory leading decompression. The single-pass makes it do double-duty: one stream-order
pass that **scans the node prologue (index-only, no store)** and, at the **node→way boundary**,
**FLIPS to storing** the VTD `ElemIndex` for ways + relations — no seek, no re-decompress.
## The flip
```
bz2 decompressor ──chunks(stream order)──▶ StreamingBoundaryDetector
│ monotonic <way / <relation (= WAYREL_NEEDLES)
│ cross-chunk straddle carried (MAX_NEEDLE-1)
┌──────────────── before flip ────────────────┼──────────── after flip ───────────────┐
▼ ▼ ▼
scan node prologue node→way boundary store ElemIndex
(count, DISCARD — no store) (first <way offset) (ways + relations)
offsets absolute in-stream
```
- **Seek-free:** boundary found by streaming the same monotonic `<way `/`<relation ` predicate as
`seek_past_nodes`, chunk-by-chunk, with a `MAX_NEEDLE-1` straddle carry → O(1) memory, no mmap.
- **Byte-identical:** the stored way/relation entries match a full two-pass scan of the same bytes
(offset · length · id · kind · tag_flags) — proven in `vtd_single_pass::tests`.
## Status
- `StreamingBoundaryDetector` (the seek-free flip) — REAL + tested (4 tests, incl. cross-chunk edges).
- `single_pass_index(chunks, store)` wires detect → discard-prologue → store-way/rel. Store side
currently buffers the post-boundary region before scanning (correct/byte-identical). **Remaining
refinement:** incremental post-boundary store (per-element carry) so the full planet never buffers
the way/relation tail. The detector already proves the seek-free flip — the rest is *adding
incremental scan*, not a rewrite.
- **Opt-in** (`store` flag) — changes no default path, so the WORLD_RECORD_DECOMPRESSORS discipline holds.
## Build + compare protocol (HEAVY — deferred to the quiet bench box)
Baseline = the current two-pass full-planet convert. Then, on a QUIET box (Loki/Odin): run the SAME
planet with the single-pass, quiet → compare wall-time + phase-log. **Keep only if FASTER AND
byte-identical** (roundtrip-verify the parquet/vtd matches the baseline). If not faster, keep behind
the flag + record the honest result. Bench data → `/home`; never a heavy run on a busy box.
See also [[WORLD_RECORD_DECOMPRESSORS]].