# Scanners (json / xml / vtd) — design
> **Class:** `design` · **Function:** `scanners` — the three zero-copy parallel record
> scanners that fan through the gatling fork-join engine. Strict: shipped shape, every
> claim a real `file:symbol`. Run them via [`scanners-guide.md`](scanners-guide.md);
> the oversplit rationale in [`scanners-reasoning.md`](scanners-reasoning.md). Engine:
> [`gatling-design.md`](gatling-design.md).
## What they are
Three scanners in the root `znippy-zoomies` crate that turn a byte buffer into a
**zero-copy span index** — byte `(offset, len)` pointers into the original input, never
a copy of the records. They are not decompressors; they are the *reading* half that a
decompressed stream feeds. All three share one shape: split at legal record boundaries,
oversplit ~8× the core count, self-dispatch the per-unit parse through
`gatling_forkjoin::gatling_for_each` (`gatling/src/gatling_forkjoin.rs:60`), and merge
partial results in unit order — deterministic and byte-identical to a serial scan for
any worker count.
## json — NDJSON / JSON-Lines (`src/json.rs`)
`scan_records_parallel(bytes, n_workers, on_record) -> u64` (`src/json.rs:127`).
Boundaries are raw `\n`: per RFC 8259 §7 a literal newline is illegal inside a JSON
string, so **every raw `\n` is a record boundary** — found with `memchr`, no parsing
needed. It oversplits into `chunks = (n * SPLIT)` units (`SPLIT = 8`, `json.rs:148`),
each candidate `i * chunk_size` forward-seeked past its next `\n` by `find_safe_split`
(`json.rs:55`); borders are monotonic so no record straddles a cut (a candidate landing
inside a prior record yields an empty unit). The fan-out is `gatling_for_each` at
`json.rs:165`, returning `Vec<Vec<RecordSpan>>` in slice order, merged to `on_record` on
the calling thread. Output span: `RecordSpan { offset, len }` (`json.rs:36`), the
newline and trailing `\r` excluded. No size threshold — the parallel path runs whenever
`n > 1`.
## xml — generic VTD-style element scanner (`src/xml.rs`)
`scan_parallel(bytes, n_workers, tags, map, on_item) -> Result<u64>` (`src/xml.rs:387`),
with `map: Fn(u8, ElemSpan) -> T + Sync`. Boundaries are top-level element starts:
`find_top_level_start` (`xml.rs:191`) does `memchr(b'<')` then `tags.starts_element`
(name followed by whitespace). It oversplits into `units = (n * SPLIT)` (`SPLIT = 8`,
`xml.rs:409`); `border[i]..border[i+1]` bound each unit so no element straddles a cut.
Fan-out is `gatling_for_each` at `xml.rs:421`; `map` runs **on the worker**, partials
merged in file order. The parallel path is gated behind a **4 MiB threshold**
(`xml.rs:402`) — below that it runs the serial `scan_slice`. Output span:
`ElemSpan { offset, len, tag_len, self_closing }` (`xml.rs:101`).
## vtd — OSM specialisation + planet-scale writers (`src/vtd.rs`)
Layers OSM decoding (`<node>`/`<way>`/`<relation>`) on `xml`, exposing **four** entry
points across **three** collection engines, all producing a `file_offset`-sorted
`ElemIndex` (32-byte `#[repr(C)]`, `vtd.rs:104`):
- `build_elem_index_parallel` (`vtd.rs:284`) — delegates straight to `xml::scan_parallel`
(`vtd.rs:292`), mapping each span to an `ElemIndex`.
- `build_elem_index_to_mmap` (`vtd.rs:347`) — `std::thread::scope` two-scope rendezvous:
a parallel count, then each thread writes its elements to its **prefix-sum offset** in
a shared mmap (file order by construction).
- `build_elem_index_revolver` (`vtd.rs:468`) — `std::thread::scope` + mpsc + a `BTreeMap`
committer draining `(chunk_index, Vec)` in file order; `n_chunks = n * 5 / 4`
(`vtd.rs:479`, "+25% over-provision keeps cores busy during commits").
- `build_elem_index_pipelined` (`vtd.rs:599`) — an I/O-ring reader thread emits 100 MB
slots (`SLOT_BYTES`, `RING_DEPTH = 4`) trimmed at `find_safe_slot_end`; each slot is
re-split into `units = n * SPLIT` sub-chunks and fanned through `gatling_for_each`
(`vtd.rs:695`), slots processed sequentially so output stays in file order with no
sort.
## Whereabouts
| Thing | file:symbol |
|---|---|
| NDJSON parallel scan | `src/json.rs:127` (`scan_records_parallel`), fan `:165` |
| json boundary seek | `src/json.rs:55` (`find_safe_split`) |
| XML parallel scan | `src/xml.rs:387` (`scan_parallel`), fan `:421` |
| xml boundary seek | `src/xml.rs:191` (`find_top_level_start`) |
| vtd OSM entries | `src/vtd.rs:284` / `:347` / `:468` / `:599` |
| shared fan-out | `gatling/src/gatling_forkjoin.rs:60` (`gatling_for_each`) |
## Saturation state (just-landed)
All three **oversplit by `SPLIT = 8`** so a dense region cannot strand one worker — the
self-dispatch engine lets siblings steal the tail. json takes the parallel path
unconditionally (`n > 1`); xml/vtd gate behind 4 MiB and carry the elaborate
oversplit-under-skew machinery (rendezvous / revolver / pipelined committers) for
planet-scale inputs. No `cores_busy` is fabricated here; the honest figures ride the
`osm-katana` bench wave (the scanners are that converter's XML front-end).