# Scanners (json / xml / vtd) — reasoning
> **Class:** `reasoning` · **Function:** `scanners`. Why oversplit, why boundary-aligned,
> and the honest note on what "xml/vtd oversplit" really means. Shipped shape in
> [`scanners-design.md`](scanners-design.md).
## Why boundary-aligned split beats byte-equal split
You cannot cut a text record stream at an arbitrary byte — a JSON line or an XML element
would be sliced in half and both halves would be garbage. So each scanner seeks its
nominal cut forward to the next *legal* boundary: json to the next raw `\n` (illegal
inside a JSON string per RFC 8259 §7, so always a true record edge), xml/vtd to the next
top-level element start (`memchr('<')` + a tag-name match). Borders are monotonic, so no
record straddles a cut and adjacent-equal borders simply yield an empty unit — the split
is correct by construction, and the parallel scan is byte-identical to the serial one.
## Why oversplit (the real saturation lesson)
The old shape cut the file into exactly `n` byte-equal slices and ran `n` workers, one
slice each, no stealing. A slice that happened to land on a *dense* region (many small
records, or one enormous element) left its worker grinding alone while the others
drained their light slices and sat idle — the classic skew stall the core-saturation law
forbids. The fix is **oversplit**: cut into many more units than cores (`units = n × 8`)
and let `gatling_for_each`'s self-dispatch hand the next unit to whichever worker is
free. A heavy unit no longer strands a core; its siblings steal the tail. `SPLIT = 8` is
the tuned sweet spot — enough granularity to balance, not so fine that the boundary
seeks fragment (`json.rs:148`, `xml.rs:409`, `vtd.rs:687`).
## The honest "xml/vtd oversplit" note
All three scanners use the **same** `SPLIT = 8` factor — json is not split more coarsely.
What distinguishes xml/vtd is not a finer numeric factor but *more elaborate machinery*
around the same oversplit:
- **xml gates the parallel path behind a 4 MiB threshold** (`xml.rs:402`); below it the
serial `scan_slice` wins (no pool spin-up for a small document). json has no such gate.
- **vtd adds planet-scale collectors** on top: the `revolver` path over-provisions
`n_chunks = n * 5 / 4` (`vtd.rs:479`, "+25% keeps cores busy during commits"), and the
`pipelined` path re-splits each 100 MB I/O-ring slot into `n × 8` sub-chunks
(`vtd.rs:687`). These exist because a planet OSM file cannot be held resident, so the
index must be committed incrementally (mmap prefix-sum, BTreeMap committer, or
sequential slots) while still keeping every core fed.
So "xml/vtd oversplit" is accurate in spirit — they carry the heavier oversplit-under-skew
commentary and the streaming committers — but the split *factor* is 8× across the board.
This doc records that straight rather than implying a fabricated difference.
## Why zero-copy spans
The output is `(offset, len)` into the caller's buffer, never a copy of the record. A
planet OSM scan touching hundreds of millions of elements cannot afford to allocate a
`String` per element; the span index lets downstream code borrow the bytes it actually
needs (`vtd`'s `ElemIndex` is a 32-byte `#[repr(C)]` row, mmap-friendly). That is the
gatling soul applied to reading: distribute by shared borrow, collect disjoint results,
copy nothing on the hot path.