Gunnar

nordisk/znippy-zoomies

.nornir/scanners-guide.md

# Scanners (json / xml / vtd) — guide

> **Class:** `guide` · **Function:** `scanners`. How to call the three parallel record
> scanners. Shape in [`scanners-design.md`](scanners-design.md); why oversplit in
> [`scanners-reasoning.md`](scanners-reasoning.md).

## Build

```sh
CARGO_TARGET_DIR=/mnt/work4t/cargo-target/zoomies cargo build --release
```

All three live in the root `znippy-zoomies` crate (`src/json.rs`, `src/xml.rs`,
`src/vtd.rs`).

## Call them

```rust
// NDJSON: count records, get a zero-copy span per line.
let n = znippy_zoomies::json::scan_records_parallel(&bytes, 0, |span| {
    let rec = &bytes[span.offset..span.offset + span.len];   // borrowed, no copy
});                                                           // json.rs:127

// XML: scan top-level elements, map each span to your own T on the worker.
let tags = /* TagSet of element names */;
znippy_zoomies::xml::scan_parallel(&bytes, 0, &tags, |kind, span| my_map(kind, span),
                                   |t| collect(t))?;          // xml.rs:387

// OSM: build a file_offset-sorted ElemIndex (pick the collector by scale).
znippy_zoomies::vtd::build_elem_index_parallel(&bytes, 0, |ix| push(ix))?;   // vtd.rs:284
znippy_zoomies::vtd::build_elem_index_to_mmap(&bytes, 0, idx_path)?;         // planet, mmap
znippy_zoomies::vtd::build_elem_index_pipelined(path, 0, idx_path)?;         // I/O-ring, streaming
```

`n_workers = 0` ⇒ one per core (the gatling default). Below 4 MiB, `xml`/`vtd` run
serially on their own (no pool spin-up).

## Which vtd entry

- `build_elem_index_parallel` — in-memory buffer, callback per element.
- `build_elem_index_to_mmap` — planet-scale, writes the index straight to an mmap at
  prefix-sum offsets.
- `build_elem_index_revolver` — mpsc + BTreeMap committer, over-provisioned chunks.
- `build_elem_index_pipelined` — reads from a path via an I/O-ring, 100 MB slots.

## Bench

`cargo bench` runs `benches/json_scan.rs` and `benches/xml_scan.rs`; the scanners also
ride the `osm-katana` convert bench (they are its XML front-end).