# osm-katana — reasoning
> **Class:** `reasoning` · **Function:** `osm-katana convert`. Why the OSM → GeoParquet
> pipeline has exactly two passes with one barrier, and why that barrier is a property
> of the data model rather than a saturation bug. Shipped shape in
> [`osm-katana-design.md`](osm-katana-design.md).
## Why two passes at all
OSM is a reference graph, not a table. A way is a list of node IDs; a node carries the
only copy of the coordinate. To emit a way's geometry you must dereference every node
ID it names — and those nodes can sit anywhere in the file, before or after the way. So
the converter cannot stream ways straight to disk in one linear scan: it must know the
whole node index first. That is the entire reason for the split into pass 1 (harvest
every node's coordinate) and pass 2 (resolve ways/relations against the harvested
index). It is a consequence of the format, the same way gzip's single LZ77 window
forces a serial fallback in lgz — the shape is dictated by the data.
## Why the barrier between the passes is inherent, not a stall
Between the passes sits the node store build (`external_merge_sort`,
`node_store.rs:250`), and pass 2 genuinely cannot begin until it finishes: the first
way to resolve might reference the numerically-last node in the file, so *no* way is
safe to resolve until the *complete* sorted index exists. This is a hard happens-before
edge, not a scheduling accident. Calling it a "single-core stage" and hunting it as a
saturation bug would be a category error.
What we *did* refuse to accept is a barrier that is also single-threaded. The sort
itself fans across cores on gatling (`node_store.rs`), heaviest
chunk first (LPT), so the barrier is a short all-core burst rather than one core
grinding while eleven idle. The serial-hunter law applies within the stage even though
the stage boundary is serial by nature: a barrier you must have is fine; a barrier that
wastes cores while you wait is not.
## Why the clip runs inside the workers
Region clip (`--clip`/`--poly`) is a per-node geographic test. The naive design runs it
as a filtering pass over the coordinates; that is a second linear scan and a second
place for one core to bottleneck. Instead the clip is folded into the pass-1 worker at
the exact point coordinates are decoded (`clip_keeps`, `lib.rs:401`, used
`reader.rs:277`) — every core that decodes a blob also clips it, no extra pass. And when
no clip is set the `if let Some` branch folds away, so the common unclipped convert
pays nothing for the feature (`lib.rs:100-107`).
## Why compression lives on the workers, not the writer
The tempting design puts one writer thread at the end that receives rows, builds Arrow
row groups, zstd-compresses them, and appends to the parquet file. That is what shipped
first, and it *pegged pass-2 at ~5 cores on planet*: the workers finished their parse
and blocked on `collector_tx.send` while the lone collector sat inside a parquet flush
doing the compression for everyone (`xml_reader.rs:2025-2037`). Compression is the
heavy per-row-group cost, and putting it on a single thread converts an all-core stage
into a one-core queue.
The fix moves the row-group build **and** the zstd into the pass-2 workers via
per-worker `thread_local` accumulators feeding a `Clone + Sync` `ColumnEncoder`
(`writer.rs:665`). The workers hand the writer *already-compressed* `EncodedRowGroup`s;
the writer's job shrinks to stitching them in order and writing the footer. That is why
Stage 4 can stay a single writer without throttling the pipeline — there is no
expensive work left on it.
## Why the final writer is still serial (and that's correct)
Parquet's file structure is inherently single-writer: row groups append sequentially
and the footer indexes their byte offsets, so two threads cannot append concurrently
without a lock that serializes them anyway. On top of that we want **deterministic row
order** — the same input must always produce byte-identical output — which requires the
collector to deliver segments in index order (`Pass1TypedSink` / `Pass2TypedSink`). A
serial stitch is the honest shape for both constraints. The few boundary elements that
straddle two segments are parsed on the collector via `flush_stub`
(`xml_reader.rs:1951` / `:2080`); this is a handful of rows per segment boundary,
negligible next to the workers' bulk. Keeping the stitch cheap (no compression) is what
lets it stay serial without becoming the bottleneck — the same discipline as the
barrier: unavoidable serial work is fine only when it is also cheap.
## Why one engine for four input formats
PBF, bzip2, gzip, and XML look nothing alike on disk, but the converter routes all four
through the same `gatling::run_typed` engine with per-format codecs
(`PbfPass1TypedCodec`, `Bz2Pass1TypedCodec`, `GzPass1TypedCodec`, `Pass1ResolvedCodec`).
The engine owns the split/worker/ordered-collector machinery; a codec only teaches it
how to find segment boundaries and decode one segment. bzip2 splits at independent block
magics, PBF at blob frames (`pbf_split` makes ~2× workers coarse ranges for
load-balance headroom, `xml_to_pbf.rs:715-717`), gzip at full-flush/bgzf boundaries or
one segment if none exist. This is deliberate reuse: the parallel structure is written
once and the awkward per-format decode is quarantined in the codec.
## Honest saturation note
Both passes are all-core by construction; only the sort barrier and the writer stitch
are serial, and both are so by nature. We do **not** claim a `cores_busy` number in
these docs — the one weighted-LPT path with a concrete balance story
(`gatling_for_each_balanced`, `reader.rs:138`) serves the separate `load_ways_pbf`
helper, not the convert CLI. Real all-core throughput and peak-RSS numbers for
`osm-katana convert` land in the [`README.md`](README.md) `zoomies-katana` mashup once
the quiet-box planet run reaches the warehouse — recorded there against `osmium` /
`osm2pgsql`, not fabricated here.