# osm-katana — design
> **Class:** `design` · **Function:** `osm-katana convert` (OSM → GeoParquet). Strict:
> the shape we ship, every claim carrying a real `file:symbol`. How to run it in
> [`osm-katana-guide.md`](osm-katana-guide.md); why the pipeline has the shape it does
> in [`osm-katana-reasoning.md`](osm-katana-reasoning.md). The streaming engine is
> [`gatling-design.md`](gatling-design.md).
## What it is
`osm-katana convert` turns an OSM extract — XML (`.osm`), gzip (`.osm.gz`), bzip2
(`.osm.bz2`), or PBF (`.osm.pbf`) — into **GeoParquet**: three files, `nodes.parquet`,
`ways.parquet`, `relations.parquet`, all-core on the shared gatling engine. The CLI
handler is `Cmd::Convert` (`osm-katana/src/bin/main.rs:183`), which builds a
`ConvertOptions` and calls `osm_katana::convert` (`main.rs:193`). The library entry is
`convert` (`osm-katana/src/lib.rs:483`) → `convert_inner` (`lib.rs:508`), which
dispatches by input extension and geometry mode (`raw` vs the default `resolved`) to a
reader in `osm-katana/src/xml_reader.rs`:
| input | reader (`xml_reader.rs`) |
|---|---|
| `.osm.pbf` | `read_resolved_pbf` (`:2442`) |
| `.osm.bz2` | `read_resolved_bz2` (`:2196`) |
| `.osm.gz` | `read_resolved_gz` (`:2337`) |
| `.osm` (XML) | `read_resolved_osm` (`:437`) |
Default worker count is `available_parallelism() − 1` (`lib.rs:488`).
## The pipeline: two gatling passes, one serial barrier
The convert is **two `gatling::run_typed` passes over the same input** with a serial
sort barrier between them. `gatling::run_typed(reader, codec, sink, n_workers, cfg)`
(`gatling/src/gatling.rs`) is the streaming typed engine: it splits the byte stream,
each worker decodes + parses + encodes its own segment in parallel, and the
`TypedSink` collector delivers segments **in order** on one thread. The four stages
below map one-to-one onto the SVG above.
### Stage 1 — Pass 1: nodes + coordinates (parallel, per worker)
Pass 1 extracts node records and their `(id, lat, lon)` coordinates. The PBF codec is
`PbfPass1TypedCodec` (`xml_reader.rs:1869`); each worker runs `pass1_parse_blob`
(`xml_reader.rs:1579`) → `reader::blob_nodes_and_coords` (`reader.rs:60`), which
decompresses the blob with `pbf_decomp::decompress` (`pbf_decomp.rs:23`, a per-thread
ZSTD `DCtx` held `thread_local`, `pbf_decomp.rs:123`) and delta-decodes dense +
regular nodes via `reader::parse_nodes` (`reader.rs:235`).
The **region clip** (`--clip`/`--poly`) is applied **inside** the pass-1 workers at
coordinate decode — `clip_keeps` (`lib.rs:401`, used at `reader.rs:277`) — so there is
no extra serial scan (see the comment at `lib.rs:100-107`: "every core that decodes a
blob also clips it").
A relation carries no coordinates, so the clip could not test it and every relation of
the planet passed through (14 632 472 rows, 1.05 GB, on a Sweden cut whose resolve stage
keeps 274 206 — measured 2026-08-31). Since 2026-09-01 every reader answers that with
the **relation clip prefilter**: with a clip active, pass 2 is two sub-passes over the
input (`xml_reader::run_pass2`, the one orchestration). The way sub-pass reports the
ids of the ways it kept; those ids plus the node store are a
`reader::RelationClipOracle`, and the relation sub-pass — opened where the relations
start: the FRAME of the first relation blob for PBF (`BlobPlan::first_rel_blob`), the
first `<relation` for `.osm`, the last `<way ` block for `.osm.bz2` (`Bz2Skip::Ways`),
byte 0 for `.osm.gz` — keeps a relation when ANY member survived: a kept way, a node in
the store, or (settled by a fixpoint on the collector, `RelationClipOracle::settle`) a
relation that was itself kept. That is the admission rule `relgeom::resolve_relations`
already applies, so the prefilter is a strict superset of what resolves. Without a clip
there is one pass and every relation is written, byte-identical to before. The plain
`.osm` reader lost its serial pass-2 sink for this: it runs the same `Pass2TypedSink`
as the other three. The measured effect is in `osm-katana-v2-design.md`.
The split and call sites differ per format but keep the same shape:
- **PBF** → wrapper `xml_to_pbf::pbf_gatling_run_typed` (`xml_to_pbf.rs:739`, body
`gatling::run_typed` at `xml_to_pbf.rs:751`), invoked at `xml_reader.rs:2480`. The
split `pbf_split` (`xml_to_pbf.rs:707`) makes ~`2×n_workers` coarse byte ranges
(comment `xml_to_pbf.rs:715-717`); each worker then aligns to blob frame boundaries
via `pbf_io::align_and_collect`.
- **bz2** → direct `gatling::run_typed` (`xml_reader.rs:2239`), codec
`Bz2Pass1TypedCodec` (`xml_reader.rs:1724`): the split finds independent bzip2 block
boundaries, workers decompress → XML, then VTD-parse.
- **gz** → `xml_to_pbf::gz_gatling_run_typed` (`xml_to_pbf.rs:679`), codec
`GzPass1TypedCodec` (`xml_reader.rs:1763`): parallel DEFLATE where full-flush / bgzf
boundaries exist, otherwise a single segment.
- **.osm XML** → codec `Pass1ResolvedCodec` (`xml_reader.rs:478`).
The pass-1 collector `Pass1TypedSink::process` (`xml_reader.rs:1999`) streams packed
16-byte `(id, lat, lon)` records to `node_coords.bin` via `CoordFileWriter::write_raw`
(`node_store.rs:213`).
### Stage 2 — node-id → coordinate store (serial barrier, internally multithreaded)
After pass 1, `store = coord_writer.finalize()` (`xml_reader.rs:2495` PBF / `:2254`
bz2 / `:2390` gz). `CoordFileWriter::finalize` (`node_store.rs:224`) calls
`external_merge_sort` (`node_store.rs:250`) — a sample sort — and writes an mmap'd
Arrow-IPC `NodeStore`.
This is a **hard barrier**: way node-ref resolution (`store.lookup_batch`,
`reader.rs:131` / `:432`) must see the **complete sorted node index** before any way
can resolve, so pass 2 cannot begin until the store is built. It is serial *between*
the passes but **not a single-core stage** — the sort runs multithreaded on gatling
(`node_store.rs`, heaviest-chunk-first sample sort / psort; the merge is
`gatling_for_each_balanced` over the partitions and the STree layer fill is
`gatling_scanlines`). It is logged as phase `sort_and_stree` (`xml_reader.rs:2494`, note
`"psort-samplesort"`).
### Stage 3 — assemble geometry + encode Arrow (parallel, pass-2 workers)
Pass 2 resolves refs and builds the way/relation geometry. Codec
`PbfPass2TypedCodec` (`xml_reader.rs:1895`); each worker runs `pass2_parse_blob`
(`xml_reader.rs:1634`) → `blob_ways_rels` (`reader.rs:70`) → `parse_ways_rels`
(`reader.rs:335`), resolving refs against the store, then encodes **WKB geometry on the
worker** via `CpuBackend.encode_wkb` (`reader.rs:86`) — points through
`geometry::encode_point_inline` (`geometry.rs:11`), area detection through
`geometry::is_closed_area` (`geometry.rs:21`).
The Arrow row-group build **and the zstd of full row groups** also happen on the
worker, through per-worker `thread_local` accumulators (`WAY_ACC` / `REL_ACC` /
`NODE_ACC`, `xml_reader.rs:1595`) feeding `writer::ColumnEncoder::encode`
(`writer.rs:665`) → `EncodedRowGroup` (`writer.rs:648`). The call site is the second
`gatling::run_typed` (`xml_reader.rs:2519` PBF / `:2285` bz2 / `:2415` gz).
`ColumnEncoder` is `Clone + Sync` and is shared to the workers by `build_pass2_sink`
(`xml_reader.rs:2130`).
Pushing compression into the workers was a real fix: the comment at
`xml_reader.rs:2025-2037` records that the collector-side flush had "pegged pass-2 at
~5 cores on planet … workers blocked on `collector_tx.send` while collector was inside
parquet flush."
### Stage 4 — write GeoParquet (serial in-order stitch, no compression on this thread)
There is one writer per table, but it only **stitches** the workers' pre-compressed row
groups in index order: `ParallelNodeWriter::append` (`writer.rs:692`) / `finish`
(`writer.rs:702`), built by `parallel_node_writer` (`writer.rs:714`),
`parallel_way_writer` (`writer.rs:748`), `parallel_rel_writer` (`writer.rs:767`). Each
sets the GeoParquet `geo` metadata (`metadata::geo_metadata`) and schema, and splits an
`ArrowWriter` into the low-level `SerializedFileWriter`
(`into_serialized_writer`, `writer.rs:727`).
The collector sinks `Pass1TypedSink` / `Pass2TypedSink` deliver segments **in order**,
giving deterministic row order. Boundary elements that straddle two segments are parsed
serially on the collector via `flush_stub` (`xml_reader.rs:1951` node / `:2080`
way/rel) — tiny, serial by design. The stitch is serial because a Parquet append +
footer must be single-writer with deterministic row-group order; it is kept off the hot
path by having done all compression in the workers.
## Correctness
The GeoParquet `geo` metadata and schema are written by the `parallel_*_writer`
builders so each of the three files is a valid GeoParquet table. `--geometry raw` is
rejected for PBF and for `--clip`/`--poly` (`lib.rs:516`, `:522`) — clip is a
node-membership test that needs resolved coordinates.
### What "the same output" means here — and how you check it
The output is **reproducible as a row multiset, not as bytes**, and the pipeline is
built that way on purpose:
* **Row VALUES are reproducible.** `tags::serialize_tags` (`tags.rs:19`) collects into
a `BTreeMap`, so the `tags` JSON has a canonical sorted key order. It used to be a
`std::collections::HashMap`, whose `RandomState` order is seeded per process — two
converts of the same input produced the same tag *set* in a different *order*, so
every row differed. Sorted keys also pack better under the parquet zstd dictionary.
* **Row-GROUP layout is not reproducible.** Row groups are Arrow-built + zstd-encoded
**on the workers** (`writer.rs:665`) and stitched by the collector, so their number
and boundaries follow whatever segmentation that run produced.
* **Row ORDER is not reproducible either.** At end of pass each worker flushes its
partial row-group accumulator through `TypedCodec::finish_worker`, and those tail
batches are appended in worker-completion order.
So `sha256 nodes.parquet` can never be the equivalence gate — it differs between two
runs of the *same* binary. The gate is
[`digest.rs`](../osm-katana/src/digest.rs), exposed as **`osm-katana verify --digest`**:
each row is canonically encoded (presence tag + raw value bytes, per column in schema
order) and FNV-1a hashed, and the per-row hashes are folded two ways over the Mersenne
prime `2^61 − 1` —
| column | fold | invariant to | catches |
|---|---|---|---|
| `ord` | positional polynomial, `H(A ++ B) = H(A)·B^{len B} + H(B)` | row-group layout | value change, drop, insert, **reorder** |
| `set` | plain sum | row-group layout **and row order** | value change, drop, insert, duplicate |
`set` is the gate. Because the combine rule needs only the right-hand run's row
*count*, each row group is hashed on its own core
(`gatling_for_each_balanced` over the row-group indices, LPT by compressed size) and
the partials folded back — the digest is a parallel read, not a serial one.
```sh
osm-katana convert in.osm.pbf -o out-a && osm-katana verify out-a --digest
osm-katana convert in.osm.pbf -o out-b && osm-katana verify out-b --digest # set must match
```
`node_coords.arrow` *is* byte-identical run-to-run (the merge writes fixed offsets),
so `cmp` is the right check for it.
## Whereabouts
| Thing | file:symbol |
|---|---|
| CLI convert handler | `osm-katana/src/bin/main.rs:183` (`Cmd::Convert`) |
| library entry / dispatch | `osm-katana/src/lib.rs:483` (`convert`) → `:508` (`convert_inner`) |
| worker count default | `osm-katana/src/lib.rs:488` (`cores − 1`) |
| PBF reader | `osm-katana/src/xml_reader.rs:2442` (`read_resolved_pbf`) |
| bz2 / gz / XML readers | `xml_reader.rs:2196` / `:2337` / `:437` |
| pass-1 worker transform | `xml_reader.rs:1579` (`pass1_parse_blob`) |
| pass-1 codec / sink | `xml_reader.rs:1869` (`PbfPass1TypedCodec`) / `:1999` (`Pass1TypedSink::process`) |
| node decode / clip | `reader.rs:60`, `:235`; `lib.rs:401` (`clip_keeps`), used `reader.rs:277` |
| PBF split | `xml_to_pbf.rs:707` (`pbf_split`), engine `:739`/`:751` |
| coord file write | `node_store.rs:213` (`CoordFileWriter::write_raw`) |
| sort barrier | `node_store.rs:224` (`finalize`) → `:250` (`external_merge_sort`) → psort |
| pass-2 worker transform | `xml_reader.rs:1634` (`pass2_parse_blob`) |
| ref resolve / WKB | `reader.rs:335` (`parse_ways_rels`), `:86` (`encode_wkb`); `geometry.rs:11`/`:21` |
| worker Arrow encode | `writer.rs:665` (`ColumnEncoder::encode`) → `:648` (`EncodedRowGroup`) |
| pass-2 sink build | `xml_reader.rs:2130` (`build_pass2_sink`) |
| in-order writer stitch | `writer.rs:692` (`append`) / `:702` (`finish`); builders `:714`/`:748`/`:767` |
| boundary stub (serial) | `xml_reader.rs:1951` / `:2080` (`flush_stub`) |
| tag JSON (canonical, sorted) | `tags.rs:19` (`serialize_tags`) |
| content digest / equivalence gate | `digest.rs:digest_file` · CLI `verify --digest` |
| census of tables already on disk | `census.rs:census` · CLI `census` / `census --pyramid` |
| the census closure identity | `census.rs:Partition::unaccounted` / `:balances` |
| a percentage that carries its denominator | `census.rs:Ratio` (the ONLY door to a `%`) |
| phase log + per-phase occupancy | `phase_log.rs:Phase` · `StopFlag` |
| optimize row-group fan-out | `optimize.rs:gatling_collect` |
| coord merge fan-out | `node_store.rs:merge_sorted_chunks_to_arrow` |
| GeoParquet → Arrow IPC | `geo2arrow.rs:geo2arrow` (`ordered::run_ordered_sink`) |
| verify row-group counters | `verify.rs:count_node_refs` / `:count_geometry` |
| search-index row-group fan-out | `side_outputs/search.rs:gatling_features` |
| the ONE sanctioned private thread | `xml_to_pbf.rs:blob_writer_loop` (single blocking-I/O sink) |
### ROOT LAW #0 status (2026-07-22)
osm-katana holds **zero** `std::thread::scope` worker pools. Every fan-out in the
crate — both convert passes, the coord merge and its reference twin, the parquet→Arrow
converter, the verify counters, the search-index build, the optimize collector, and the
row-group encode in `writer.rs`'s tests — routes through `gatling`. The crate has no
entries left in `SCOPED_POOL_ALLOW` (`tests/rayon_free_law.rs`), so a scope pool
anywhere under `osm-katana/src` now trips the law test with no exception to hide behind.
The single exception is `xml_to_pbf.rs`'s `blob_writer_loop`: ONE named, long-lived
thread doing blocking file I/O off the gatling collector, fed by a bounded mpsc channel.
That is IO-offload, not CPU fan-out — there is exactly one of it, it never scales with
core count, it claims no units from a shared cursor, and it does no parse/encode. It is
licensed by `PRIVATE_POOL_ALLOW` and commented at the call site.
## Measuring it — per-phase core occupancy, not just wall time
Wall time cannot tell you *which* phase drops the box. Every phase opens a
`phase_log::Phase` (`phase_log.rs`), whose `done` event carries the phase's REAL
occupancy: `cpu_cores` (process CPU-seconds / wall-seconds over that phase's window),
`n_cores`, `busy_pct = cpu_cores / n_cores`, and `collector_pct` (the fraction of wall
the single collector spent inside `process()`). Phases run sequentially over one
process-wide CPU counter, so this attributes occupancy per phase with no per-thread
accounting. Read them with `--log <file.jsonl>` — or let the bench do it:
`examples/nornir-bench.rs` parses that log back and publishes one row per phase
(`osm-katana_pass1` / `_sort_and_stree` / `_pass2` / `_convert`).
Two signatures to read for:
* `convert.busy_pct` **below every individual phase's** ⇒ the pool drains *between*
bursts — a barrier, not a slow phase.
* high `collector_pct` with low `busy_pct` ⇒ collector-bound; the workers are starving
behind the single in-order sink.
The phase guard's teardown used to be part of the problem: the sampler thread only
noticed the stop flag between 100 ms sleeps, and `done()` joins it *before* reading the
clock, so every phase boundary charged up to 100 ms of dead time with the whole pool
idle — four times per convert. `StopFlag` (Mutex + Condvar) makes that immediate.
## Saturation state
Both passes fan across every core on gatling; the sort barrier (Stage 2) and the
single-writer parquet stitch (Stage 4) are serial-by-design and exempt. The LPT
weighting seen in the separate `read_pass2_mmap` mmap API — `gatling_for_each_balanced`
weighted by blob byte length (`reader.rs:138`, comment `reader.rs:130-136`) — serves
the `load_ways_pbf` helper, **not** the convert CLI (which uses `run_typed`). The
`SlotFill::Big` choice (`xml_to_pbf.rs:728-734`) is what keeps slot turnover from
starving the workers on planet-scale PBF. This doc records **no fabricated
`cores_busy`**: honest all-core numbers land in the [`README.md`](README.md)
`zoomies-katana` mashup on the next heavy-bench wave (the planet run); the measured
per-phase occupancy on a 319 MB PBF is in
[`benchmarks.md`](benchmarks.md#osm-katana--per-phase-core-occupancy).
**Known remaining drains** (measured, not guessed — see `benchmarks.md`):
1. `sort_and_stree` is the phase barrier and holds only ~4 of 32 cores: pass 1 must be
fully drained before the tail chunk can be sorted, and pass 2 cannot start until the
STree exists.
2. Pass 1 is **collector-bound** — `collector_pct` ≈ 50 %. The single in-order sink
copies every packed coord buffer into the sorter *and* writes the whole
`nodes.parquet` (565 MB + 533 MB on one thread for the 319 MB input). Measured:
the *coord copy* is **not** the cost — it already runs at memory bandwidth
(3.83 GB/s; a hand-written `memcpy` was 0.5 % faster and was reverted, see
`benchmarks.md`). The remaining ~50 % is the parquet row-group stitch + footer
metadata, which is single-writer by format.
3. Pass 2 re-reads and re-inflates **every** blob, including the node-only blobs it
will find nothing in. Pass 1 already knows which blobs carry ways/relations; passing
that set forward would let pass 2 skip them outright. Not attempted — the streaming
codec's `transform` is handed a slot-relative slice and does not know the blob's
absolute file offset, so the map has no key yet.
## The pyramid stopped carrying what a reader cannot see (2026-08-31)
Three findings from a full Sweden cut on oden, each measured before it was acted on.
### K16 — an untagged node is a vertex, not a feature
`nodes.parquet` was placed into the pyramid in full: `nodes scanned: 103 488 583 · placed
103 488 583`. In OpenStreetMap the overwhelming majority of nodes exist only to give a way
its shape, and the ways that use them are already in the pyramid (or in a network table),
so the vertex adds nothing a reader can see and nothing a classifier can name.
MEASURED **whole-country, from the run's own scan ledger** (no sampling):
```
SCAN LEDGER nodes: file 103 488 583 = placed 2 406 592 + untagged nodes 101 081 991
+ NULL geometry 0 + undecodable 0 + no geometry extent 0
· rows read 103 488 583 · UNACCOUNTED 0
```
**101 081 991 of 103 488 583 nodes — 97.7 % — carried no tags at all.**
> ### A correction, and how the wrong number was produced
>
> An earlier revision of this section (and the commit that landed it) claimed *"107 299 of
> 188 112 rows (57.0 %) over all nineteen z6 tiles"*. That figure is **wrong**, and the
> error is worth recording because it is invisible in the output. The census flattened the
> pyramid's paths when it copied the tiles — `cp z6/<x>/<y>.parquet` into one directory —
> so `z6/33/15.parquet` and `z6/34/15.parquet` overwrote each other and **7 of the 19 tiles
> survived**. The run printed `7` and the table above it still said "nineteen".
>
> A sample that silently loses two thirds of its population still produces a clean-looking
> percentage. The replacement number is taken from the tiler's own ledger, which counts
> every row of every file by construction and closes to `UNACCOUNTED 0` — there is nothing
> to sample and nothing to get wrong.
`korp`'s own
`map_networks` module header had already measured the same thing from the other side —
removing ways from the `[6,9,12]` pyramid shrank it by *"only 5.54 % of bytes … because
**nodes dominate at those zooms**"*.
The skip is `ScanLedger::untagged_nodes`, a **ledger column** and not a silent `continue`,
because a row that leaves has to leave through a name — that is the whole of `K15`.
Two things it must not do, both guarded:
* **All three spellings of "no tags"** count: a NULL cell, an empty string and `{}`. A cut
writes `{}` for a bare vertex and the monolith path has left the cell null; a test that
covered one of them would have reported a fix that measured as nearly no change.
* **It fails OPEN on a missing `tags` column.** A pre-tags `nodes.parquet` cannot answer
the question at all, and there a null cell means *this file does not say*, not *this node
carries nothing*. Skipping on it would drop every node of such a file — the guard
inverted. Caught by three existing fixtures; now a guard of its own.
### K18 — the fan-out cap measured tiles, and the cost is bytes
`MAX_WAY_TILE_FANOUT` is a fixed count of TILES, and a tile is a fixed fraction of the
world — so the ground area it permits shrinks 4× per zoom step: ≈ 620 × 620 km at z12,
≈ 40 × 40 km at z15. The consequence is that **the same feature is accepted at one zoom and
refused at the next for no reason that has anything to do with the feature.** MEASURED on
the 2026-08-30 cut: `z12 = 4` capped, `z15 = 455` — coastline, river and railway runs
placed in their CENTROID TILE ONLY, so the shore is served in one of the many tiles it
crosses and vanishes exactly when a reader zooms in far enough to want it.
What the cap protects is the staging buffer, and that is measured in bytes. A long feature
is cheap to duplicate *because* it is long and simple — Douglas–Peucker has already thinned
a coastline run at z15 to a handful of vertices — while a dense polygon is expensive
whatever its span. `fanout_budget_tiles(per) = MAX_WAY_FANOUT_BYTES / per`, clamped:
* never **below** `MAX_WAY_TILE_FANOUT`, so the change can only ever place MORE geometry
than before — it cannot newly refuse anything;
* never **above** `MAX_WAY_FANOUT_TILES_CEILING` (512 × 512), so a mis-estimated `per`
cannot ask for an unbounded number of placements.
### K17 — a ratio whose denominator is the wrong set is a different claim
The relation probe printed `274 206 rows, 225 799 carry assembled geometry (82.3 %)`, which
reads as *one relation in six failed* and cost a reader an afternoon. Only an AREA relation
is ever GIVEN geometry (`RelationKind::Area`): a `type=route` is an ordered set of ways, a
`type=restriction` is a rule, a `type=public_transport` is a stop group. MEASURED on that
same file: **226 967 area relations, 225 799 assembled = 99.49 %**; the 48 407 without
geometry are 14 588 routes, 14 412 stop areas, 5 633 turn restrictions, 2 463 waterway
relations and so on. The real failure set is the ~1 324 areas that did not assemble (0.58 %).
The probe does not classify relations, so it now prints **no ratio at all** and names both
populations instead. A percentage it cannot compute honestly is worse than none.
## The `geometry` column says what it is (2026-08-31)
Every parquet this crate writes carried WKB in a bare `BYTE_ARRAY`. A reader learned it
was geometry only by parsing the GeoParquet `geo` key in the footer — an out-of-band JSON
convention, not the schema. Parquet's own `GEOMETRY` logical type exists now, and
`parquet 59.2.0` implements both ends of it, so the column can carry its own type.
**`shared::schema::geometry_field()` is the ONE writer** (LAW 5). It returns the field with
the `parquet_geospatial::WkbType` arrow extension type, which is the only door through
which `ArrowWriter` emits `LogicalType::Geometry` and runs the writer-side bounding-box
accumulator that fills `ColumnChunkMetaData::geo_statistics`. Five production schema
builders route through it — `nodes_schema`, `ways_schema`, `relations_schema`,
`tiles::tile_fields`, `network`'s table schema, `graph`'s two, and `optimize`'s repack.
`parquet`'s `geospatial` feature is **not** on by default; a build without the feature
line in `Cargo.toml` writes the old bare column and nothing complains.
### What it buys, and what it does not
MEASURED 2026-08-31, `geometry_columns_carry_the_logical_type`, both arms on identical
bytes: the declared field comes back `Geometry(GeometryType { crs: Some("srid:0") })` with
`geo_statistics` present and the bbox `x (10.594, 24.178) · y (55.137, 69.06)`. The bare
field comes back `logical type: None`, `geo_statistics: NONE`.
* **Interop.** DuckDB, Spark and skade read the geometry from the schema. No `geo` blob to
agree on first.
* **A second source for the prune.** The chunk bbox exists whether or not a covering
column was appended. korp's `mapquery.rs` documents the hole this closes: a freshly-cut
`ways.parquet` has no covering bbox and is therefore "served slowly and honestly".
* **No speed.** `geo_statistics` lives on `ColumnChunkMetaData` — the same granularity
korp's `mapquery::row_group_intersects` already prunes at through the four `bbox.*` leaf
statistics — and `parquet`'s page index (`src/file/page_index/`) carries no geospatial
variant, so there is no page-level spatial prune to reach for. Nothing on any read path
gets faster.
* **The `covering.bbox` column STAYS.** It is what korp culls per ROW with, before any WKB
is decoded. Deleting it to reclaim its 19.04 MB of `roads.parquet` (6.36 % compressed)
would push that cull onto the geometry column — 146.50 MB compressed, **487.90 MB
uncompressed**. Measured on `~/.korp/work/sweden/networks/roads.parquet`, 2 277 368 rows.
### It cannot force a re-cut
`TileRecipe`'s `columns` term is built from `f.name()` and `short_type(f.data_type())`, and
an extension type is field METADATA that changes neither. Were that ever to stop being
true, every cut pyramid on every box would be judged *Stale* — 1 635 971 tiles re-cut for a
metadata edit. `the_logical_type_does_not_move_the_recipe` pins the name, the `DataType`,
and that the field carries metadata at all, so the test cannot pass by the field being
plain. Both guards seen RED with the bare field restored.
## `census` — asking a question of tables that already exist (2026-08-31)
The 57.0 % correction above records *what* went wrong. This verb records *why it could*:
there was no way to ask a question about a pyramid on disk except by hand-rolling a
script, and the script that got hand-rolled lost two thirds of its population to a `cp`.
`ScanLedger` cannot be sampled wrong — but it **only exists inside a tiling run**. Once
the tiles are written, the discipline was gone. `census.rs` is that discipline, applied to
files that already exist:
```sh
osm-katana census ~/.korp/work/sweden # the four extract tables
osm-katana census ~/.korp/work/sweden/tiles --pyramid # the pyramid, footer only
osm-katana census ~/.korp/work/sweden/tiles --pyramid \
--tags --geometry --classes --json census.json
```
### The property that makes it worth having
**Every population sums to the file's own footer `num_rows`, and the books close.** A
`Partition` carries its denominator, `Partition::unaccounted()` is printed on every line,
and a partition that does not balance **prints no percentage at all** and exits non-zero.
A ratio over books that lost rows is precisely the machine that produced the wrong number.
Three further properties, each a direct answer to how the 57.0 % happened:
* **One accumulator produces both the population size and every ratio.** The bug's
signature was a table saying nineteen above a run that read seven. `files` and `rows`
and every bucket come out of one `GroupCensus`, so they cannot disagree — and the verb
does the walk itself, keying every file by its path **relative to the root**, so there
is no copy step to lose a tile in.
* **A flattened directory is REFUSED, not sampled.** In `--pyramid` mode a file must sit
at `z<Z>/<X>/<Y>.parquet`; a `cp z6/*/15.parquet flat/` leaves a `15.parquet` at depth 1
and the census reports `NONCONFORMING 1` and exits 1.
* **Path identity is checked against content identity.** A tile moved to a legal-looking
address keeps a legal-looking layout, so the layout check cannot see it — but the file's
own `covering.bbox` footer statistics can. A file whose extent does not *intersect*
`tile_bounds(z, x, y)` is `MISPLACED`. The test is intersection and not containment on
purpose: a feature is placed in every tile its bbox overlaps and the stored geometry is
the whole feature, so a healthy tile routinely spills past its own bounds. **0 misplaced
of 1 635 971** on the live cut.
### Every gate was seen RED
Five mutations of the shipped code, each killing exactly the tests it should and no
others (`tests/census_books_close.rs`, 9 tests):
| mutation | tests killed |
|---|---|
| `unaccounted()` always returns 0 | the three closure gates |
| `note_nonconforming` never increments | the flattened-pyramid gate |
| `Extent::intersects` always true | the misplaced-tile gate |
| the non-vacuity check removed | the zero-files gate |
| a missing `tags` column fails CLOSED | the fail-open gate |
`Sabotage::{DropTagged, DropPoints, InflateTotal}` drop or mis-attribute a population
**inside the real counting loop**, so the red arm travels the same code the green one
does. It is reachable from a shell: `OSM_KATANA_CENSUS_SABOTAGE=drop-tagged`, which on the
live z6 prints `UNACCOUNTED 343937`, suppresses every `%` on the line, and exits 1.
### What it costs — MEASURED on oden, 2026-08-31, 32 cores
The live Sweden pyramid, `/home/korp/.korp/work/sweden/tiles`, page cache warm (korp-server
was serving the same objects):
| run | files | wall | what was read |
|---|---|---|---|
| `--pyramid` | 1 635 971 | **11.4 s** | footers only — no data page |
| `--pyramid --tags` | 1 635 971 | **11.3 s** | + the `tags` column (646 MB compressed) |
| `--pyramid --tags --geometry` | 1 635 971 | **23.7 s** | + `geometry` (34.53 GB compressed) |
| tables, `--tags` | 4 | **3.5 s** | 127 720 403 rows |
Seconds, not minutes, and the difference between the first two rows is the whole design:
rows, bytes, row groups, column bytes and covering extents need **no data page at all**.
Fan-out is `gatling` (ROOT LAW #0), **one unit per `x` directory**, and each worker folds
straight into its own `GroupCensus`. The pyramid is 1 635 971 files under **1 417** `x`
directories; a `Vec<PathBuf>` over the leaves would be ~100 MB and the largest allocation
in a process whose whole job is to hold nothing. 1 417 units over 32 cores is ~44 units
each, so the longest single unit is a small fraction of a worker's total and gatling's
shared cursor absorbs the z6-vs-z15 spread without a weight function.
### What the first run found
MEASURED 2026-08-31, on the pyramid written at 06:15–06:17 UTC that day (mtimes from the
census's own JSON), recipe token identical on all 1 635 971 files — one cut, not a mix:
```
ALL files 1 635 971 · bytes 44 449 184 179 (44.45 GB) · rows 33 357 988
geometry: LineString 9 052 208 (27.1 %) + MultiPolygon 16 961 280 (50.8 %)
+ Point 7 344 500 (22.0 %) · UNACCOUNTED 0
tags: tagged 28 959 168 (86.8 %) + untagged `{}` 4 398 820 (13.2 %) · UNACCOUNTED 0
geometry × tags: every one of the 4 398 820 untagged rows is a LineString —
0 untagged Points, 0 untagged MultiPolygons.
```
Two figures reproduced exactly, which is what a validation is for:
* `nodes.parquet`: **101 081 991 of 103 488 583 untagged (97.7 %)** — the `ScanLedger`
number, from the file rather than from the run, `UNACCOUNTED 0`. All 101 081 991 are the
`{}` spelling; the NULL and empty-string spellings are 0 in this cut, which is why a
census that tested only one of the three could report a fix as nearly no change.
* z6: **31 981 of 375 918 (8.51 %)** untagged, over **19 tiles** — `korp f6aac7e`'s
corrected figure, from all nineteen this time. That commit adds *"and NONE of them is a
point"*; the `geometry × tags` cross-tabulation confirms it, and generalises it to all
four zooms.
> ### ⚠ A figure in `osm-katana-v2-design.md` that this pyramid does not support
>
> That document quotes *"sub-pixel areas shipped as `Point` (11 101 365 of 12 628 347 area
> placements, **87.9 %**)"*, beside a centroid-only figure explicitly updated for the 08-31
> recut. **The pyramid that recut produced holds 7 344 500 `Point` rows in total, across
> all four zooms** — 3 756 865 fewer than the claim needs, before any of them is asked
> whether it is an area. The claim cannot be true of the tables now on disk.
>
> Which of the two is wrong **is not determinable from the tables**, and that is the point:
> the 87.9 % carries no date, no file list and no recipe token, so there is nothing to
> check it against. Every figure this verb prints ships with all three in its JSON.