# osm-katana — guide
> **Class:** `guide` · **Function:** `osm-katana convert`. Build, run, tune. The shape
> is in [`osm-katana-design.md`](osm-katana-design.md); the why in
> [`osm-katana-reasoning.md`](osm-katana-reasoning.md).
## Build
```sh
CARGO_TARGET_DIR=/mnt/work4t/cargo-target/zoomies cargo build --release -p osm-katana
```
Binary lands at `target/release/osm-katana`.
## Run
```sh
osm-katana convert planet.osm.pbf # → out/{nodes,ways,relations}.parquet
osm-katana convert region.osm.bz2 -o region_geo # bz2 input, custom output dir
osm-katana convert extract.osm.gz # gz input
osm-katana convert small.osm # plain XML input
```
Input format is chosen by extension (`convert_inner`, `osm-katana/src/lib.rs:508`):
`.osm.pbf`, `.osm.bz2`, `.osm.gz`, `.osm`. Output is always three GeoParquet tables —
`nodes.parquet`, `ways.parquet`, `relations.parquet` — in the `-o` directory (default
`out`).
Every run is the same two-pass, all-core pipeline: pass 1 extracts node coordinates,
a serial sort barrier builds the node store, pass 2 resolves ways/relations and encodes
geometry, and a single in-order writer stitches the pre-compressed row groups. See the
design doc's flow diagram.
## Flags
| flag | default | meaning |
|---|---|---|
| `-o, --output <dir>` | `out` | output directory for the three parquet files |
| `--nodes / --ways / --relations <bool>` | `true` | include each table (`main.rs:19-23`) |
| `--compression <c>` | `zstd` | parquet compression: `zstd` \| `snappy` \| `none` |
| `--vtd-workers <N>` | `0` | worker count; `0` ⇒ `available_parallelism() − 1` (`lib.rs:488`) |
| `--geometry <mode>` | `resolved` | `resolved` (real coords) or `raw` (node-ID lists) |
| `--clip <bbox> / --poly <file>` | — | region clip; **requires** `resolved` geometry |
| `--skip-changesets` | `false` | skip the changeset prologue |
| `--version`, `-V` | — | print `osm-katana v<semver> (<git-hash>)` and exit — the hash `build.rs` read off `.git` at build time; run it before quoting any number from a binary |
Notes on the guardrails (all enforced in `convert_inner`):
- `--geometry raw` supports XML inputs only; PBF must use the default `resolved`
(`lib.rs:516`).
- `--clip` / `--poly` require `resolved` — the clip is a node-membership test that
needs decoded coordinates, so it cannot run on `raw`'s node-ID lists (`lib.rs:522`).
The clip is applied inside the pass-1 workers at coordinate decode (`clip_keeps`,
`lib.rs:401`), not as a separate pass.
- With a clip, EVERY convert runs pass 2 as two sub-passes: ways first, then relations
from where the relations start (PBF: the first relation blob; `.osm`: the first
`<relation`; `.osm.bz2`: the last `<way ` block; `.osm.gz`: byte 0 again, gzip cannot
seek), and a relation is written only when at least one member survived the clip (a
way in `ways.parquet`, a node in the node store, or a relation that was itself kept).
Without this `relations.parquet` was the whole PLANET's relation set — 14.6 M rows,
1.05 GB, for a Sweden cut whose resolve stage keeps 274 k — and that cut is made from
`planet-latest.osm.bz2`, so the bz2 arm is the one that pays. The pass-2 `done` line
of `convert.jsonl` says which happened: `rel_prefilter` is `clip` or `none`, with
`rels_dropped_by_clip`, `rels_kept_via_relations`, `rel_oracle_ways`,
`rel_pass_start_offset` (PBF, `.osm`, `.osm.bz2`) and `rel_blobs_planned` (PBF) beside
it. Measured on Liechtenstein with a bbox over Vaduz: 938 → 188 relation rows,
2 663 580 → 991 170 bytes, the same 188 ids via bz2 and via pbf.
## Tune
- `--vtd-workers N` pins the worker count; leave it `0` to get `cores − 1`.
- Watch `htop`: both passes should sit near-flat across cores. Two short dips are
expected and correct — the serial sort barrier between the passes (Stage 2, itself
multithreaded) and the single-writer parquet stitch at the tail (Stage 4). Neither is
a saturation bug; see the reasoning doc.
- For a bz2 input the progress bar length is the compressed size × 2 (two passes), so
it looks slow during pass-1 decompression and fast afterward — the message field
always shows the current activity (`lib.rs:536-539`).
- `--compression none` trades file size for the least tail work; `zstd` (default) is
compressed **on the pass-2 workers**, not on the writer thread, so it does not stall
the stitch.
## Related subcommands
`osm-katana` ships sibling verbs for partial conversions (`osm-katana/CLAUDE.md`):
`xml2pbf` (OSM XML/BZ2 → PBF), `pbf2geo` (PBF → GeoParquet), `geo2arrow` (GeoParquet →
Arrow IPC), `verify` (inspect/verify output) and `census` (below). This guide covers
`convert`, the end-to-end path.
## What is actually in these tables? — `census`
**Reach for this instead of writing a script.** `verify` answers *is this fit to serve?*;
`census` answers *what is in it?* — and it answers with books that close, so the number
you quote cannot have quietly lost part of its population.
```sh
osm-katana census ~/.korp/work/sweden # the four extract tables
osm-katana census ~/.korp/work/sweden/tiles --pyramid # 1.6 M tiles, footer only
```
```
nodes.parquet files 1 · bytes 709045761 (0.71 GB) · rows 103488583 · row groups 217
rows cross-check: footer num_rows 103488583 vs row-group sum 103488583 — AGREE
column bytes: 707771791 compressed / 3618034439 uncompressed
geometry 514933550 B compressed (514933550 of 707771791 (72.8 %)) …
tags: 103488583 rows = tagged 2406592 of 103488583 (2.3 %)
+ untagged: {} 101081991 of 103488583 (97.7 %) · UNACCOUNTED 0
VERDICT: books close, every population accounted for.
```
**Read `UNACCOUNTED` first.** It is on every population line, it is the file's own footer
`num_rows` minus what the buckets claim, and if it is not `0` the verb prints **no
percentage on that line at all** and exits non-zero. Every ratio carries its denominator
in the same string — `101081991 of 103488583 (97.7 %)`, never a bare `97.7 %`.
| flag | what it adds | what it reads |
|---|---|---|
| *(none)* | rows, bytes, row groups, column bytes, covering extents, recipe token | **footers only** |
| `--pyramid` | one group per `z<N>`; layout + address checks | footers only |
| `--tags` | tagged / untagged, all three spellings (NULL, empty, `{}`) | the `tags` column |
| `--geometry` | Point / LineString / Polygon / Multi\* histogram, + `geometry × tags` | the `geometry` column (the fat one) |
| `--classes` | per-class breakdown via the ONE classifier | the `tags` column |
| `--json <p>` | the machine-readable census, with input identities | — |
Timings on oden (32 cores, page cache warm), the live Sweden pyramid of **1 635 971**
tiles / **44.45 GB**: footer-only **11.4 s**, `--tags` **11.3 s**, `--tags --geometry`
**23.7 s**. The four extract tables with `--tags`: **3.5 s** over 127 720 403 rows. If a
census ever takes minutes, it is because you asked for `--geometry`.
### When it refuses
It exits non-zero, and says which of these it is:
* `read ZERO parquet files` — you pointed it at the wrong directory. It never prints a
clean empty table.
* `NONCONFORMING n file(s) are not at a z<Z>/<X>/<Y>.parquet address` — you pointed
`--pyramid` at a **flattened copy** of a pyramid. Point it at the real root; a flat copy
has already lost every tile that shared a `<y>` name with another.
* `MISPLACED n of m addressed file(s)` — a tile's own covering bbox does not intersect the
tile its path names. Something moved files.
* `partition <name> does not close — UNACCOUNTED n` — a bug in the census itself. Do not
quote anything from that run.
### Quote the JSON, not the terminal
`--json` writes the whole census plus the **identity of what was measured**: every input's
path relative to the root, its bytes, its mtime, and the `osm-katana:tile-recipe` token
from its footer. A number without provenance is how the 57.0 % survived for a week in two
design documents. 53 KB for the whole Sweden pyramid.
### Proving the guard works
```sh
OSM_KATANA_CENSUS_SABOTAGE=drop-tagged osm-katana census <dir> --pyramid --tags; echo $?
```
It drops a whole population inside the real counting loop. Expect `UNACCOUNTED` non-zero,
every `%` gone from that line, `VERDICT: REFUSED` and exit `1`. Arms: `drop-tagged`,
`drop-points`, `inflate-total`.
### `buildings` — the height table
```
osm-katana buildings <extract-dir> -o <out-dir>
```
Cuts `buildings.parquet` + `buildings.json` out of `ways.parquet` +
`relations.resolved.parquet`: one row per `building=*` / `building:part=*` feature, with
`height_m`, `min_height_m`, `levels` and a `height_source` stamp. Hilbert-packed with
`covering.bbox` through the same `spatial::pack_batches` the networks use, so korp reads
it through the same seam. Write it BESIDE a served pyramid, never into it.
**Read the histogram it prints — that is the point of the verb.** Sweden, 2026-08-31,
3 841 483 rows / 297.5 MB / **16.0 s** (154.3 MB / 11.3 s with `--no-generalise`, which
drops DELTA 2's `geometry_z6`/`z9`/`z12` — +92.8 % of the file is those three columns):
```
absent 3675499 95.6792 %
tagged_metres 9551 0.2486 %
tagged_feet 1 0.0000 %
derived_from_levels 156167 4.0653 %
rejected 265 0.0069 %
```
95.68 % of Swedish buildings have no height in OSM at all. `derived_from_levels` is
`building:levels × 3.2 m` — an assumption with a ±20 % median error, which is why it is a
separate bucket from `tagged_metres` rather than the same `f32`. `rejected` means a tag
WAS there and could not be used (`height=1;12`, `height=B`): if that number jumps, the
parser regressed — the mappers did not. Every number is in `buildings.json` too, so an
operator never has to open the parquet. See `.nornir/osm-katana-v2-design.md` § DELTA 1b.
## Did my change alter the output? — `verify --digest`
`sha256 nodes.parquet` is **not** the check: row groups are encoded on the workers and
their boundaries move run-to-run, and the tail row groups land in worker-completion
order, so two runs of the *same* binary give different bytes. Compare the content
digest instead:
```sh
osm-katana convert in.osm.pbf -o out-before && osm-katana verify out-before --digest
# …change something…
osm-katana convert in.osm.pbf -o out-after && osm-katana verify out-after --digest
```
```
nodes.parquet: 353167 rows set 16a8386cf4b7ad76 ord 057676bc3141a944
ways.parquet: 41702 rows set 02ed64602fadfe31 ord 1a55d7e5a445c7a6
relations.parquet: 938 rows set 119bffec8cc479f0 ord 03cc98ab2913b696
```
Compare the **`set`** column (and the row count): it is the order-independent multiset
digest, so it ignores the layout/order the pipeline legitimately varies while still
catching any changed, dropped, added or duplicated row. `ord` additionally pins row
order — use it when a change is *expected* to preserve order too. And
`cmp a/node_coords.arrow b/node_coords.arrow`: that file **is** byte-identical.
## Bench
Driven by nornir through the shared harness. Honest all-core numbers land in the
[`README.md`](README.md) `zoomies-katana` mashup after the quiet-box planet run reaches
the warehouse. Run `nornir bench run znippy-zoomies`, then
`nornir docs book znippy-zoomies` to fill the mashup.
### PREVIEW — the seconds-turnaround lens for the optimise loop
The heavy bake-off builds a 1 GB corpus in every rival format before it gets anywhere
near osm-katana. `NORNIR_ZOOMIES_CORPUS_MB=0` means "no decompressor corpus", so the
decoder roster is empty and **only** the osm-katana arm runs:
```sh
NORNIR_ZOOMIES_CORPUS_MB=0 NORNIR_ZOOMIES_WORK=/tmp/zb \
cargo run --release --example nornir-bench # uses ~/work/liechtenstein-latest.osm.pbf
NORNIR_ZOOMIES_CORPUS_MB=0 NORNIR_ZOOMIES_WORK=/tmp/zb \
NORNIR_OSM_KATANA_INPUT=/path/big.osm.pbf NORNIR_ZOOMIES_ITERS=3 \
cargo run --release --example nornir-bench
```
Same code path, same rows, same metric keys as the heavy run — including the per-phase
occupancy rows and the digest gate — just small. It prints, per phase:
```
osm-katana per-phase core occupancy (fastest iteration):
pass1 1.145s 16.35 / 32 cores busy 51.1% collector 55.3%
sort_and_stree 0.450s 4.74 / 32 cores busy 14.8% collector 0.0%
pass2 1.346s 20.70 / 32 cores busy 64.7% collector 13.5%
convert 2.963s 16.45 / 32 cores busy 51.4% collector 0.0%
osm-katana output digest (3 runs): nodes.parquet=…/35316700 … → STABLE
```
A run counts as PREVIEW when the corpus is under the heavy 1024 MB default **or** the
osm-katana input is under 1024 MB, and **a preview never emits the `BenchRun` contract
line** — stdout says `PREVIEW RUN`, the numbers go to stderr. So `nornir bench run`
cannot warehouse a light number and it can never replace a real host row in a README
table. There is no override; run it heavy to publish.