nordisk/skade

.nornir/README-full.md

<!-- nornir:class: reasoning -->
# skade — complete overview

> The **real** README: the full pitch, the competitive matrix, design, quickstart
> and every measured benchmark. The repo-root [`README.md`](../README.md) is the slim
> face (title · mashup · pointers); this chapter is the heavy detail, and it is also
> folded into the manual [`docs/manual.pdf`](../docs/manual.pdf) by `nornir docs book`.

<p align="center"><em>Skaði, the winter queen — she keeps the icebergs in order.</em></p>

**Why skade** — the whole Iceberg stack as one in-process Rust dependency, where the
catalog is a single redb file rather than a JVM service — the rows marked **NA** are
things the rivals have no concept for:

<!-- skade:gen:start:capabilities — generated from bench/src/capabilities.rs (nornir static-capabilities seam, issue #13); regenerate: UPDATE_CAPABILITIES=1 cargo test -p skade-katalog-bench --test capabilities -->
| capability | **skade** | Iceberg (Java) | PyIceberg | Delta Lake |
|---|:--:|:--:|:--:|:--:|
| Pure-**Rust**, in-process — no JVM, no network catalog hop | **✅** | ✗ (JVM) | ◐ (Python) | ✅ |
| Embedded **single-file ACID catalog** (redb) — no catalog service to run | **✅** | ✗ (Nessie/Polaris/Hive/Glue) | ✗ (SQL/REST) | ◐ |
| **Native gRPC catalog** — binary/HTTP-2 wire (`skade.katalog.v1`), no JSON-per-call | **✅** | ✗ (REST only) | ✗ (REST only) | ✗ |
| Catalog reads **~492× Nessie · ~677× Polaris** (`table_exists`, 2.0M ops/s) | **✅** | ◐ | ◐ | — |
| **Multi-table atomic commit** — flip N tables in one txn (`atomic_release`) | **✅** | ✗ | ✗ | **NA** |
| Lock-free static **time-travel index** (Ragnar `STree64`) | **✅** | **NA** | **NA** | **NA** |
| All-core **parallel ingest** that saturates the box (gatling no-barrier) | **✅** | ◐ (Spark) | ✗ | ◐ |
| Mature engine ecosystem (Spark/Trino/Flink) + schema-evolution writes today | ◐ | **✅** | ◐ | **✅** |
<!-- skade:gen:end:capabilities -->

*Full numbers below — generated by `nornir bench run skade-katalog`.*

## What it does

- **`skade`** — the data plane: Arrow `RecordBatch` in, SQL out; gatling
  multi-core ingest, zero-copy recast, identity-partitioned + compressed writes,
  V3 tables. (`skade/`)
- **`skade-katalog`** — the catalog: pure-Rust, redb-backed, single-file ACID `iceberg::Catalog`. No SQL, no C deps.


---

# skade-katalog

![The three Norns — Urðr (past), Verðandi (present), Skuld (future) — recording the world's icebergs in Catalogus Icebergorum / Glacius Type Index](assets/skade_katalog.webp)


[![crates.io](https://img.shields.io/crates/v/skade-katalog.svg)](https://crates.io/crates/skade-katalog)
[![docs.rs](https://docs.rs/skade-katalog/badge.svg)](https://docs.rs/skade-katalog)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](../LICENSE-APACHE)

A pure-Rust [`iceberg::Catalog`](https://docs.rs/iceberg/0.9.1/iceberg/trait.Catalog.html) with cross-table transactions. A lock-free static search tree (Ragnar `STree64`) out front, continuously regenerated from a [redb](https://www.redb.org/) ACID backend. In-process — no network hop, no JVM, no JSON round-trip.

* **`table_exists` — 2.0M ops/s, ~492× Nessie · ~677× Polaris** (0.31 µs p50). Every core reads the front lock-free.
* **Ingest saturates the box — 100% CPU, fully multicore.** All cores encode Parquet in parallel; a [gatling](https://codeberg.org/nordisk/znippy) no-barrier pool keeps the writes overlapping, even over S3.
* **ACID.** Every mutation is one redb `WriteTransaction`. Atomic multi-table commits via [`RedbCatalog::atomic_release`](#multi-table-atomic-commits).

Full leaderboard (skade-katalog vs Nessie vs Polaris, every storage variant, all four capabilities) → **[Benchmarks](#benchmarks)**.

<!-- nornir:gen:start:bench_chart file=mashup_table_exists metric=ops_sec name=table_exists title="table_exists — catalog reads (ops/sec, log scale · higher is better)" -->
![table_exists — catalog reads (ops/sec, log scale · higher is better)](.nornir/assets/mashup_table_exists.svg)
<!-- nornir:gen:end:bench_chart -->

### Best-of, at a glance

The tight head-to-head — **skade-katalog vs Nessie vs Polaris**, side by side so the
gap is obvious (the rival's number is right there). Full per-storage detail (every
NVMe/RAM/S3 variant, all four capabilities) is under **[Benchmarks](#benchmarks)**.

**Reads — `table_exists` (storage-free catalog RPC, higher ops/s better):**

<!-- nornir:gen:start:benches cols=ops_sec,p50_us,p99_us compare=skade_katalog_embedded,skade_katalog_grpc,skade_katalog_rest,iceberg_memory,nessie,polaris name=table_exists winner=row -->
**v0.1.3 · loki · 12 cores · 2026-07-12**

| workload | ops_sec | p50_us | p99_us |
|---|---|---|---|
| skade_katalog_embedded_table_exists | 2.87M | 0.25 | 0.40 |
<!-- nornir:gen:end:benches -->

**Writes — `data_pipe` (create→ingest→scan, higher rows/s better):**

<!-- nornir:gen:start:benches cols=rows_per_sec,files compare=skade_nvme,skade_s3,nessie,polaris name=data_pipe winner=row -->
**v0.1.3 · loki · 12 cores · 2026-07-12**

| workload | rows_per_sec | files |
|---|---|---|
| data_pipe_skade_nvme | 24.16M | 10 |
<!-- nornir:gen:end:benches -->

## Status

The Iceberg `Catalog` trait is fully implemented and tested against `iceberg = "0.9.1"`. One gap: schema evolution (the upstream `Transaction` actions aren't public until iceberg-rust 0.10). See [Known shortcomings](#known-shortcomings-013).

## skade — the data-plane companion crate

This repo also ships [`skade`](../skade/) ("winter queen" — Skaði): **fast Iceberg
table writing/reading + ergonomic DataFusion SQL** on top of `RedbCatalog`. One
directory = catalog (`catalog.redb`) + warehouse; Arrow `RecordBatch` in, SQL
out (`wh.sql("SELECT … FROM a JOIN b …")`). It packages the writer stack,
read-to-Arrow primitive, schema bridges (incl. the unsigned `widen`/`unwiden`
reinterprets znippy needs), and the windowed-Parquet bulk-ingest helpers that
were proven in [`bench/`](../bench/). Like `bench/`, it is a detached cargo
workspace (`skade/`), so this crate's manifest and lockfile stay untouched.
See [`skade/README.md`](../skade/README.md).

## Read & write paths

Every call resolves left-to-right through these layers, falling through only on a
miss. **Latest-state** reads ride the L1 pointer mirror; **time-travel**
(snapshot-id) reads ride the Ragnar `STree64` index. redb is always the source of
truth behind them — the in-memory layers are keyed by immutable, content-addressed
identifiers, so they can only be *evicted*, never *stale*.

![Read paths as a circuit board: each catalog function enters from the left and is wired through the in-memory cache layers — the L1 pointer mirror, L0 metadata cache, L1.5 handle cache, and the Ragnar STree64 snapshot index — falling through to the redb source-of-truth rail only on a miss.](assets/read_paths_circuit.svg)

| layer | key → value | type | populated |
|---|---|---|---|
| **L1** pointer mirror | `table_key → metadata_location` | `ArcSwap<imbl::HashMap<String, Arc<str>, foldhash>>` | full scan at open, then write-through |
| **L1.5** handle cache | `metadata_location → Table` | `moka::Cache<String, Table>` (capacity-bounded) | on `load_table` miss |
| **L0** metadata cache | `metadata_location → TableMetadata` | `moka::Cache<String, Arc<TableMetadata>>` (byte-bounded, single-flight) | on metadata miss |
| **Ragnar** static index | `snapshot_id → (table_key, metadata_location)` | `ArcSwapOption<STree64>` | warm-built at open, rebuilt by bg compactor (≥1024 commits) |
| **redb** source of truth | `tables` · `commits` · `namespaces` · `namespace_props` · `meta` | `Mutex<redb::Database>` (ACID) | every committed write |

The same routing in full — including the time-travel, write, and redb-direct
paths the diagram leaves out:

```text
LATEST READS
  table_exists(id) ───────► L1 mirror ──hit──► true
                                └─miss─► redb `tables` ─────► bool

  resolve_metadata(id) ───► L1 mirror ──► loc ──► L0 cache ──hit──► Arc<TableMetadata>
       (skips Table::build)     └─miss─► redb         └─miss─► FileIO read + parse JSON

  load_table(id) ─────────► L1 mirror ──► loc ──► L1.5 cache ──hit──► Table  (~100 ns clone)
                                └─miss─► redb          └─miss─► L0 cache ──► Table::build() + insert

TIME-TRAVEL READS  (by snapshot_id: i64)
  load_table_at / resolve_metadata_at(id, sid)
        ──► Ragnar STree64 ──hit──► loc ──► L1.5 / L0 ──► Table / Arc<TableMetadata>
                 └─miss (sid above cutoff)─► redb `commits` live tail ──► loc ──► …

  resolve_many(id, [sid; N])
        ──► Ragnar batch probe (1 pipelined pass) ──► redb `commits` (1 txn, misses only) ──► L0 per loc

WRITES  (create · register · update · drop · rename)
  …──► FileIO write …/<uuid>.metadata.json ──► redb WriteTransaction { tables CAS · commits log · meta++ }
        ──commit──► L1 mirror write-through (insert / remove) ──► maybe rebuild Ragnar (background)

  update_table ──► group-commit: N concurrent commits coalesce into 1 redb txn / 1 fsync   ← commit-burst lever

REDB-DIRECT  (no cache layer)
  list_namespaces · get_namespace · namespace_exists · list_tables ──► redb read txn (range scan)
  create / update / drop_namespace ──────────────────────────────────► redb WriteTransaction
```

## Benchmarks

All numbers below are **auto-filled by `nornir docs render`** from the latest
`nornir bench run` (machine/cores/version in each header). Don't hand-edit inside
the generated regions — re-run the bench instead. Reproduce with the harness in
[`bench/`](../bench/) (`cargo run --bin bench-containers -- up` in `bench/`, then
`nornir bench run skade-katalog` from `workspace_skade/`).

### Catalog read RPC — `table_exists` (storage-free, runs on every backend)

`table_exists` is a pure catalog RPC (no object storage), the apples-to-apples
comparable; Nessie/Polaris go through the same `iceberg::Catalog` REST client.

<!-- nornir:gen:start:benches cols=ops_sec,p50_us,p90_us,p99_us,mean_us,min_us name=table_exists winner=row -->
**v0.1.3 · loki · 12 cores · 2026-07-12**

| workload | ops_sec | p50_us | p90_us | p99_us | mean_us | min_us |
|---|---|---|---|---|---|---|
| skade_katalog_embedded_table_exists | 2.87M | 0.25 | 0.27 | 0.40 | 0.26 | 0.20 |
<!-- nornir:gen:end:benches -->

### Embedded read fast paths (nornir-only)

`resolve_metadata` skips `Table::build()` (the lock-free L1+L0 floor);
`load_table` is the full `Catalog` trait path after the L1.5 handle cache.

<!-- nornir:gen:start:benches cols=ops_sec,p50_us,p90_us,p99_us,mean_us,min_us exclude=table_exists name=skade_katalog_embedded winner=row -->
**v0.1.3 · loki · 12 cores · 2026-07-12**

| workload | ops_sec | p50_us | p90_us | p99_us | mean_us | min_us |
|---|---|---|---|---|---|---|
| skade_katalog_embedded_load_table | 0.969M | 0.59 | 0.64 | **3.47** | 0.65 | 0.53 |
| skade_katalog_embedded_resolve_metadata | **1.548M** | **0.49** | **0.52** | 3.56 | **0.54** | **0.44** |
<!-- nornir:gen:end:benches -->

### Data plane — single-writer / many-processor ingest

All cores encode Parquet; one writer streams + commits (`data-pipe`). Backend-
agnostic and storage-bound (shared client-side Parquet + S3), so rows/s converge
across backends — this lifts all boats, it is **not** a catalog lever. Nessie
runs the *same* shared RustFS S3 warehouse. Scan verifies the round-trip. The
nornir local-FS row runs **two file destinations** — `_nvme` (PCIe-4.0 NVMe) and
`_ram` (`/dev/shm` tmpfs) — to expose the pure storage floor.

<!-- nornir:gen:start:benches cols=rows_per_sec,files name=data_pipe winner=row -->
**v0.1.3 · loki · 12 cores · 2026-07-12**

| workload | rows_per_sec | files |
|---|---|---|
| data_pipe_skade_nvme | 24.16M | 10 |
| data_pipe_skade_ram | **29.41M** | 10 |
<!-- nornir:gen:end:benches -->

### Commit-bursty — the catalog lever (group-commit)

Many small concurrent metadata commits — where skade-katalog coalesces commits into one
redb txn/fsync. This is where the embedded catalog genuinely pulls ahead of a
REST/JVM catalog (throughput **and** tail latency).

<!-- nornir:gen:start:benches cols=commits_per_sec,p50_us,p90_us,p99_us,mean_us,min_us name=commit_burst winner=row -->
**v0.1.3 · loki · 12 cores · 2026-07-12**

| workload | commits_per_sec | p50_us | p90_us | p99_us | mean_us | min_us |
|---|---|---|---|---|---|---|
| commit_burst_skade_nvme | 1,189 | 27,021 | 28,892 | 32,044 | 26,892 | 1,876 |
| commit_burst_skade_ram | **6,622** | **4,728** | **6,012** | **6,387** | **4,825** | **235.61** |
<!-- nornir:gen:end:benches -->

### TPC-H — analytical SQL (all 22 queries via DataFusion)

The full 8-table TPC-H schema loaded into Iceberg tables, queried through
DataFusion (`iceberg-datafusion`) — per-query latency for all 22 canonical
queries over the catalog's scan/manifest path. Authentic `tpchgen` data;
`TPCH_SF` sets the scale (small by default). Answer values aren't validated here
(that needs SF=1) — this measures plan + scan + execute latency end-to-end.

<!-- nornir:gen:start:benches bold=false name=tpch_q -->
**v0.1.3 · loki · 12 cores · 2026-07-12**

| workload | max_us | mean_us | min_us | ops_sec | p50_us | p90_us | p999_us | p99_us | rows |
|---|---|---|---|---|---|---|---|---|---|
| tpch_q01 | 9,362 | 8,399 | 7,477 | 119.06 | 8,388 | 9,362 | 9,362 | 9,362 | 4 |
| tpch_q02 | 14,217 | 13,847 | 13,584 | 72.22 | 13,884 | 14,217 | 14,217 | 14,217 | 4 |
| tpch_q03 | 8,226 | 7,382 | 6,908 | 135.46 | 6,964 | 8,226 | 8,226 | 8,226 | 138 |
| tpch_q04 | 4,644 | 4,484 | 4,362 | 223.00 | 4,458 | 4,644 | 4,644 | 4,644 | 5 |
| tpch_q05 | 12,205 | 10,683 | 9,015 | 93.61 | 11,096 | 12,205 | 12,205 | 12,205 | 5 |
| tpch_q06 | 2,969 | 2,842 | 2,707 | 351.85 | 2,824 | 2,969 | 2,969 | 2,969 | 1 |
| tpch_q07 | 14,223 | 13,923 | 13,326 | 71.82 | 14,027 | 14,223 | 14,223 | 14,223 | 4 |
| tpch_q08 | 16,293 | 15,523 | 15,055 | 64.42 | 15,507 | 16,293 | 16,293 | 16,293 | 2 |
| tpch_q09 | 13,516 | 12,920 | 11,138 | 77.40 | 13,342 | 13,516 | 13,516 | 13,516 | 173 |
| tpch_q10 | 12,997 | 12,768 | 12,073 | 78.32 | 12,910 | 12,997 | 12,997 | 12,997 | 399 |
| tpch_q11 | 9,170 | 8,475 | 7,606 | 117.99 | 8,462 | 9,170 | 9,170 | 9,170 | 359 |
| tpch_q12 | 7,766 | 7,530 | 7,210 | 132.81 | 7,576 | 7,766 | 7,766 | 7,766 | 2 |
| tpch_q13 | 6,034 | 5,722 | 5,466 | 174.75 | 5,568 | 6,034 | 6,034 | 6,034 | 33 |
| tpch_q14 | 4,103 | 3,956 | 3,796 | 252.80 | 3,968 | 4,103 | 4,103 | 4,103 | 1 |
| tpch_q15 | 8,255 | 8,035 | 7,768 | 124.45 | 8,070 | 8,255 | 8,255 | 8,255 | 1 |
| tpch_q16 | 7,950 | 7,672 | 6,973 | 130.35 | 7,805 | 7,950 | 7,950 | 7,950 | 296 |
| tpch_q17 | 5,318 | 5,251 | 5,149 | 190.44 | 5,310 | 5,318 | 5,318 | 5,318 | 1 |
| tpch_q18 | 9,834 | 9,437 | 9,108 | 105.97 | 9,427 | 9,834 | 9,834 | 9,834 | 2 |
| tpch_q19 | 7,895 | 7,748 | 7,630 | 129.06 | 7,729 | 7,895 | 7,895 | 7,895 | 1 |
| tpch_q20 | 10,426 | 10,246 | 10,062 | 97.60 | 10,284 | 10,426 | 10,426 | 10,426 | 1 |
| tpch_q21 | 15,671 | 15,444 | 15,142 | 64.75 | 15,512 | 15,671 | 15,671 | 15,671 | 1 |
| tpch_q22 | 6,001 | 5,480 | 4,823 | 182.48 | 5,521 | 6,001 | 6,001 | 6,001 | 7 |
<!-- nornir:gen:end:benches -->

### TPC-H large warehouse (opt-in, big iron)

A full-scale run: ingest the 8-table warehouse at a large scale factor across all
cores (partitioned parallel ingest), then run all 22 queries over it — reporting
build vs query time, ingest rate, and the slowest query. Runs by default at a
moderate scale; `TPCH_WAREHOUSE_SF` tunes it (default 10 ≈ 60 M lineitem rows /
a few min; 100-200 for a 10-60 min big-iron workout), `TPCH_WAREHOUSE=0` disables
it, `TPCH_WAREHOUSE_PARTS` sets ingest partitions (default = all cores).

<!-- nornir:gen:start:benches name=tpch_warehouse -->
**v0.1.3 · loki · 12 cores · 2026-07-12**

| workload | build_s | ingest_rows | ingest_rows_per_sec | parts | query_s | result_rows | scale_factor | slowest_query | slowest_query_s | total_s |
|---|---|---|---|---|---|---|---|---|---|---|
| tpch_warehouse | 7.04 | 86.59M | 12.29M | 12 | 12.64 | 534,307 | 10 | 18 | **2.42** | 19.68 |
<!-- nornir:gen:end:benches -->

### TPC-H across catalogs × storage (skade-katalog vs Nessie vs Polaris)

The same SF warehouse built and all 22 queries run over each **(catalog × storage)**
combination through DataFusion — the analytical-SQL path compared apples-to-apples
(not just the `table_exists` RPC). The matrix covers every combo that's possible:
nornir on **file-NVMe, file-RAM, and S3**; Nessie **S3-only** (it rejects a local
file warehouse); Polaris on **file** (its S3 credential vending 301s against the
RustFS endpoint — see `bench/src/containers.rs`). The REST targets need their
container + the shared RustFS S3 warehouse (`bench-containers up all`), else they
skip. `TPCH_COMPARE_SF` sets the scale (default 1.0 ≈ 8.66M rows).

> **Parallel scan patch:** stock `iceberg-datafusion` 0.9.1 scans a table as a
> single DataFusion partition (one serial Parquet-decode stream), so on a 32-core
> box only ~1 core works and queries are ~7-8× slower than they should be. The
> repo vendors a patched `iceberg-datafusion` (`bench/vendor/`, relative-path
> `[patch]` used by both `bench/` and `skade/`) that exposes **one partition per
> file**, so DataFusion decodes files across all cores. SF100 FILE query dropped
> ~1660s → ~197s (8.4×). These numbers reflect that patch; published `skade`
> builds get stock 0.9.1 until the patch is upstreamed.

At SF100
(866M rows) both S3-backed paths (`skade_s3`, `nessie_s3`) fail with `hyper:
connection closed before message completed` finishing the Parquet writer — an S3
write-robustness limit under sustained heavy PUT load (rust-s3 / RustFS), not a
catalog issue; the FILE-backed paths complete cleanly.

<!-- nornir:gen:start:benches name=tpch_cmp winner=row -->
**v0.1.3 · loki · 12 cores · 2026-07-12**

| workload | build_s | q01_s | q02_s | q03_s | q04_s | q05_s | q06_s | q07_s | q08_s | q09_s | q10_s | q11_s | q12_s | q13_s | q14_s | q15_s | q16_s | q17_s | q18_s | q19_s | q20_s | q21_s | q22_s | query_s | rows | scale_factor | slowest_query | slowest_query_s | total_s |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| tpch_cmp_skade_file_nvme | 1.74 | 0.08 | **0.05** | **0.05** | 0.03 | **0.07** | **0.02** | **0.10** | 0.07 | **0.11** | **0.07** | 0.04 | 0.07 | **0.06** | **0.02** | 0.04 | 0.02 | 0.09 | 0.20 | 0.05 | 0.05 | 0.13 | 0.02 | 1.43 | 8.66M | 1 | 18 | 0.20 | 3.17 |
| tpch_cmp_skade_file_ram | **0.74** | **0.08** | 0.05 | 0.05 | **0.02** | 0.07 | 0.02 | 0.11 | **0.07** | 0.11 | 0.08 | **0.03** | **0.05** | 0.06 | 0.02 | **0.04** | **0.02** | **0.09** | **0.17** | **0.05** | **0.04** | **0.12** | **0.02** | **1.39** | 8.66M | 1 | 18 | **0.17** | **2.13** |
<!-- nornir:gen:end:benches -->

### OSM GeoParquet ingest (data plane)

End-to-end ingest of **authentic OSM data**: a `nodes.parquet` produced by the
`katana-osm` / `osm2geoparquet` converter (OSM `.osm`/`.bz2`/`.pbf` → GeoParquet)
is read, its schema mapped to Iceberg, and ingested into a fresh `RedbCatalog`
table via the single-writer / many-processor pipeline, then scanned back to
verify. NVMe vs RAM destinations isolate storage cost. Point `OSM_GEOPARQUET` at
the file (`OSM_MAX_ROWS` caps rows); the `osm_ingest_*` benchers record a failure
if it's unset, like the container-backed targets.

<!-- nornir:gen:start:benches name=osm_ingest winner=row -->
**v0.1.16 · oden · 32 cores · 2026-07-22**

| workload | commits | encode_workers | files | mb_per_sec | read_rows_per_sec | read_threads | rows | rows_per_sec | telem_cores_busy_avg | telem_cores_busy_max | telem_cpu_pct_avg | telem_cpu_pct_max | telem_elapsed_ms | telem_mem_pct_max | telem_mem_peak_mb | telem_n_cores | windows | write_rows_per_sec |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| osm_ingest_skade_nvme | 5 | 32 | 39 | **761.29** | 169.72M | 32 | 20.45M | **18.53M** | 2 | 2 | 15.65 | 18.41 | **1153** | 5.36 | 2651 | 32 | 1 | **20.80M** |
| osm_ingest_skade_ram | 5 | 32 | 39 | 706.09 | **263.69M** | 32 | 20.45M | 17.98M | 2 | 2 | 13.88 | 18.12 | 1232 | 5.48 | 2749 | 32 | 1 | 19.29M |
<!-- nornir:gen:end:benches -->

### ZSTD decode — `zstd-sys-rs` vs the `zstd` crate

Aggregate (all-core) decompression throughput on a real OSM corpus (WKB geometry
bytes from the converted Europe extract, ~parquet-page-sized frames), decoding
the same frames two ways: the stock `zstd` crate (one-shot, allocates per frame)
and `zstd-sys-rs`'s zero-copy path (a reused `ZSTD_DCtx` + reused output buffer).
Both link the same static libzstd 1.5.7, so this validates the bindings and the
zero-copy API on real data rather than chasing a codec difference. Needs
`OSM_GEOPARQUET` (`ZSTD_CHUNK_KB` sets frame size).

<!-- nornir:gen:start:benches name=zstd_decode winner=row -->
**v0.1.16 · oden · 32 cores · 2026-07-22**

| workload | decompressed_mb | frames | max_us | mb_per_sec | mean_us | min_us | ops_sec | p50_us | p90_us | p999_us | p99_us | telem_cores_busy_avg | telem_cores_busy_max | telem_cpu_pct_avg | telem_cpu_pct_max | telem_elapsed_ms | telem_mem_pct_max | telem_mem_peak_mb | telem_n_cores | threads |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| zstd_decode_crate | 429.39 | 6559 | 58924 | 10127 | 42399 | 31842 | 23.59 | 45606 | 49200 | 58924 | 58924 | 13.67 | 32 | 50.37 | 91.04 | 2452 | 5.31 | 2743 | 32 | 32 |
| zstd_decode_sys_zerocopy | 429.39 | 6559 | **27254** | **17731** | **24217** | **22478** | **41.29** | **23154** | **26911** | **27254** | **27254** | 29 | 29 | 58.44 | 58.44 | **267.46** | 5.22 | 2743 | 32 | 32 |
<!-- nornir:gen:end:benches -->

### Ours vs rivals — skade vs reference Iceberg vs raw Parquet (one dataset)

The data-plane head-to-head over ONE identical dataset and ONE storage root. Read
throughput is meaningless beside a core count, so **every read arm reports its own
per-phase `read_cores_busy` and `read_rows_per_sec_per_core`**, and skade's scan
appears **twice — single-core AND all-core** — so its own single↔multi scaling is
visible, not just skade-vs-rivals:

* **skade** (ours) — `skade::Table::{ingest_pipelined,read}`: all-core gatling
  encode + snapshot commit, and the **all-core gatling scan** (manifest parse,
  footer load, and Parquet decode fanned across every core on the ONE fork-join
  engine);
* **skade_serial** (ours, the single-core control) — the SAME `Table::read` scan
  over the SAME layout, run through `skade::with_scan_serial`: identical code,
  identical row order, but the decode dispatched on **one core**. `skade ÷
  skade_serial` is skade's measured scan scaling on this box, the exact analogue
  of the two raw-Parquet arms below;
* **iceberg** — upstream iceberg-rust's stock writer stack (`DataFileWriterBuilder`
  + `ParquetWriterBuilder` + `fast_append`) and stock scan (`table.scan()…to_arrow()`),
  over the *same* `RedbCatalog`;
* **parquet** — the floor: raw `ArrowWriter` files, no catalog / no snapshots / no
  ACID, read back **single-threaded** with `ParquetRecordBatchReaderBuilder`;
* **parquet_par** — the SAME raw files read through the SAME gatling row-group
  fan-out skade uses. `parquet_par − parquet` is the fan-out contribution;
  `skade − parquet_par` is what the catalog/snapshot layer costs.

Held equal: the byte-identical deterministic dataset; the storage root and
tempdir-per-pass discipline; the UNCOMPRESSED+dictionary codec; the 8192-row
decode batch; the **row-group count** (the read fan-out's work units, equalised at
`OXR_ROWS/OXR_BATCH`); and the read output contract (every arm materialises the
full `Vec<RecordBatch>` `Table::read` returns). **Not** equal, and intrinsic to
each writer: the file cadence (skade writes one file per input batch; the rivals
one per commit) and write parallelism (skade's encode is all-core, the rivals
single-writer) — which is why write and read core counts are reported *separately*
and per phase (`CLOCK_PROCESS_CPUTIME_ID`), never one whole-pass number that
charges skade's all-core encode against its scan. Correctness gate: every pass
must scan back exactly the rows it wrote. `OXR_ROWS`/`OXR_BATCH`/`OXR_COMMIT_EVERY`/
`OXR_ITERS` tune it; `OXR_RAM=1` roots on tmpfs.

**Single↔multi, measured on oden (32c, `oxr`-default 1M rows / 20 units,
UNCOMPRESSED), 2026-07-24:**

| arm | read_rows_per_sec | read_cores_busy | read_rows_per_sec_per_core |
|---|---|---|---|
| skade (all-core) | **234.1M** | 9.5 | 24.6M |
| skade_serial (1 core) | 51.2M | 1.0 | **51.2M** |

skade's scan scales **4.6×** single→all-core here (51.2M/1 core → 234.1M/9.5 cores)
— but note the per-core figure *falls* 51.2M → 24.6M as cores are added, and pushing
the unit count up (20 → 160 units) raises `read_cores_busy` 8.8 → 15.5 while total
throughput *drops* slightly (222M → 185M, `examples/oxr_scaling`). That is the honest
verdict for this dataset: the all-core scan of a cheap, UNCOMPRESSED 4-column shape
is **DRAM-bandwidth bound at ~9 busy cores**, not fan-out bound — the extra cores
stall on memory, they are not idle for want of work units. The fan-out is real (4.6×
is real, and the manifest-parse + footer-load prologues are now fanned across
gatling, not serial); the ceiling above ~9 cores is this box's memory subsystem, not
the layout. When the decode is instead **compute-bound** — the
`scan_core_saturation` test's ZSTD, high-entropy data — the identical scan keeps
**~21 of 32 cores** busy at a **13× single→all-core** speedup, because there the
cores have real per-byte work and are not waiting on DRAM. "Need enough data" means
enough work units to reach the cores *and* enough per-unit work that the cores are
compute-bound; give it both and it fills them.

<!-- nornir:gen:start:benches cols=write_rows_per_sec,read_rows_per_sec,read_cores_busy,read_rows_per_sec_per_core compare=skade,skade_serial,iceberg,parquet,parquet_par name=oxr winner=row -->
**v0.1.4 · oden · 32 cores · 2026-07-24** — measured (OXR_RAM tmpfs, OXR_ITERS=7).
Hand-filled from the `oxr` bench run on this host (the gen pipeline persists to the
nornir warehouse, which this workstation's harness is not wired to; the numbers are
the harness's own JSON output, `NORNIR_BENCH_ONLY=oxr NORNIR_MACHINE=oden`). `files`
≠ equal by design (skade writes one file per input batch; the rivals one per commit);
the **row-group count is equalised at 20** — the read fan-out's work units.

| workload | write_rows_per_sec | read_rows_per_sec | read_cores_busy | read_rows_per_sec_per_core |
|---|---|---|---|---|
| oxr_skade | **44.4M** | 234.1M | 9.5 | 24.6M |
| oxr_skade_serial | 42.6M | 51.2M | 1.0 | 51.2M |
| oxr_iceberg | 10.0M | 37.0M | 1.0 | 36.8M |
| oxr_parquet | 9.2M | 62.0M | 1.0 | **62.0M** |
| oxr_parquet_par | 9.1M | **393.7M** | **12.7** | 31.0M |
<!-- nornir:gen:end:benches -->

Reading the table honestly: **skade wins the WRITE** (44.4M — 4.4× iceberg, 4.8× raw
Parquet — its all-core gatling encode) and scales its READ 4.6× single→all-core
(51.2M → 234.1M). `oxr_parquet_par` reads *faster* (393.7M at 12.7 cores) than skade
(234.1M at 9.5): it is the SAME gatling row-group fan-out over raw files with **no
catalog and no snapshot**, and it reads 3 physical files (20 row groups) where skade
reads 20 files — so `skade − parquet_par` is the catalog/snapshot/per-file-footer
cost, not a decode deficit. Every read number sits beside its own per-phase
`read_cores_busy`, because 234.1M at 9.5 cores (24.6M/core) and 62.0M at 1 core
(62.0M/core) are the SAME per-core decode — the difference is purely how many cores
each engine puts on the job.

## Quickstart

```rust,no_run
use std::collections::HashMap;
use std::sync::Arc;

use iceberg::io::LocalFsStorageFactory;
use iceberg::spec::{NestedField, PrimitiveType, Schema, Type};
use iceberg::{Catalog, CatalogBuilder, NamespaceIdent, TableCreation};
use skade_katalog::RedbCatalogBuilder;

# async fn run() -> anyhow::Result<()> {
let catalog = RedbCatalogBuilder::default()
    .db_path("/var/lib/nornir/catalog.redb")
    .warehouse_location("file:///var/lib/nornir/warehouse")
    .with_storage_factory(Arc::new(LocalFsStorageFactory))
    .load("nornir", HashMap::new())
    .await?;

let ns = NamespaceIdent::new("bench".to_string());
catalog.create_namespace(&ns, HashMap::new()).await?;

let schema = Schema::builder()
    .with_schema_id(0)
    .with_fields(vec![
        NestedField::required(1, "run_id", Type::Primitive(PrimitiveType::String)).into(),
        NestedField::required(2, "ops_sec", Type::Primitive(PrimitiveType::Double)).into(),
    ])
    .build()?;

let table = catalog
    .create_table(
        &ns,
        TableCreation::builder()
            .name("bench_runs".to_string())
            .schema(schema)
            .build(),
    )
    .await?;
# Ok(()) }
```

## Multi-table atomic commits

Iceberg's per-table `Transaction::commit` only gives single-table atomicity.
When a release logically spans several tables (e.g. publishing `bench_runs`,
`dep_graph`, and `components` from a single CI run), readers can otherwise
observe a half-published state.

redb gives every write transaction global atomicity across all of its
tables. `RedbCatalog::atomic_release(commits)` exploits this: each
`TableCommit` is staged (metadata blobs written to object storage), then a
single redb write transaction performs an optimistic-concurrency check on
every base pointer and either flips them all or none.

```text
       prepare metadata files       single redb txn
TableCommit ──┐                    ┌─ check base = current ┐
TableCommit ──┼─► fileio.write_to ─┤  …for every table…    ├─► commit
TableCommit ──┘                    └─ swap pointer         ┘
```

If iceberg-rust later exposes `TableCommit` construction publicly, the
ergonomic story improves. Until then, `atomic_release` is most useful when
you build commits via crates that have direct access to internal helpers
(e.g. a nornir release driver).

## Storage layout

Inside the redb file:

| keyspace          | key                                                 | value              |
|-------------------|-----------------------------------------------------|--------------------|
| `namespaces`      | `<catalog>\x1f<ns_path>`                            | empty marker       |
| `namespace_props` | `<catalog>\x1f<ns_path>\x1f<prop>`                  | property value     |
| `tables`          | `<catalog>\x1f<ns_path>\x1f<table>`                 | metadata location  |

Where `<ns_path>` is the dot-joined namespace ident (`"a.b.c"`). One redb
file may host multiple logical catalogs by reusing different `<catalog>`
prefixes; in practice you'll point each catalog at its own file.

## Known shortcomings (0.1.3)

* **Schema evolution is not callable yet.** `iceberg-rust 0.9.1` seals the
  `TransactionAction` trait (`pub(crate)`), so downstream code can't construct
  `AddSchema` / `SetCurrentSchema`. The on-disk metadata already supports
  schema evolution — only the writer API is sealed — so it works here unchanged
  once upstream exposes it (0.10). These Transaction actions **do** work today:
  `update_table_properties`, `fast_append`, `replace_sort_order`,
  `update_location`, `update_statistics`, `upgrade_format_version`.

* **No orphan-file garbage collection.** A commit that fails *after* writing
  metadata blobs to object storage but *before* the redb pointer-swap commits
  leaves those blobs behind — harmless but unreferenced. Reap them with
  Iceberg's standard orphan-file procedures; there is no built-in sweeper.

## License

Licensed under [Apache License, Version 2.0](../LICENSE-APACHE), matching the
upstream Apache Iceberg project.