Gunnar

nordisk/skade

.nornir/reasoning.md

# skade — reasoning (f0)

> **Class:** `reasoning` · **Function axis:** `f0` (why this overall shape).
> The descriptive *what* is in [`design.md`](design.md); this file holds the
> *why*, the trade-offs, and the roads not taken. Per-function reasoning:
> [`read-path-reasoning`](read-path-reasoning.md),
> [`commit-reasoning`](commit-reasoning.md),
> [`time-travel-reasoning`](time-travel-reasoning.md).

## Why embedded, why redb — vs a REST catalog service

The obvious alternative to an embedded catalog is a catalog *service* — Nessie,
Apache Polaris, a Hive Metastore. Those were rejected for the nordisk use case
(one process owns the warehouse) because a service means a JVM to run, a network
hop per catalog op, and an operational surface to keep alive. skade instead links
as a crate and calls functions: no JVM, no REST endpoint, no Postgres.

redb was chosen as the single-file store over the alternatives:

- **vs SQLite** — SQLite needs `libsqlite3` (a C dependency); the nordisk
  constellation is pure-Rust by policy (airgap-friendly, no FFI). redb is pure
  Rust.
- **vs a log-structured file of our own** — redb already gives ACID, a single
  exclusive file lock, MVCC read snapshots, and crucially **global** write-txn
  atomicity across all tables in the file. That last property is what
  [`atomic_release`](commit-reasoning.md) is built on; we would have had to
  re-implement it.
- **vs "no catalog, just files"** — Iceberg needs an atomic compare-and-swap on
  the table pointer; a bare filesystem doesn't give that portably. redb's write
  txn does.

The cost we accept: **single process.** redb takes an exclusive file lock, so two
processes can't open the same catalog. For the "one process owns the warehouse"
model this is a feature (it is exactly what makes the lock-free pointer mirror
correct — see [read-path-reasoning](read-path-reasoning.md)); if you need
multi-writer, you need a service, and this crate is the wrong tool.

## Why the whole design leans on immutability

Iceberg metadata files are immutable and content-addressed: a commit writes a
brand-new `<uuid>.metadata.json` and flips the pointer. Every architectural
shortcut in skade falls out of that one fact — the caches need no invalidation,
the time-travel index can be an immutable-swap structure, the commit log is
append-only. Rather than fight it, the design makes immutability load-bearing
everywhere. The per-function reasoning docs each trace their piece of this.

## Why two crates (`skade-katalog` vs `skade`)

The published crate is `skade-katalog` — lean: redb + iceberg + caches. The heavy
data-plane deps (arrow, parquet, datafusion) live in a separate detached
workspace crate `skade` so the catalog's manifest and lockfile stay small and its
airgap/pure-Rust story stays clean. A consumer that only needs a catalog pays for
only a catalog. The data-plane design notes (read-delta / CDC, object-store,
S3-ingest, the arrow-58 fork and its perf patch) are preserved alongside this
book as `reasoning` chapters; they describe the `skade` crate, not the published
`skade-katalog` surface this manual documents.

## Rejected / deferred

- **A built-in REST endpoint** — deliberately out of scope; if you need REST, put
  an axum shim in front (the bench harness has one). Keeping it out keeps the
  library dependency-light.
- **Multi-process access** — see above; a non-goal, not a gap.
- **Full schema-evolution `TableUpdate`s** — bounded by what iceberg-rust 0.9.1
  exposes publicly; unblocks on upstream 0.10. See
  [`catalog-design`](catalog-design.md).