Gunnar

nordisk/skade

.nornir/read-path-design.md

# The read path — design

> **Class:** `design` · **Function:** `read-path` · **Uses:** `store`.
> Only the shape that runs; the *why* (why three layers, why no TTL, why foldhash)
> is in [`read-path-reasoning.md`](read-path-reasoning.md).

![The three-layer read accelerator over the redb live tail](assets/read_paths_circuit.svg)

## f0 — the layered resolve

A read resolves **left to right, falling through only on a miss**. The invariant
that makes it coherence-free: *every in-memory layer is keyed by an immutable,
content-addressed identifier* (a `metadata_location` or a snapshot id), so an
entry can only be **evicted**, never **stale**.

```
table_exists / resolve_metadata / load_table
        │
   L1 pointer mirror ──miss──▶ L2 redb `tables`   (source of truth)
        │ (metadata_location)
   L1.5 Table handle cache ──miss──▶ L0 metadata cache ──miss──▶ FileIO read+parse
```

The warm-read entry points on `RedbCatalog`:

| method | source | warm cost |
|---|---|---|
| `table_exists` | `catalog.rs:827` | L1 hit, ~lock-free |
| `resolve_location` | `catalog.rs:107` | L1 hit, no redb |
| `resolve_metadata` | `catalog.rs:99` | L1 → L0, no `Table` build |
| `load_table` | `catalog.rs:790` | L1 → L1.5 clone |

## f1 — L1: the pointer mirror (`pointer_cache.rs`)

`PointerCache` (`pointer_cache.rs:60`) is an `ArcSwap<PointerSnapshot>` wrapping an
`imbl::HashMap<String, Arc<str>, foldhash::fast::RandomState>` (`pointer_cache.rs:42`).
It maps `table_key → metadata_location` for **every** live table.

- Built once from a full scan of the `tables` keyspace at open (`store.rs:187`).
- Maintained **write-through**: each commit path calls `insert`/`remove`
  (`pointer_cache.rs:94`) after its redb commit, still under the redb write lock.
- Reads (`get`, `pointer_cache.rs:80`) load the snapshot lock-free.
- Every published `PointerSnapshot` carries a monotonic `generation`
  (`pointer_cache.rs:46`); `mutate` clones the map (`imbl` structural sharing, O(1))
  and stores a new snapshot, so readers observe a whole snapshot atomically.

A key absent from the mirror means the table does not exist; the redb fallback in
`resolve_location` / `table_exists` is a safety net that only runs on a genuine
miss.

## f2 — L0: the parsed-metadata cache (`meta_cache.rs`)

`MetadataCache` (`meta_cache.rs:34`) is a byte-bounded `moka::future::Cache<String,
Arc<TableMetadata>>` keyed by `metadata_location`. `get_or_load`
(`meta_cache.rs:62`) returns a cached `Arc` on a hit, else reads + parses the
metadata JSON via `FileIO`, coalescing concurrent misses on the same location into
one `read_from` (moka single-flight). Entry weight is a structural estimate
(`weight_of`, `meta_cache.rs:117`); budget defaults to
`DEFAULT_METADATA_CACHE_BYTES` = 128 MiB (`meta_cache.rs:27`). A budget of `0`
disables it.

## f3 — L1.5: the built-`Table` handle cache (`table_cache.rs`)

`TableHandleCache` (`table_cache.rs:42`) is a capacity-bounded
`moka::future::Cache<String, Table>` keyed by `metadata_location`. `get`
(`table_cache.rs:67`) returns a **clone** of a built `Table` (which shares the
underlying `Arc<ObjectCache>`), short-circuiting before L0 — but only if the
cached handle's `identifier()` matches, so two entries aliasing the same metadata
file (via `register_table`) don't cross. `build_and_insert` (`table_cache.rs:79`)
builds on a miss and caches. Default capacity
`DEFAULT_TABLE_HANDLE_CACHE_BYTES` = 256 MiB (`table_cache.rs`); `0` disables. It is a BYTE budget, not a handle count: one handle pins its parsed `TableMetadata` plus iceberg's per-`Table` `ObjectCache`, so a count bounded nothing (512 handles x iceberg's 32 MiB default was ~16 GiB). Each handle's object cache is now pinned to `TABLE_OBJECT_CACHE_BYTES` (4 MiB) and charged at that ceiling.

## Correctness

All three caches share one property: the key **is** the immutable metadata
location (or an immutable snapshot id). A new commit writes a new file at a new
location and flips the pointer, so a stale value is unrepresentable — no TTL, no
invalidation, no coherence protocol. The full argument is in
[`read-path-reasoning.md`](read-path-reasoning.md).