Gunnar

nordisk/skade

.nornir/read-scan-decode-guide.md

<!-- nornir:class: guide -->
# Reading a whole table fast — what `Table::read` does for you

> **Class:** `guide` · **Function:** `read`. How the full-scan read behaves and
> when it takes which path. You call one method; skade picks the cheapest correct
> decode. See `read-scan-decode-design.md` for the shape.

## Just call `read()`

```rust
let batches = table.read().await?;   // full current-snapshot scan → Arrow
```

You never choose a decode strategy — `read_all` does:

| Table shape | Path | Cost |
|---|---|---|
| append-only, single schema, no deletes, **local** warehouse | stream each file from its fd, all-core (gatling) | fastest, low memory |
| append-only, single schema, no deletes, **remote** FileIO (S3) | read whole bytes via FileIO, all-core (gatling) | fast; holds file bytes in memory |
| schema evolution / any delete file | Iceberg engine `to_arrow` (merge-on-read) | correctness-complete |
| no current snapshot | zero batches | free |

All paths return the **same rows** — the only difference is how much I/O and
memory it takes to get them.

## What changed (and what didn't)

For a **local** warehouse (anything opened with `Warehouse::open` / `skade::open`)
a full scan no longer reads every data file's whole bytes into memory before
decoding — it streams each file straight from disk. On a big table this drops the
peak memory from *the whole dataset* to *the batches you get back*, and speeds the
scan (no serial pre-read in front of the decode).

Nothing about the result changes: same batches, same order, same schema. A
remote/S3 warehouse behaves exactly as before (whole-bytes read through FileIO —
the only thing an object store can serve).

## Want fewer rows / only some columns?

`read()` materializes the whole snapshot. If that's more than you need:

* `read_columns(&["a", "b"])` — project columns (less I/O + decode).
* `read_filtered(&filter, &cols)` — push a predicate down (file/row-group pruning).
* `read_limited(max_rows)` — stop after N rows (cancels the rest of the scan).
* `lookup(&key, as_of)` — a single row by key (bloom-pruned raw-Parquet fast path).

## Observing it

`skade/read` emits a `functional_status` marker on each full scan (gated
`#[cfg(feature = "testmatrix")]`). In tests, the `LOCAL_STREAMED_READS` /
`BYTES_SLURP_READS` counters (white-box, `#[cfg(test)]`) tell you which decode
strategy a scan took — used to prove a local warehouse takes the streamed path.
The `skade.read_local_stream` bench arm A/Bs the two strategies (rows/sec +
the whole-dataset byte footprint the slurp holds that streaming avoids).
</content>