<!-- nornir:class: guide -->
# Un-serialized committer — `ingest_pipelined_overlap` — guide
> **Class:** `guide` · **Function:** `ingest_pipelined_overlap`. How to use the
> overlap-commit fast lane and when to prefer it over `ingest_pipelined` /
> `ingest_multi`.
## What it does
`Table::ingest_pipelined_overlap(groups, files_per_commit, channel_depth)` (and
the free `skade::ingest_pipelined_overlap[_props]`) bulk-ingest exactly like
`ingest_pipelined`, but the committer runs **beside** the writer instead of after
it: the `fast_append` commit fsyncs of already-written file-groups overlap the
FileIO writes of the later groups. The resulting table is **identical** — same
rows, same order, same per-`files_per_commit` snapshot grouping, same commit
count — only the commit *timing* changes.
```rust
let mut t = wh.create_table("events", &schema).await?;
// 64 files, commit every 8 → 8 snapshots; up to 4 writes in flight, whose
// FileIO round-trips overlap the committer's fsyncs.
let stats = t.ingest_pipelined_overlap(groups, 8, 4).await?;
println!("{} rows in {} commits", stats.rows, stats.commits);
```
`WriteProps` control (compression, bloom, row-group size, dictionary) is applied
through the `Table` builder setters (e.g. `.compression(ZSTD)`) exactly as for the
other ingest methods; the free `..._props` form takes an explicit `&WriteProps`.
## The knobs
- **`files_per_commit`** — files folded into each `fast_append` snapshot. Fewer
files per commit = more snapshots = more fsyncs (and more overlap opportunity);
more files per commit = fewer, larger snapshots. Identical meaning to
`ingest_pipelined`.
- **`channel_depth`** — how far the writer may run ahead of the committer: the
bound on both in-flight FileIO writes **and** the hand-off channel. Larger =
more overlap and more peak memory (≈ `channel_depth` encoded files buffered);
it backpressures the writer when the committer (fsync) falls behind.
## When to use which
| Situation | Use |
|---|---|
| Single table, many commit groups, latency-bound FileIO (S3 / slow fsync) | **`ingest_pipelined_overlap`** — hides the serial fsync tail behind writes |
| Single table, one atomic snapshot for the whole load | `ingest_parallel` (or `ingest_multi` with one table) |
| Several tables in one flush, want **one** fsync total | `ingest_multi` |
| You need all-or-nothing across the whole ingest | `ingest_multi` (one atomic redb txn) — **not** the overlap path |
| Conservative / proven serial-commit path | `ingest_pipelined` |
## The one caveat
Like `ingest_pipelined`'s multi-commit behaviour, the overlap path is **not**
atomic across its commits: if a later write or commit fails, an earlier snapshot
prefix may already be durable. No *partial* snapshot is ever produced (a cancelled
`redb` txn rolls back cleanly), but the ingest is not all-or-nothing. If you need
that, reach for `ingest_multi`. This is why the overlap path is a separate opt-in
function rather than a silent change to `ingest_pipelined`.
## Verifying
`skade/tests/ingest_overlap.rs` proves, by direct equality to the serial
`ingest_pipelined`, that the overlap path is byte/row-identical, preserves row
order, and produces the same snapshot grouping. The `skade.committer_overlap`
bench arm measures the throughput win (commit-bound shape); the authoritative
numbers come from the Loki HEAVY pass.