Gunnar

nordisk/znippy

.nornir/iceberg-guide.md

# Iceberg store & seal — guide

> **Class:** `guide` · audience: a techie growing an archive live, then sealing it.
> The *what* is in [`iceberg-design.md`](iceberg-design.md); siblings [`plugins-guide.md`](plugins-guide.md) · [`signing-guide.md`](signing-guide.md).

![Arrow-IPC seal flow](assets/arrowipc_seal_flow.svg)

## f0 — grow live, then freeze (the whole workflow)

Two phases. First compress with metadata in an Iceberg **warehouse** (dynamic, resumable,
queryable); later **seal** it into a static content-addressed `.znippy`. Both need the
`iceberg` feature:

```bash
cargo build --features iceberg
```

## f1 — dynamic compress into a warehouse

```bash
znippy compress -i <dir> -o sidecar.znippy \
    --meta-format iceberg --warehouse <dir>
```

- `sidecar.znippy` holds **pure blobs**; all metadata goes into the warehouse
  (`catalog.redb` + Iceberg tables).
- The store is **durable**: kill the process and re-run `compress` with the same
  `--warehouse` to **resume** — new commits land against the existing rows.
- It is **queryable live** while you write — point DuckDB or Polars at the Iceberg tables.

## f2 — seal to a static artifact

```bash
znippy seal -i sidecar.znippy --warehouse <dir> [--namespace NAME] -o sealed.znippy
```

`sealed.znippy` is **content-addressed and blob-preserving**: blobs copied **verbatim** (no
recompress), only the metadata tail rewritten. The result is **byte-identical** to a
one-shot `znippy compress` of the same content, and opens with the ordinary reader:

```rust
let archive = znippy::ZnippyArchive::open("sealed.znippy")?;   // warehouse not needed
```

Entry point: `znippy-cli/src/lib.rs` `Commands::Seal` (`:479`).

## f3 — write → restart → resume → seal, no warehouse

Against a native `.znippy` directly (via `ArrowIpcSinkAppend`,
`znippy-common/src/meta_sink_append.rs`):

1. `open_existing(path)` reopens the archive, finds `blob_end` (lowest `index_offset`), and
   recovers existing rows from the sorted lookup.
2. `append_files()` codec-compresses new files, appends their blobs **past `blob_end`**, and
   re-seals the merged old+new rows — the tail (lookup/trie/manifest/`ZNPYMIDX` footer) is
   rewritten, blob bytes untouched.

Appending a `relative_path` the archive **already contains REPLACES it** (last writer wins): the
pre-existing rows for that path are dropped from the re-sealed index and its old blob bytes
become unreferenced dead payload. `AppendReport::rows_replaced` counts them. Without that, the
archive would carry two live row-sets for one path and a read would silently return the stale
copy — replace semantics is what makes `znippy append` safe to re-run over a changed directory.

The full cycle is byte-exact — see
`znippy-iceberg/tests/lifecycle_seal.rs::iceberg_write_restart_resume_seal_static`.