nordisk/znippy

.nornir/compress-guide.md

# The compress engine — guide

> **Class:** `guide` · audience: a techie compressing a directory or a stream with znippy.
> The *what* is in [`compress-design.md`](compress-design.md); the *why* in [`compress-reasoning.md`](compress-reasoning.md).
> This file is the workflow.

![The Gatling flow](assets/gatling_flow.svg)

## f0 — zero to an archive (the whole workflow)

Three moves take a directory to a seekable `.znippy`:

1. **Build.** `cargo xtask build --suite ORIGINAL --atom znippy` — the ONE build door,
   which reads `.nornir/build-variants.toml`. (`znippy-cli` is the LIBRARY that holds the
   code; the binary is `znippy`, and since 2026-08-03 it is the only one.)
2. **Compress.** `target/release/znippy compress -i ./mydir -o mydir.znippy` — this drives
   `Commands::Compress` (`znippy-cli/src/lib.rs:299`), which calls `znippy_compress::compress_dir`
   (`znippy-compress/src/slot_packer.rs:88`).
3. **Verify.** `znippy list mydir.znippy` — every original file appears, big ones as several
   `chunk_seq`-ordered rows sharing one `relative_path`.

That is the Gatling directory path end-to-end: io_uring reader → 200 MB slots → N workers →
one finalizer → Arrow index.

## f1 — the compress CLI

```
znippy compress -i <dir> -o <out.znippy> \
  [--no-skip] \
  [--format rust|python|maven] \
  [--meta-format arrow-ipc|iceberg] [--warehouse DIR] \
  [--sign KEY --sign-cert CERT --sign-alg p256|ed25519]
```

| flag | effect |
|---|---|
| `-i <dir>` | input directory to pack |
| `-o <out.znippy>` | output archive |
| `--no-skip` | compress **every** slice — disables the "store-raw for `.jar`/`.gz`/`.crate`/`.png`/`.parquet`" skip rule |
| `--format rust\|python\|maven` | ecosystem-aware layout/ignore profile for the input tree |
| `--meta-format arrow-ipc\|iceberg` | container tail backend; `iceberg` needs `--warehouse DIR` |
| `--sign KEY --sign-cert CERT --sign-alg p256\|ed25519` | sign the archive |

## f2 — driving the engine from Rust

Directory in, archive out:

```rust
use std::path::PathBuf;
znippy_compress::compress_dir(
    &PathBuf::from("mydir"),
    &PathBuf::from("mydir.znippy"),
    /* no_skip = */ false,
    /* plugin  = */ None,
)?;
```

In-memory / streaming (HTTP, holger) over `Arc<Vec<u8>>` rounds — same worker/writer/finalizer
shape as the directory path:

```rust
// default inline ArrowIpcSink:
let mut c = znippy_compress::compress_stream(&PathBuf::from("out.znippy"), /* no_skip = */ false)?;
// …or choose the metadata backend (e.g. an Iceberg sink) via a factory:
let mut c = znippy_compress::compress_stream_with_sink(
    &PathBuf::from("out.znippy"), false, Some(sink_factory))?;
```

## f3 — codec and store-raw behaviour

Compression is `Codec::compress_into(&mut self, input, out)`
(`znippy-common/src/codec.rs:42`) — OpenZL zstd/lz4 into a **reused** `out` buffer. You get
store-raw automatically in two cases (index `compressed` flag = `false`): an already-compressed
extension on the skip path, and any slice whose frame ends up **≥** its input. Pass `--no-skip`
when you want the extension skip disabled and every slice pushed through the codec anyway.

## f4 — picking the metadata backend

Default is inline Arrow IPC (`ArrowIpcSink`, `znippy-common/src/meta_sink.rs:73`): sub-indexes +
lookup + fst trie + manifest + `ZNPYMIDX` footer, all appended after the blobs in the `.znippy`.
`--meta-format iceberg --warehouse DIR` swaps in a warehouse-backed sink via the
`MetaSinkFactory` (`meta_sink.rs:43`) without changing the blob layout. Both share the
compress engine unchanged.