Gunnar

nordisk/Skidbladnir

.nornir/archive-design.md

# archive — design

> **Class:** `design` · **Function axis:** `f2`. States only the shape we ship; the *why* and the
> znippy-consolidation plan are in [`archive-reasoning.md`](archive-reasoning.md).

![The Archive seam](assets/archive-format.svg)

## The `Archive` trait — the engine seam

`pub trait Archive` (`archive.rs:86`) is the one seam every caller uses; the engine behind it is
swappable without touching the pipeline.

| method | signature | what it does |
|---|---|---|
| `pack` | `pack(staging, version, out) -> BundleManifest` (`archive.rs:88`) | seal a staging tree into a bundle, return the manifest |
| `list` | `list(bundle) -> Vec<String>` (`archive.rs:91`) | list relative entry paths |
| `extract_one` | `extract_one(bundle, entry, dest)` (`archive.rs:94`) | extract one entry atomically (temp → rename) |
| `extract_all` | `extract_all(bundle, dest_dir)` (`archive.rs:96`) | extract the whole bundle, tree preserved |
| `read_entry` | `read_entry(bundle, entry) -> Vec<u8>` (`archive.rs:102`) | read one entry's bytes (znippy overrides for O(1)) |
| `verify_entries` | `verify_entries(bundle, manifest) -> Vec<String>` (`archive.rs:117`) | return the bad-entry paths (missing **and** extra both count) |
| `delta` | `delta(old, new_staging, version) -> Patch` (`archive.rs:143`) | binary delta against a prior manifest (znippy) |
| `apply` | `apply(old_bundle, patch, new_bundle) -> BundleManifest` (`archive.rs:151`) | reconstruct a new bundle from an old one + a patch (znippy) |

## The content-addressed manifest

Every bundle carries a sidecar `<bundle>.manifest.json` (`archive::manifest_path`, `archive.rs:338`;
`load_manifest`, `archive.rs:343`):

- `EntryDigest { path, sha256, len }` (`archive.rs:41`) — one per entry, the content-addressed unit.
- `BundleManifest { version, entries, root_sha256 }` (`archive.rs:52`); `verify_root()`
  (`archive.rs:65`) recomputes the root over the sorted entries and compares.
- Hash helpers: `sha256_hex` / `sha256_file` (streaming) — thin seams that CONSUME the
  shared `nornir-hash` leaf (`sha256_hex` / `sha256_file` / `Sha256Hasher` / `hex_encode`)
  rather than hand-rolling `Sha256::digest` + a `{b:02x}` hex loop (L5 dedup); digests are
  byte-identical to the retired twin.

This manifest is what `fetch::verify_on_receipt` (f3) gates the unpack on.

## `TarZstd` — the always-on default

`pub struct TarZstd { level }` (`archive.rs:192`; `new()` → zstd level 3, `archive.rs:198`). Pure
Rust (`tar` + `zstd`), always builds and runs offline. `impl Archive` (`archive.rs:203`):

- `pack` (`archive.rs:204`) — two-pass: content-digest the tree (`collect_digests`), stream
  tar→zstd into `out.tmp`, atomic rename, write the sidecar manifest.
- `verify_entries` (`archive.rs:296`) — a **single streaming pass** (`O(total bytes)`), hashing
  each entry on the fly; an archive entry absent from the manifest is pushed bad (`archive.rs:329`),
  a manifest entry absent from the archive is pushed bad (`archive.rs:332`).
- `read_entry` uses the trait default (temp-extract one entry).

## `ZnippyArchive` — feature `znippy` (REAL znippy-core, one canonical format)

`pub struct ZnippyArchive { level }` (`#[cfg(feature = "znippy")]`) is a **thin adapter over the
real znippy stack** (`../znippy`) — L5 dedup: there is now ONE archive format in the constellation,
not a Skidbladnir look-alike twin. A bundle sealed here IS a genuine `.znippy` archive, readable by
the `znippy` CLI / DuckDB / Polars. On-disk format (the real znippy layout):

```text
[ zstd/OpenZL chunk blob region ][ Arrow-IPC sub-index(es) ][ manifest + global footer (ZNPYMIDX) ]
```

- **seal** — `pack` content-digests the tree for the SHA-256 sidecar manifest, then
  `znippy_compress::compress_dir` writes the real archive (`seal` renames the forced-`.znippy`
  output onto the caller's exact path). Per-chunk **blake3** is znippy's own integrity (Design law 3).
- **selective read** — `read_entry` calls `znippy_common::get_file`: locate the entry via the lookup
  sub-index, `pread` only its chunk blobs, blake3-verify each, reassemble — never decompresses the
  whole archive. `list` uses `znippy_common::ZnippyArchive::open(..).list_files()`; `extract_all`
  uses the parallel gatling `decompress_archive` (blake3-verified, zip-slip-safe, fails on any
  corrupt file).
- **delta** — `delta` keeps entries whose hash is unchanged and ships only new blobs (deduped);
  `apply` reconstructs the target tree (kept entries via `get_file` from the old bundle, added from
  the patch), then **re-seals it as a fresh real znippy bundle** — the result reproduces the target's
  content exactly (same per-entry SHA-256 → same manifest `root_sha256` as a clean pack). `Patch`
  carries `kept`/`added`/`added_blobs`; `shipped_bytes()` and `reused()` are the metrics.

## Security guards

- **zip-slip** — `reject_unsafe_entry_path` rejects absolute paths and `..` that climbs above the
  root; enforced on every znippy `extract_one`/`read_entry`/`apply`. `decompress_archive` applies
  its own `safe_output_path` gate on full extract.
- **extra-entry / tamper** — `verify_entries` treats extra and missing entries as bad; a tampered
  archive byte fails znippy's per-chunk blake3 (surfaced by `read_entry`/`verify_archive_integrity`);
  a tampered manifest fails `verify_root`.

## Contract (inject-assert tests)

`pack_then_extract_roundtrip_with_manifest_checksums`, `tampered_manifest_fails_root_verify`,
`verify_rejects_extra_entry_absent_from_manifest` (TarZstd); and over REAL znippy-core:
`znippy_seal_verify_unpack_roundtrip_over_znippy_core`,
`znippy_tampered_byte_fails_verification` (RED-when-broken),
`znippy_content_addressing_identical_entries_round_trip`,
`znippy_selective_read_returns_one_entry`,
`znippy_delta_round_trip_reproduces_target_and_ships_only_changes`,
`znippy_extract_rejects_zip_slip_paths`, `znippy_full_extract_round_trips`.