# znippy index — design
> **Class:** `design` · **Function axis:** `f0` (the seekable footer) + `f1` whole-index open · `f2` `locate_file` resolver · `f3` external SQL · `f4` metadata browse.
> Strict: this file states *only the shape we run*. Every *why*, every alternative, lives in [`index-reasoning.md`](index-reasoning.md); the how-to is in [`index-guide.md`](index-guide.md). The engine that consumes these rows is in [`decompress-design.md`](decompress-design.md).
## f0 — the seekable footer (the whole)
znippy is a content-addressed, seekable store: **O(key) per-file seeks**, no full scan. That
property lives in a self-describing footer written at finalize. Layout, in order:
```text
[ blobs ]
[ data sub-index per (pkg_type,repo) ]
[ lookup sub-index (base cols, re-sorted by relative_path,chunk_seq) ]
[ fst trie blob ]
[ manifest (Arrow IPC) ]
[ 8-byte MAGIC "ZNPYMIDX" ]
[ u64 manifest_offset ]
```
The 16-byte footer (MAGIC + `manifest_offset`) is **unchanged** from the pre-seekable format,
so old readers still open new archives. Base columns (schema `znippy-common/src/index.rs`):
| column | meaning |
|---|---|
| `relative_path` | file path inside the archive |
| `chunk_seq` | chunk ordinal within the file (**write order — never reordered**) |
| `fdata_offset` | byte offset of this chunk in the reassembled file |
| `compressed` | codec flag (false ⇒ raw copy) |
| `uncompressed_size` | decoded chunk length |
| `blob_offset` / `blob_size` | position + length of the chunk's bytes in `[ blobs ]` |
| `checksum` | 32-byte `blake3`, bound to this `chunk_seq` (Law 3) |
Two derived structures ride in the **ordinary manifest** under reserved discriminants so old
readers skip them: `pkg_type = i8::MIN` (`RESERVED_PKG_TYPE`, `index.rs:312`) with
`module_name = "__znippy_lookup__"` (`LOOKUP_MODULE`, `index.rs:293`) and `"__znippy_trie__"`
(`TRIE_MODULE`, `index.rs:298`). `index.rs:594 read_znippy_manifest` filters both out, so the
data view is unaffected.
- **LOOKUP sub-index** — the base columns re-sorted by `(relative_path, chunk_seq)`, plugin
columns stripped: a **second sorted copy**. The main index keeps its original `chunk_seq`
write order (Law 3 untouched). Sorting makes a file's chunks contiguous and lands paths
byte-lexicographic (what `fst` needs).
- **TRIE** — `fst::Map<relative_path → first lookup row>`. Exact lookup is O(key length);
prefix/range scans fall out for free.
**Every** reserved `module_name` is listed once in `RESERVED_MODULES` (`index.rs`), which
`is_reserved_module` is the only reader of — so adding a reserved section is one edit, not
two that can disagree. Beyond the lookup and the trie the catalog carries the two `sign`
sections, `__znippy_meta__`, and the `git` package format's three:
`__gunnar_oid__` (raw `stree` over the first 8 bytes of each oid → its first lookup row),
`__gunnar_graph__` (the commit graph as Arrow) and `__gunnar_reach__` (roaring reachability
bitmaps). A writer adds its own through `ArrowIpcSink::with_reserved_builder`, which is
handed the **final sorted lookup** at seal time — so an extra index addresses the rows the
archive actually has — and which **refuses** a module name `is_reserved_module` does not
know, because such a section would be merged into the data index and corrupt `list`,
`decompress` and the iceberg sink.
Both are built once by `meta_sink.rs:148 write_lookup_and_trie` during finalize (append path:
`meta_sink_append.rs:245`).
## f1 — whole-index open (`ZnippyArchive`)
`znippy-common/src/archive.rs:73 ZnippyArchive::open` reads the **whole index up front** into a
`HashMap<String, FileEntry>` — O(n) parse + memory per open, then O(1) per lookup.
`archive.rs:239 extract_file(relative_path)` pulls a file from that map. This is the shape for
**one archive, many files**: pay the parse once, amortize over many extractions.
## f2 — the `locate_file` resolver
`znippy-common/src/index.rs:736 locate_file(path, target)` never materializes the whole index.
It returns the target's contiguous `ChunkLoc` run (`index.rs:342`) via a **three-tier
fallback, fastest first**:
| tier | mechanism | cost | used when |
|---|---|---|---|
| **a** | trie `fst::Map` → first lookup row | O(key) | trie present (current archives) |
| **b** | binary search of the sorted lookup's `relative_path` column | O(log n) | trie absent |
| **c** | linear scan of the merged main index (`locate_file_via_index`, `index.rs:902`) | O(n) | legacy archives (no lookup) |
Once the first row is found, it expands to the contiguous chunk run; `get_file`
(`decompress.rs:318`) then `pread`s each blob, `decompress_into` (or raw), `blake3`-verifies,
places at `fdata_offset`. Use for one/few files ("Holger asks for a path").
**Cold-read cost the design pays:** a cold `locate_file` re-decodes the whole lookup, so
per-call latency grows ~linearly with N. Acceptable for occasional seeks; a cached reader is
future work (see [`index-reasoning.md`](index-reasoning.md)).
## f3 — external SQL (no znippy code)
The lookup sub-index is a **valid Arrow IPC stream** sorted by `relative_path`. Slice it out of
the archive by the manifest's `index_offset` / `index_len` and hand the bytes to
DuckDB / Polars / DataFusion:
```sql
SELECT blob_offset, blob_size, fdata_offset, compressed, uncompressed_size, checksum
FROM lookup
WHERE relative_path = 'org/foo/Bar.class'
ORDER BY chunk_seq;
```
Zero znippy code runs: the on-disk index *is* the query surface.
## f4 — metadata browse (no bytes)
Two calls list files without reading any blob, returning `ArtifactMeta { relative_path,
uncompressed_size (summed across chunks), chunk_count, compressed }` (`index.rs:774`):
| fn | returns |
|---|---|
| `index.rs:787 get_all_files_meta(path)` | every file, one slim row each, sorted |
| `index.rs:797 get_files_meta_with_prefix(path, prefix)` | one subtree — the trie's `map.range().ge(prefix)` seeks in O(prefix len) |