<!-- nornir:class: design -->
# Searchable archive metadata — the `__znippy_meta__` sub-index
**Status:** implemented (2026-07-22, oden)
**Question it answers:** *which entries in this archive carry key `X`, and what
are their values* — read from an index, never by streaming the payload.
**First consumer:** dwarves' **build thing** (a wasm module describing how to
build an app, embedded in artifacts and pulled from a registry). Finding one
inside an archive without extracting the archive is the same problem as reading
it from an ELF section without executing the binary.
---
## Why this is a sub-index and not a new format
A sealed v0.7 `.znippy` already ends:
```
[ blob_0 … blob_N ][ data sub-idx(es) ][ lookup ][ trie ][ manifest ][ ZNPYMIDX ][off]
```
The 16-byte footer gives the manifest offset with one seek; the manifest frames
every section by `(offset, len)`. A metadata region that is independently
readable at O(index) cost is therefore already a solved problem in this format —
it only needed a module. `META_MODULE = "__znippy_meta__"` is registered exactly
like `LOOKUP_MODULE` and `TRIE_MODULE`, written through the existing
`ArchiveMetaSink::push_subindex`, and read back through the existing reserved-
section reader.
**There is deliberately no second directory.** The metadata rows key on
`relative_path` — the same identity the data sub-indexes and the lookup already
use — and a hit is resolved to bytes through the existing `locate_file` /
`ArchiveReader`. Nothing here duplicates the index, so nothing here can drift out
of agreement with it. (This was an explicit constraint: twinned sources of truth
bit this constellation three times in one session.)
---
## On-disk shape
Arrow IPC, one `RecordBatch`, schema metadata carrying `znippy_format_version`
(so a section from a newer znippy is refused rather than mis-parsed — the same
guard every other sub-index gets).
| column | type | null | meaning |
|---|---|---|---|
| `relative_path` | Utf8 | yes | the entry this fact is about; **NULL = archive-level** |
| `key` | Utf8 | no | the search key |
| `value_type` | Int8 | no | 0 Str · 1 I64 · 2 F64 · 3 Bool · 4 Bytes |
| `v_str` | Utf8 | yes | exactly one of the five is non-null, |
| `v_i64` | Int64 | yes | chosen by `value_type` |
| `v_f64` | Float64 | yes | |
| `v_bool` | Boolean | yes | |
| `v_bytes` | Binary | yes | |
Columnar rather than a serialized blob so the section stays directly queryable by
DuckDB/Polars — `SELECT relative_path FROM … WHERE key = 'build-thing'` — which
is the same property the rest of znippy's index layer sells.
**Rows are stored sorted by `(key, relative_path)`.** Key-major, so `find_by_key`
and `find_by_prefix` are `partition_point` ranges (O(log n) + O(hits)), not scans,
and archive-level rows for a key sort ahead of its entry-level rows (NULL first).
`value_type` and the value columns are cross-checked on read: a row that declares
a string with a NULL string column, or an unknown tag, is an **error**. Nothing
defaults. The section is attacker-reachable input.
---
## API
```rust
// write
let mut t = MetaTable::new();
t.insert("app/main.wasm", "build-thing", MetaValue::Bytes(wasm))
.insert("app/main.wasm", "build-thing.abi", "wasi-p2")
.insert_archive("producer", "znippy");
create_archive_with_meta(path, &files, 3, Some(t))?; // fresh
append_files_with_meta(path, &more, 3, Some(extra))?; // merge into existing
ArrowIpcSink::new(f, b).with_meta(t) // compress path, via MetaSinkFactory
// read
match read_archive_meta(path)? {
ArchiveMeta::NoMetadata => …, // nothing was searched
ArchiveMeta::Index(ix) => {
ix.find_by_key("build-thing"); // &[MetaEntry] — entry refs
ix.find_by_prefix("build-thing"); // sweeps build-thing.abi too
ix.archive_value("producer");
ix.keys(); ix.iter(); ix.len();
}
}
// or, keeping the distinction at the call site:
meta.find_by_key("build-thing").hits() // Option<&[MetaEntry]>
```
Queries return **entry refs**, so the caller reads the index, decides, and only
then extracts the one entry it wants:
```rust
for hit in meta.find_by_key("build-thing").hits().unwrap_or(&[]) {
let bytes = ArchiveReader::open(path)?.read_file(hit.path().unwrap())?;
}
```
### CLI
```
znippy append -i a.znippy -a dir \
--meta 'app/main.wasm=build-thing=@build.wasm' \
--meta 'app/main.wasm=build-thing.abi=wasi-p2' \
--meta-archive 'producer=znippy'
znippy meta -i a.znippy -k build-thing # exit 0 — index exists, hits
znippy meta -i a.znippy -p build-thing --paths-only
znippy meta -i a.znippy -k absent # exit 1 — searched, no match
znippy meta -i old.znippy -k anything # exit 2 — NO index; nothing searched
```
Values are typed by shape, narrowly and predictably: `@file` → raw bytes,
`true`/`false` → bool, integer → i64, decimal → f64, anything else → string
(so `1.0.2` stays a string rather than half-parsing).
---
## `NoMetadata` is not an empty index
This is the part with a scar on it. An archive written before this module has no
`__znippy_meta__` entry; a reader that turns that into an empty result set will
answer *"searched, found nothing"* about an archive it never searched.
So:
* `read_archive_meta` returns `ArchiveMeta::{NoMetadata, Index(_)}` — **not**
`Option<MetaIndex>`, and with **no `Default`**, so there is no
`unwrap_or_default()` that quietly collapses them.
* `ArchiveMeta::find_by_key` returns `MetaSearch::{NoMetadata, Hits(&[..])}`,
which carries the distinction all the way to the call site. `Hits(&[])` means
*searched, found nothing*; `NoMetadata` means *there was nothing to search*.
`MetaSearch::hits()` is `Option<&[MetaEntry]>` — naming the absent case is the
point.
* On the write side the same distinction is a real on-disk difference:
`create_archive_with_meta(.., None)` seals **no section** (byte-identical to
`create_archive`, and what every pre-existing archive looks like), while
`Some(MetaTable::new())` seals a **present, empty** section.
* The CLI carries it into exit codes: `2` no index · `1` searched, no match ·
`0` hits.
### The re-seal trap, closed
`ArrowIpcSinkAppend::open_existing` truncates the entire metadata tail and
rebuilds it. Without carrying the metadata section forward, **every append would
silently erase it** — and the reader would then honestly report `NoMetadata` for
an archive that used to have some, which is the worst possible version of this
bug because it looks correct. `open_existing` therefore recovers the section into
the sink, and a plain `append_files` (which says nothing about metadata)
preserves it. Asserted in
`meta_sink_append::tests::metadata_is_searchable_survives_a_reseal_and_absence_is_reported_as_absence`.
### One more trap, closed
`push_subindex` accumulates every pushed batch into the random-access lookup. Its
guard was `!= LOOKUP && != TRIE`; the metadata sub-index is Arrow IPC and does go
through `push_subindex`, so its key/value rows would have been folded into the
lookup as bogus chunk locations. The guard is now `!is_reserved_module(..)`. The
test asserts the lookup and the data index still contain exactly the file rows.
---
## Cost — measured, not asserted
Two independent forms of evidence, because a timing bench on a shared box can be
contaminated and a structural fact cannot.
**Structural (always-on test).** The bytes a search reads are the footer, the
manifest and the metadata section. `a_search_reads_only_the_index_and_its_size_is_independent_of_the_payload`
builds two archives with identical metadata and payloads 128× apart, and asserts
those byte counts are **equal**: 129 952 B on a 0.6 MB archive and 129 952 B on a
13.7 MB one — 0.95 % of the larger archive. This runs in the ordinary suite, so a
regression that made the search touch the payload fails CI.
**Timing (`--ignored` benches).** Metadata held constant at 512 keyed entries
(1538 rows); only the payload grows. The `verify all` column is the **control** —
it reads and hashes every chunk, so it is O(payload) by construction, and the
bench refuses to call a flat search number evidence unless the control actually
moved. (An earlier draft used single-entry extraction as the control; the numbers
showed it stays flat too, because znippy's random access is *also* O(1) in
payload. It is now reported but not used as the control.)
### `oden · 32 cores` — non-heavy
| entries | bytes/entry | archive on disk | search `build-thing` | hits | extract 1 entry | CONTROL: verify all |
|--------:|------------:|----------------:|---------------------:|-----:|----------------:|--------------------:|
| 1024 | 8 192 | 1.3 MB | 224.9 µs | 512 | 264.0 µs | 1 003.4 µs |
| 1024 | 131 072 | 13.7 MB | 223.8 µs | 512 | 368.3 µs | 5 110.3 µs |
| 1024 | 1 048 576 | 106.4 MB | 182.5 µs | 512 | 951.6 µs | 35 010.8 µs |
payload **82.4×** · search **0.81×** · control **34.9×**
### `oden · 32 cores` — HEAVY
| entries | bytes/entry | archive on disk | search `build-thing` | hits | extract 1 entry | CONTROL: verify all |
|--------:|------------:|----------------:|---------------------:|-----:|----------------:|--------------------:|
| 4096 | 32 768 | 14.2 MB | 206.1 µs | 512 | 931.9 µs | 5 917.7 µs |
| 4096 | 524 288 | 204.4 MB | 188.9 µs | 512 | 1 185.4 µs | 72 712.3 µs |
| 4096 | 2 097 152 | 813.3 MB | 190.1 µs | 512 | 2 275.2 µs | 265 935.5 µs |
payload **57.1×** · search **0.92×** · control **44.9×**
Reported figure is the **minimum** of 25 repeats — the least contaminated
estimator on a box with other agents on it. **New host row (`oden`, 32 physical
cores, SMT off, 499 GB).** Never merge with the `loki · 12 cores` tables.
Run them:
```
cargo test -p znippy-tests --release --test meta_search_bench -- --ignored --nocapture
ZNIPPY_BENCH_HEAVY=1 cargo test -p znippy-tests --release --test meta_search_bench \
-- --ignored --nocapture meta_search_heavy
```
---
## Not done
* **The compress path does not seal metadata by itself.** `compress_dir` takes a
`MetaSinkFactory`, and `ArrowIpcSink::with_meta` is the injection point, so a
caller can already do it with no change to `compress_dir`'s signature — but no
CLI flag on `compress` wires it, because there is no per-file metadata *source*
for a directory walk yet. The obvious next step is a plugin hook: a
`ZnippyPlugin` that returns `MetaEntry`s for the files it claims, which is how
a maven/python/rust handler would publish an artifact's build thing without the
caller enumerating it.
* **No index over `value`.** Search is by key (exact or prefix). "Which entries
have `build-thing.abi == wasi-p2`" is a filter over the key's hits, which is
fine at these row counts and would need a second sort order to be better.
* **The section is not deduplicated.** Two entries carrying the same 4 MB wasm
store it twice. If build things get large, the value should become a
`relative_path` reference into the archive's own blobs rather than inline
bytes — the format already supports that, and the row model would carry a
`MetaValue::Str(path)` under a `*.ref` key. Not needed yet; noted so it is a
decision rather than an oversight.