Gunnar

nordisk/znippy

.nornir/index-guide.md

# znippy index — guide

> **Class:** `guide` · audience: a techie querying a `.znippy` index for locations or metadata.
> The *what* is in [`index-design.md`](index-design.md); the *why* in [`index-reasoning.md`](index-reasoning.md). This file is the workflow.

![The Arrow-IPC index seal flow](assets/arrowipc_seal_flow.svg)

## f0 — pick your access shape (the whole)

There are three ways to get a row and two ways to browse metadata. Choose by how many files you
touch and whether you want to run znippy code at all:

| you… | use | cost |
|---|---|---|
| hold one archive, pull **many** files | `ZnippyArchive::open` + `extract_file` (`f1`) | O(n) once, then O(1)/file |
| want **one/few** files | `get_file` / `locate_file` (`f2`) | O(key) per lookup |
| want to query **without znippy** | slice the lookup Arrow IPC into DuckDB/Polars (`f3`) | your engine's cost |
| want a **listing**, no bytes | `get_all_files_meta` / `get_files_meta_with_prefix` (`f4`) | O(n) / O(prefix len) |

Every path is O(key)-seekable because the footer carries a sorted lookup sub-index plus an
`fst` trie — no full scan needed for a single file.

## f1 — many files from one archive

```rust
use std::path::Path;
use znippy_common::archive::{ZnippyArchive, ArchiveReader};

let arc = ZnippyArchive::open(Path::new("archive.znippy"))?;  // reads whole index once
let a = arc.extract_file("org/foo/Bar.class")?;               // O(1) from the HashMap
let b = arc.extract_file("org/foo/Baz.class")?;               // amortized — no re-parse
```

Pay the index parse once, then every `extract_file` is a hash hit. Use this when you'll pull a
batch of files from the same archive.

## f2 — one file, fastest resolver

```rust
use std::path::Path;
use znippy_common::{get_file, index::locate_file};

// full read (locate + pread + verify + reassemble):
let bytes = get_file(Path::new("archive.znippy"), "org/foo/Bar.class")?;

// just the locations (a contiguous ChunkLoc run), no bytes read:
let locs = locate_file(Path::new("archive.znippy"), "org/foo/Bar.class")?;
```

`locate_file` tries the trie first (O(key)), falls back to binary search of the sorted lookup
(O(log n)), then a linear index scan for legacy archives. Never materializes the whole index.
Note: a *cold* `locate_file` re-decodes the lookup, so back-to-back single seeks on a huge
archive pay ~O(N) each — batch through `f1` if you're pulling many.

## f3 — query the index with DuckDB (no znippy code)

The lookup sub-index is a plain Arrow IPC stream sorted by `relative_path`. Slice it by the
manifest's `index_offset`/`index_len`, then point any Arrow engine at it:

```sql
-- lookup.arrow = the bytes you sliced out of the archive footer
SELECT blob_offset, blob_size, fdata_offset, uncompressed_size, compressed, checksum
FROM read_ipc('lookup.arrow')
WHERE relative_path = 'org/foo/Bar.class'
ORDER BY chunk_seq;
```

You now have every chunk's location without linking znippy — pread those `blob_offset`s
yourself, or just inspect the layout. Works in DuckDB, Polars, and DataFusion.

## f4 — browse metadata without reading bytes

```rust
use std::path::Path;
use znippy_common::index::{get_all_files_meta, get_files_meta_with_prefix};

// every file, one slim sorted row each:
for m in get_all_files_meta(Path::new("archive.znippy"))? {
    println!("{}  {} bytes  {} chunks", m.relative_path, m.uncompressed_size, m.chunk_count);
}

// just one subtree (trie range-seek, O(prefix len)):
let sub = get_files_meta_with_prefix(Path::new("archive.znippy"), "org/foo/")?;
```

`ArtifactMeta` carries `relative_path`, `uncompressed_size` (summed across the file's chunks),
`chunk_count`, and `compressed`. No blob is read — this is the cheap way to list or size an
archive's contents.