Gunnar

nordisk/znippy

.nornir/plugins-guide.md

# Package-aware plugins — guide

> **Class:** `guide` · audience: a techie writing a plugin or driving a handler.
> The *what* is in [`plugins-design.md`](plugins-design.md); siblings [`iceberg-guide.md`](iceberg-guide.md) · [`signing-guide.md`](signing-guide.md).

![znippy system](assets/znippy-system.svg)

## f0 — writing a plugin (the whole workflow)

Two moves ship an ecosystem handler:

1. **Implement the six trait methods** of `ZnippyPlugin` (`znippy-common/src/plugin.rs`):
   `meta`, `run_command`, `extract_metadata`, `schema_fields`, `matches_path`,
   `extract_batch`.
2. **Register it** in `znippy-cli/src/handlers.rs::builtin_handlers()`. Done — the CLI
   autodiscovers it via `meta()`; there is no name table to touch.

A native plugin builds **twice**: `cdylib` (native host) + `rlib` (compiled to WASM).

## f1 — filling the trait

```rust
struct MavenPlugin;

impl znippy_common::ZnippyPlugin for MavenPlugin {
    fn meta(&self) -> HandlerMeta {                       // the discovery record
        HandlerMeta { name: "maven".into(), aliases: vec!["mvn".into()],
                      ecosystem: "maven".into(), extensions: vec!["jar".into()],
                      /* subcommand API … */ }
    }
    fn matches_path(&self, p: &Path) -> bool { p.extension() == Some("jar".as_ref()) }
    fn schema_fields(&self) -> Vec<Field> { vec![/* only maven's columns */] }
    fn extract_metadata(&self, path: &Path, data: &[u8]) -> RecordFields {
        // read ONLY pom.xml out of the jar — filter before explode:
        let pom = ljar::decompress_jar_filter(data, "pom.xml");
        pom::parse(&pom).unwrap_or_else(|_| filename_fallback(path))   // never panic
    }
    fn extract_batch(&self, items: &[BatchItem]) -> _ { /* optional zero-copy, once */ }
    fn run_command(&self, cmd: &str, args: &[String]) -> _ { /* `coords`, etc. */ }
}
```

Rule of thumb: parsers must **fall back to filename parsing** on garbage or oversized
input, never panic (mirror the `*_bomb_falls_back_to_filename` tests).

## f2 — driving a handler from the CLI

| you want… | run |
|---|---|
| list the register | `znippy handlers` |
| compress with a built-in | `znippy compress --format rust\|python\|maven -i <dir> -o out.znippy` |
| pick a handler by alias | `znippy compress --format mvn …` |
| use a WASM plugin | `znippy compress --plugin foo.wasm --plugin-type-id N …` |
| run a handler subcommand | `znippy run <handler> <cmd> <args>` — e.g. `znippy run rust coords foo.crate` |

`--format` selects a single ecosystem; **one archive = one package type**. Entry points:
`znippy-cli/src/lib.rs` `Commands::Handlers` (`:516`), `Commands::Run` (`:520`).

## f3 — reading the columns back (typed views)

Each ecosystem's columns come back through its **typed view** — round-trip per ecosystem:

```rust
let archive = ZnippyArchive::open("out.znippy")?;
for row in archive.maven_view()? {        // or rust_view() / python_view()
    println!("{} {}", row.group_id, row.version);   // maven's own columns, out of the union
}
```

Views recover only their ecosystem's branch of the DenseUnion. Round-trip coverage lives
in `plugin_read_api_test.rs`.

## f4 — znippy-plugin-git: sizing and reading the exploded cache

The git store keeps a **read cache** of unpacked objects beside the verbatim packs.
It has a byte budget, it evicts, and none of it is data — every row is re-derivable
from the pack by `resolve::materialize_at`. What the *design* of that is:
[`plugins-design.md`](plugins-design.md) §"what a push costs on DISK". What you do
with it:

### Choose what a push explodes eagerly

```
ZNIPPY_GIT_EXPLODE=commits   # DEFAULT: commits and tags. Trees and blobs are lazy.
ZNIPPY_GIT_EXPLODE=graph     # + trees eagerly. Fast closure walks, 3.24x the pack on linux.
ZNIPPY_GIT_EXPLODE=full      # everything eagerly. The 22.4x shape; a bench arm, not a setting.
ZNIPPY_GIT_EXPLODE=off       # nothing eagerly; every pack is re-queued on reopen.
```

An unknown value is an **error**, never a silent default: a typo that quietly
disables a paid tier is not noticed until a bill is wrong.

### Size the cache

```
ZNIPPY_GIT_EXPLODE_BUDGET=8G          # per repository
ZNIPPY_GIT_EXPLODE_BUDGET=unlimited   # a deliberate answer, not an absent one
```

Unset takes `min(10 % of the filesystem holding the store, 8 GiB)`. In-process:
`GitStore::set_explode_budget(bytes)` — it takes effect at the next rotation, never
refuses a read and never deletes on the caller's thread.

### Read the band

```rust
let band = store.explode_cache_use();
println!("{}", band.band());          // "3.1G/8G"
band.eager_bytes;   // the graph pool — never evicted
band.lazy_bytes;    // what reads have cached
band.direct_reads;  // reads served by inflating straight from the pack, nothing cached
band.evicted_bytes; // what the budget has already reclaimed, measured before the unlink
```

`exploded_cache::use_all()` gives `(store root, CacheUse)` for **every** open store in
the process without taking a store lock — the shape a dashboard tick wants; it is the
same `Weak` registry `progress::snapshot_all` uses.

### Empty it

`GitStore::drop_explode_cache()` unlinks every lazy segment and keeps the graph pool.
That is what `repo gc` calls.

### Do not expect every read to be cached

The cache admits on **rebuild cost**, not on being read. A whole object is one inflate
of a mapped byte range — microseconds — so shallow (depth ≤ `CHEAP_DEPTH`) and small
(< `CHEAP_BYTES`) objects are read directly and nothing is written. On a real
repository that is about six reads in ten. A test that asserts "the second read is
served from the table" must ask `direct_reads` how many were declined, or it is
asserting that the policy is switched off.