# Package-aware plugins — design
> **Class:** `design` · siblings: [`plugins-guide.md`](plugins-guide.md) · [`iceberg-design.md`](iceberg-design.md) · [`signing-design.md`](signing-design.md).
> Strict: this file states *only the shape we run* — the trait, the registry, the built-ins. No alternatives, no *why*.
## f0 — the plugin system (the whole)
znippy is a content-addressed, seekable compressor: an Arrow-IPC index over zstd/lz4
chunks with blob dedup. A **plugin** teaches that index about **one packaging
ecosystem** (rust, python, maven, media, …). One archive carries **one** package type,
selected at write time by `--format`.
A plugin is any type implementing `znippy-common/src/plugin.rs` `trait ZnippyPlugin`
(`:82`). The writer composes the archive's Arrow schema as **base columns + the union of
every registered plugin's fields**, laid out as a **DenseUnion**; each plugin fills
**only its own columns**. Two rules make the system hold:
| law | statement | enforced by |
|---|---|---|
| **P-1** | one archive = one ecosystem | `--format` selects a single handler at write |
| **P-2** | a plugin writes only the columns it declared in `schema_fields()` | DenseUnion layout; writer routes per-plugin |
| **P-3** | discovery is by record, never a name table | CLI reads `meta()`; no hand-maintained list |
| **P-4** | a decompress-bomb never panics | garbage/oversized input **falls back to filename parsing** |
**Reuse map (what the plugin subsystem `uses`):** `plugin.rs` depends only on `arrow`
(schema/arrays) and the archive's `BatchItem` borrow. It is *used by* the writer (schema
composition), by every native plugin crate, and by `wasm_loader.rs` (`.wasm` plugins).
The whole workspace is tokio-free here — plugins do no async I/O.
## f1 — the `ZnippyPlugin` trait
`pub trait ZnippyPlugin` (`znippy-common/src/plugin.rs:82`). Six members:
```rust
fn meta(&self) -> HandlerMeta; // :93 DISCOVERY record
fn run_command(&self, cmd: &str, args: &[String]) -> Result; // :107 handler subcommands
fn extract_metadata(&self, path, data: &[u8]) -> RecordFields; // :112 per-file
fn schema_fields(&self) -> Vec<Field>; // :117 the Arrow columns it adds
fn matches_path(&self, path) -> bool; // :121 which files it claims
fn extract_batch(&self, items: &[BatchItem]) -> …; // :131 optional zero-copy, once
```
- `meta()` returns a **`HandlerMeta`**: name, aliases, ecosystem, extensions, subcommand
API. It is the *only* discovery surface — the CLI never hard-codes handler names.
- `matches_path` decides file ownership; `schema_fields` declares the columns the plugin
contributes to the composed schema.
- `extract_metadata` runs **per file**; `extract_batch` is an **optional** zero-copy pass
that borrows `&[BatchItem]` and runs **once, before compression**, for handlers that
benefit from seeing the whole set at once.
## f2 — the `PluginRegistry` and schema composition
`PluginRegistry` (`plugin.rs`) holds the registered handlers and is the seam the writer
uses to build the final schema and fill the union:
| fn | at | role |
|---|---|---|
| `schema_fields` | `plugin.rs:287` | folds base columns + every plugin's `schema_fields()` into the composed DenseUnion schema |
| `extract_batch` | `plugin.rs:257` | drives each plugin's optional once-before-compression borrow pass |
Each plugin fills only the union branch it declared; the reader recovers per-ecosystem
columns through **typed views** (`f4`).
## f3 — the built-in register, and the two places a handler's source lives
Built-ins register at **one point**: `znippy-cli/src/handlers.rs::builtin_handlers()`.
The CLI **autodiscovers** the register through `meta()` — there is no second name table to
keep in sync. Both tiers below register through that one function and are
indistinguishable to `znippy handlers` and to `--format`; the tier decides only where the
*source* sits.
### P-5 — the tier rule
> A handler's source lives in **`znippy-common/src/plugins/`** if and only if it needs
> nothing beyond `std`, `arrow` and `znippy-common` itself. The moment it needs a
> third-party parser, a decompressor, a network fetch, its own crate-type or its own
> release cadence, it is a **`znippy-plugin-*` crate under `plugins/native/`**. If it is
> an ABI shim over one of those, built for `wasm32-unknown-unknown`, it is a crate under
> **`plugins/wasm/`**. There is no fourth place.
The reason is one-directional: `znippy-common` sits *under* every other crate in the
workspace, so a dependency added there is paid by all of them, on every target including
`wasm32`. Neither shape is a staging area for the other — a handler is born into the tier
its dependencies dictate, and a `znippy-plugin-*` crate at `src/` root is the **normal**
Rust layout, not a misplacement.
**Why tier A did not move into `plugins/native/` (2026-08-05).** The obvious tidy-up —
one crate per handler, all under one directory — is blocked by a package cycle, not by
taste. A `znippy-plugin-cargo` crate needs `znippy_common::plugin::ArchiveTypePlugin`, so
it must depend on `znippy-common`; keeping the old path alive would need `znippy-common`
to depend back on it, which cargo rejects. Without the re-export, the *published* path
`znippy_common::plugins::cargo_native::CargoPlugin` disappears — and it has a live
out-of-workspace consumer: `holger/traits/src/lib.rs` names six of the handlers in
**non-test** code, plus four holger repository crates in tests. The move would strand
holger until six new crates were published. The dependency-weight argument that would
justify it does not exist either: MEASURED 2026-08-05 on oden, gunnar's `cargo xtask
guard` reports 281 crates / 267 third-party both before and after the plugin move, and
znippy appears **zero** times in that graph — `gunnar-coldtier` is `exclude`d from
gunnar's workspace on purpose. The six handlers import only `std` and `arrow`, so they
add no crate to anyone. What they cost is compile time in every `znippy-common`
consumer, which is real but is not what the guard measures.
### Tier A — in-common handlers (`znippy-common/src/plugins/native/`)
Promoted, real handlers: `cargo_native.rs` (1), `npm_native.rs` (6), `rpm_native.rs` (8),
`deb_native.rs` (9), `gem_native.rs` (11), `conda_native.rs` (14). Alongside them
`skeletons.rs` holds the *stubs* — filename-only `coords`/`extract_metadata`, promoted
in place once someone writes real parsing (go 4, nuget 5, elf 7, flatpak 10, docker 12,
helm 13, snap 15, appimage 16, composer 17, hex 18 …). `mod.rs` is the module list.
These files import **only** `std`, `arrow` and `crate::` — zero third-party crates. That
is P-5 holding. A `.wasm` plugin loads through `wasm/wasm_loader.rs` (`WasmPlugin`, behind
feature `wasm-plugins`).
**The tier-A source is itself split `native/` + `wasm/` (2026-08-05).** Same rule as the
repo-root `plugins/` directory, one tier down: no plugin source sits loose beside
`plugins/mod.rs`. The two root directories hold **crates**, the two here hold **modules** —
which tier a handler is born into is still decided by its dependencies, and the move
changed no handler code (nine pure `git mv` renames). `mod.rs` `pub use`s each module back
to its old path, because `znippy_common::plugins::<x>_native::<X>Plugin` is API of a
**published** crate: `holger/traits/src/lib.rs` names six of them in non-test code, and
several `holger/server/repository/*` crates take `znippy-common` by *version*, so they
follow the published path and cannot follow a directory rename.
`znippy-common/tests/plugin_source_layout.rs` guards both halves from outside the crate.
### Tier B — separate `znippy-plugin-*` crates, all under `plugins/native/`
All six are `rlib` compiled-in host handlers. **maven** and **python** additionally
declare `cdylib`, but nothing loads that host `.so`; the WASM artifact comes from tier W
below, not from these crates.
| crate | id | crate-type | modules under `src/` | what puts it outside `znippy-common` |
|---|---|---|---|---|
| `znippy-plugin-python` | 2 | cdylib + rlib | `wheel.rs`, `native.rs` | `linflate` always, `lzip-parallel` behind `host-decompressors` |
| `znippy-plugin-maven` | 3 | cdylib + rlib | `pom.rs`, `resolver.rs`, `native.rs` | `quick-xml`, `ljar`, and `ureq` — a **network fetch** — behind `resolve`. Reads only pom.xml from a JAR via `ljar::decompress_jar_filter`: filter-before-explode |
| `znippy-plugin-media` | 25 | rlib | `native.rs`, `tests.rs` | five media parsers: `image`, `kamadak-exif`, `lofty`, `mp4`, `matroska` |
| `znippy-plugin-skidbladnir` | 40 | rlib | `member.rs` | its own release cadence — a Skidbladnir *bundle-format* change must not bump `znippy-common` and cascade through the whole publish order |
| `znippy-plugin-rust-toolchain` | 41 | rlib | `member.rs` | same shape and same reason as skidbladnir — sibling airgap-bundle classifier |
| `znippy-plugin-git` | 42 | rlib | `object.rs`, `oid_index.rs`, `graph.rs`, `reach.rs`, `refs.rs`, `secrets.rs`, `pushlog.rs`, `sections.rs`, `bench_kernels.rs` | `znippy-zoomies`, `roaring`, `sha1`/`sha2`. Keyed by **oid hex as `relative_path`**; adds `object_type`/`object_size` and the RESERVED `__gunnar_oid__` / `__gunnar_graph__` / `__gunnar_reach__` / `__gunnar_refs__` / `__gunnar_secrets__` sections. No WASM — the stree hot path needs AVX2 + mmap + prefetch, which WASM has not |
**Where the `ArchiveTypePlugin` impl sits is not uniform, and does not need to be.**
maven/python/media put it in a private `native.rs` re-exported from `lib.rs`
(`pub use native::NativeMavenPlugin`); git/skidbladnir/rust-toolchain implement it in
`lib.rs` directly. `member.rs` is **not** the counterpart of `native.rs` — it is a pure
path→`MemberKind` classifier with no trait impl in it at all.
**Publishing consequence.** Tier-B crates are published separately and are depended on
*by version, from outside this workspace*: holger's `znippy-maven` / `znippy-python`
server crates name `znippy-plugin-maven` and `znippy-plugin-python` from crates.io.
Folding a tier-B crate into `znippy-common` breaks those consumers. The 2026-08-05 move
into `plugins/native/` changed **directories only** — every package name and every
published version is untouched, so crates.io is unaffected; what did move is the
`path = "…"` string in every sibling that names one of these crates
(`holger`, `nornir`, `skade/bench`, `dwarves`, `dwarves/bench`, `gunnar-coldtier`).
### Tier W — the wasm shims, under `plugins/wasm/`
| crate | shims | crate-type | what it holds |
|---|---|---|---|
| `znippy-plugin-maven-wasm` | `znippy-plugin-maven` | `cdylib` (wasm32) | the five exports `wasm_loader.rs` imports, and nothing else |
A tier-W crate is defined by what it does **not** contain. `znippy-plugin-maven-wasm`
declares one dependency — `znippy-plugin-maven` with default features — and reuses its
`pom.rs`, its `linflate` JAR scan and its `row.rs` field writer verbatim. `row.rs` is the
single writer of the column set: the native `schema_fields()` builds Arrow fields from
`MAVEN_SCHEMA`, the wasm `plugin_schema` serializes the same constant, and both call
`maven_fields()` — so the two surfaces cannot drift by construction (LAW 5).
What does **not** cross to `wasm32-unknown-unknown`, and why:
| native only | why |
|---|---|
| `ljar::decompress_jar_filter` | fans DEFLATE across the gatling pool; wasm32 has no threads. The wasm side runs the single-threaded `linflate` scan — the *same* inflate engine ljar fans out over, so the decoded pom bytes are identical |
| `extract_batch` / `gatling_map_balanced` | same: no threads, and the loader has no batch entry point — it calls `extract` once per file |
| `znippy_common::plugin::ExtensionRow`, `arrow` | znippy-common carries arrow, io_uring and gatling. The wire form is JSON, parsed back host-side by `wasm_loader::parse_json_to_row` |
| `resolver.rs` (`ureq`) | a network fetch; no socket in the sandbox and no host import for one |
The JSON ABI is lossy in one specific way and the code accounts for it rather than
papering over it: `parse_json_to_row` produces only `ExtensionValue::Str` and knows no
null, so `fields_to_json` **omits** a null column instead of emitting `null` (which would
land as the literal string `"null"`), and it replaces `,` `:` `"` in a value — that parser
splits on them before it strips quotes, so an unescaped separator would shear a value into
the wrong column.
`tests/maven_native_wasm_agree.rs` is the guard: it loads the real `.wasm` through the
real `WasmPlugin` and compares rows against `NativeMavenPlugin` over a bare `.pom`, a
stored JAR, a **DEFLATE** JAR (the case where the two decompressors genuinely differ), a
classified artifact and a JAR with no pom, plus `schema_fields()` and `matches_path()`.
It fails — never skips — when the `.wasm` is missing or older than any maven source.
## f4 — the read side (typed views)
Because each ecosystem's columns live in a known union branch, the reader exposes a
**typed view per ecosystem** — `maven_view`, `rust_view`, `python_view` — that round-trips
that ecosystem's columns out of the DenseUnion (integration: `plugin_read_api_test.rs`).
## f5 — bomb defense
A plugin's parsers are hardened: garbage or oversized inputs **fall back to filename
parsing** and never panic (`*_bomb_falls_back_to_filename` tests). Metadata quality
degrades to the filename; the archive still writes.
## znippy-plugin-git — the indexer's memory model (lane P, 2026-08-30)
MEASURED 2026-08-29 21:10 UTC on the gunnar appliance (16 GB, 4 vCPU, no swap, PID 1 =
gunnar-server): a `push --mirror` of torvalds/linux (8.12 GiB pack, 13 948 244 objects,
11 215 872 deltas) was acked, then the post-ack indexer grew past the box and the kernel
panicked (`System is deadlocked on memory`). The table in `resolve.rs`'s module doc is the
per-object rule; this is the whole worker, after the fix:
| what | held | bound | where |
|---|---|---|---|
| the pack bytes | mapped, not copied (page cache) | 0 anonymous; `MemAvailable` counts it as reclaimable | `git_ops::absorb_one` → `archive_map` |
| the pack-row SHA-1 | a 1 MiB window | 1 MiB | `indexer::build_row` |
| the walk (`PackEntry` per object) | for the pack | O(objects) — ~56 B each, the one O(pack) structure left | `pack_walk` |
| the delta dependents | one `Vec` per base, for the pack | O(deltas) — `ofs_deps` + `ref_deps`, built once from the walk and live until the pack ends | `resolve::Resolver::run` |
| the oid → offset map and the remaining/stuck vectors | for the pack | O(objects) | `resolve::Resolver::{oid_to_offset,run}` |
| the readers' scanned packs | until a pause or the absorb | O(pending pack) per store, built only when a reader has to answer out of one | `git_ops::Absorber::scanned` |
| delta bases | until the last dependent, **under a byte ceiling** | `max(ZNIPPY_GIT_BASE_BUDGET_BYTES, largest base)`; default 25 % of MemTotal; over it the base whose next dependent is farthest is dropped (Belady) and re-inflated from the pack when asked | `resolve::Resolver::{hold,evict,materialize}` |
| every other payload | one reused buffer | one object | `resolve::Resolver::whole_buf` |
| the delta stream | one reused buffer | one delta | `resolve::Resolver::delta_buf` |
| index rows | one batch | `ZNIPPY_GIT_INDEX_BATCH_ROWS` (262 144) rows, then `objects.append` | `git_ops::absorb_one` row closure |
| exploded rows | one batch | `exploded_arrow::FLUSH_BYTES` (64 MiB) | unchanged |
| the derived graph | after the fold | commits only, O(commits). The tree payloads that used to sit beside it — **28.24 GB, 3.24× the pack, on `torvalds/linux`** — are gone; see the disk model below (lane Q) | `git_ops::refold` |
**The watchdog** (`memguard.rs`): before every pack and every 4096 objects inside one,
`MemAvailable` off `/proc/meminfo`; under the floor (`ZNIPPY_GIT_MEM_BUDGET_PCT`, 20 %) the
resolver drops every held base, flushes the sink and the row batch, and waits (250 ms
re-check; one WARN `index.paused`, one INFO `index.resumed`; cancel ends it). It parks the
indexer worker and nothing else — it is taken before the absorb gate.
**The two checkpoints.** `Absorber::absorb` takes one between packs, deliberately *before*
the absorb gate, so a parked worker holds no gate, no row and no base; `Resolver::checkpoint`
takes one every 4096 objects inside a pack. Both call the same `pause_until_ok`: it parks
while `MemAvailable` is under the budget and returns when `MemAvailable` reaches
`budget + budget / 8`. That eighth is hysteresis — without it an 8 GB box flapped fourteen
pause/resume pairs in a minute — and it is why a pause that trips on a shortfall of a
fraction of a percent needs 12.5 % of headroom to end. The flush runs once as the pause
opens and again every 10 s while it holds, because a reader can build a table *after* the
first flush.
**What a pause drops.** `Absorber::spill` clears the readers' scanned pending-pack rows and
calls `ExplodedTable::drop_caches` (the Arrow oid index and the batch cache);
`Resolver::checkpoint` additionally drops every held base, flushes the sink and writes the
row batch. `index.spilled`'s `scanned_rows_dropped` counts only the first of those, in rows,
so it reads 0 whenever no reader ever had to answer out of a pending pack — a push with no
concurrent clone or fetch drops nothing there. It says nothing about the exploded caches,
which go unconditionally and uncounted, and `clear()` releases the map's `Arc`, not a clone a
reader is still holding.
**The livelock.** What a parked worker still holds is the part that cannot be dropped: the
walk, `ofs_deps`/`ref_deps`, `oid_to_offset` and the `remaining`/`stuck` vectors are all
O(pack) and all live until the pack ends. When those — and not the held bases — are what the
box is short of, the flush frees nothing, `MemAvailable` never climbs by `budget / 8`, and
`pause_until_ok`, which has no timeout and no give-up, parks until the store is cancelled.
The floor still does its job: the box does not panic, the control plane and the read path
keep answering. Bounding the walk-sized structures is what lane Q's lazy explode has to
reach; more spilling cannot get there, because there is nothing left to spill.
**Nobody waits on the indexer.** Reads never take the gate: the rows already batched in are
in the index, and an oid the worker has not reached is resolved read-only from the pending
pack on the reader's thread (`Absorber::scan_pending`, same resolver, bounded bases, no sink).
`reachable` absorbs for itself if the gate is free and otherwise refuses by name with the
progress numbers. `AccountIndexer::drop` cancels before it joins.
**The lazy read path lane Q builds on**: `resolve::Resolver::materialize(offset)` — one
object's content from a pack and an offset with only its chain inflated — and
`Absorber::scan_pending` — one oid (rows + content) from the pending packs.
**Measured** (`resolve::tests::a_bounded_base_set_…`, release): facett pack, 9668 objects:
unbounded peak 731 051 B; budget 1 B → peak 287 000 B (one base at a time), 839 evictions,
7028 re-inflations, identical rows; budget 861 000 B → 0 re-inflations. Plugin lib suite 220/220.
**MEASURED 2026-08-30 on the oden KVM rig (lane P), gunnar `v1.1.3-180-g01ce6c58`, 4 vCPU,
4×75 GiB raid0 xfs `/data`, no swap — the same `push --mirror` of torvalds/linux at two RAM
sizes.** The ack path holds: the 8.12 GiB pack (8 723 679 817 B, 13 948 244 objects, 2874
refs) was spooled to disk over the 64 MiB threshold, mapped, and ingested in 269 s on both
boxes with no packfile written. **Neither box panicked** — the 2026-08-29 kernel deadlock is
gone at 8 GB and at 16 GB. Both then parked and neither index finished. At 16 GB the worker
reached 78 % (11.0M/13.9M) before `index.paused` at `MemAvailable` 3 286 511 616 B against a
budget of 3 361 581 460 B — a shortfall of 75 MB, 2.2 %; at 8 GB it reached 22 % (3.2M/13.9M)
and paused 1.65 MB under a 1 672 962 040 B budget, 0.1 %. Neither resumed: 0 `index.resumed`
in 31m44s and 1h45m41s of parking, `index.spilled` every 10 s throughout (190 and 627 of
them) with `scanned_rows_dropped=0` on every one — no reader had scanned a pending pack, so
the spill had nothing to give, and at 8 GB `MemAvailable` had *fallen* to 1 276 284 928 B by
the time the pause ended, further from the resume line than when it opened. Both pauses ended
only on cancel. The consequence is the push: the connectivity check that follows the ingest
never completed, no reference moved, and the client sat unacked for 2h14m30s (16 GB) and
2h12m24s (8 GB) against stock git's ~11 min for the same mirror. What kept answering while
the worker was parked: the control plane instantly, and `ls-remote` in 4 ms.
**The exploded tier's cost, measured the same run.** `repo.purged` reports what one mirror
had written when it was cancelled: 195 704 627 966 B (182.3 GiB) at 78 % indexed, and
46 641 441 342 B (43.4 GiB) at 22 % — **22.4× and 5.4× the pack, both projecting to ~24–29×
at 100 %**, so linux.git needs roughly 200–235 GiB of `/data` for an 8.12 GiB pack. Delete
reclaims all of it: `bytes_after=0` in 604 ms and ~2 s, `/data` back to 33 MiB on the band.
## znippy-plugin-git — the ACK path, and the two hours it used to take (lane Z, 2026-08-30)
Lane P2 measured an appliance that survived a kernel-sized push and never reported it. This
is where that two hours went, and what the ordering is now.
### What `receive_pack.checking` was doing
The connectivity check reads objects **by oid**. This arm acks a push's *bytes* before its
*oids* exist — that is §13.9, and it is what lands an 8.12 GiB mirror in 4m29s — so the
check's very first `graph.peel` of a pushed tip missed the exploded table, missed the
`objects` index, and fell into `Absorber::resolved`'s third arm, `scan_pending`. That arm
calls `scanned_pack`, which runs **the full resolver over the whole pending pack on the
connection thread** and then throws the rows away:
```text
07:43:49 git.receive_pack.checking commands=2874
09:20:21 index.scanned "a reader scanned a pending pack for itself:
13948244 objects in 5792442 ms"
```
1h36m32s at 16 GB, 1h44m14s at 8 GB, for one answer. It is not a per-ref walk — `S-062`
already batched that — and it does not wait on the indexer. It is worse than waiting: it is
the **same work, twice, at the same time**, two resolvers over one pack on four vCPUs, each
holding its own `PackWalk::entries` and sorted oid table. That second copy is also *why the
watchdog paused*, and why `index.spilled` then reported `scanned_rows_dropped=0` on all 190
and 627 of them: the scan the spill would have dropped was the one still being built, and a
half-built scan is not in the `scanned` map yet. After the scan finally landed at 09:20 the
check still had 13.9 M objects to walk one at a time; the delete cancelled it 32 min later.
### The ordering, and it is git's
`GitServe::index_now()` (git-storage-trait, default `Ok(0)`) — *"make every object this
store has accepted addressable by oid, now"*. `receive_pack::apply` calls it **once**, before
the frontier is built, and everything after it is a point lookup. That is `index-pack` and
then `check_connected`, which is what stock git has always done.
`GitStore::index_now` → `Absorber::index_pending_now`: take the absorb gate, absorb every
pending pack, keep the rows. If the account worker already holds the gate this blocks on it
and then finds `is_absorbed` set — whoever got there first did the work exactly once.
### The read path is deliberately NOT this
*Nobody waits on the indexer* still holds for every other caller, and
`a_read_before_the_drain_is_served_from_the_pack_and_never_waits` is its red test: a clone,
an `ls-remote` and a `control status` must be answered while the drain is parked, and they
still are, through `scan_pending`, without touching the gate. The first version of this
change had every reader index and that test turned the suite from 50 s into a hang.
The connectivity check is the one caller for which waiting is the right trade, because it
*cannot* answer without the pack resolved and the only question is once or twice.
`IN_ABSORB` (thread-local, `InAbsorb` guard) is what makes the re-entry safe: the resolver
asks this same store for delta bases while the gate is held, and that call must absorb the
*other* pending packs inline (lane U's thin-push-on-a-pending-pack case) and must never take
a non-reentrant mutex it already holds.
### The watchdog has three ways out now, not one
| | before | now |
|---|---|---|
| resume line | `budget + budget / 8` — 410 MB of headroom at 16 GB, to leave a pause a 75 MB shortfall opened | `budget + RESUME_MARGIN_BYTES` (64 MiB, absolute, capped at the floor; `ZNIPPY_GIT_MEM_RESUME_MARGIN_BYTES`) |
| deadline | none — 31m44s and 1h45m41s, ended only by the delete | `PAUSE_DEADLINE` 300 s (`ZNIPPY_GIT_MEM_PAUSE_MAX_MS`) → `PauseEnd::Deadline` |
| a flush that frees nothing | invisible: the closure returned `()` | the closure returns a count; three zeros running with memory no better than it started → `PauseEnd::NothingToGive` |
| what a give-up does | goes round the loop again | fails the pack BY NAME (`ResolveError::out_of_memory`), row on the band as `index.gave_up` |
A watchdog that parks for two hours and never resumes is worse than one that says *I cannot
index this pack on this box*: the pack's bytes stay durable, nothing partial is indexed, and
the operator gets a sentence naming what was available, what the floor is, how far the pack
got and the three things they can do about it.
### What an operator can see while it runs
`progress::ProgressTicker` logs `index.progress` every 10 s for as long as a pack is being
absorbed, carrying `peak_rss` and `reinflations` — which rode only on `index.finished`, so
lane P2's two boxes, which never finished a pack, printed neither, in exactly the failure the
two numbers exist for. It is a thread and not a hook in the resolve loop because a hook does
not run while the resolver is parked, which is when the numbers matter most.
**Read `peak_rss` with care.** It is `VmHWM`, which counts resident *file-backed* pages, and
this arm maps the whole pack — so on the linux mirror it reads ~11–13 G on a 16 GB box while
`MemAvailable` is still 11–12 G. `mem_available` on the same row is the honest number and the
one the watchdog acts on.
### MEASURED — the same mirror, the same rig, before and after
oden KVM, 4 vCPU, 4×75 GiB raid0 xfs `/data`, no swap, 16 GB.
`push --mirror` of `torvalds/linux`: 8 723 679 817 B pack, 13 948 244 objects, 2874 refs.
Before = lane P2, gunnar `v1.1.3-180-g01ce6c58`. After = lane Z, `v1.1.3-207-g83ce408`.
| | before | after |
|---|---|---|
| **time to ACK** | **never** — client unacked 2h14m30s, ended by deleting the repo | **3974 s = 66m14s**, `push exit=0` |
| `git.receive_pack.checked` | never fired | `tips=2874 objects=13948244 elapsed_ms=3704687` |
| — the deferred index inside it | two resolvers, neither finished | `indexed_for_check packs=1 elapsed_ms=3225796` (53m46s) |
| — **the connectivity walk itself** | never reached | **478 891 ms = 7m59s** for 13.9 M objects and 2874 refs |
| reader scans of the pending pack | 1, `13948244 objects in 5792442 ms` | **0** |
| index complete | never (78 %) | walk 54 801 ms · resolve 3 159 178 ms · fold 11 346 ms |
| pauses / resumes / give-ups | 1 / 0 / — | **0 / 0 / 0** |
| spills that freed nothing | 190, `scanned_rows_dropped=0` on every one | 0 spills at all |
| re-inflations / evictions | unobservable (rode on a row that never fired) | **0 / 0**, peak bases 627 670 217 B |
| peak RSS (`VmHWM`) | unobservable | 15 078 100 992 B = 14.0 GiB |
| `MemAvailable` at the end | 3 286 511 616 B — *below* the 3 361 581 460 B floor | 5 316 239 360 B — comfortably above it |
| `/data` at ack | 182.3 GiB at 78 % indexed = 22.4× the pack | 27 509 MiB = 26.9 GiB = **3.15×** |
| `repo delete` | 195 704 627 966 B → 0 | 21 328 789 718 B → 0 |
| `git clone` | impossible — 0 refs advertised | **exit 0** in 723 s, `HEAD 08dbfad3f504` = source |
| `git fsck` | impossible | **exit 0** in 810 s (warnings only, and they are upstream's: `missingTaggerEntry` on tags linux.git itself ships without a tagger line) |
**Where the 66 minutes are, stated plainly: 53m46s of it is the single-threaded resolve.**
`Resolver::run`'s fixpoint is a `for` over `remaining` on one thread — 13.9 M objects at
~4 200/s on a four-vCPU box — and it is now the whole cost of an ack. The connectivity walk
that was blamed for the two hours is 8 minutes. Stock git acks the same mirror in ~11 min and
does it by resolving the pack on every core. That is the next thing to fix and it is a
`gatling`/`znippy-zoomies` job, not another ordering change: each entry has at most one base,
so the dependents form a forest and every root's subtree is an independent chain.
**Why there were 0 pauses.** Not because the watchdog was lucky — because the second resolver
is gone. Two O(pack) copies of `PackWalk::entries` + oid table is what put lane P2's 16 GB box
75 MB under its floor; one copy finishes with 4.95 GiB to spare. The livelock fixes above are
still load-bearing (the 8 GB box has a 1.67 GB floor and far less room), and they are covered
by unit tests carrying the rig's own numbers rather than by this run.
## znippy-plugin-git — the resolver is a FOREST, and the forest is the parallelism (lane AM, 2026-08-31)
Lane Z left one number on the table: of the 66m14s an ack took, **52.6 minutes
was `Resolver::run`'s fixpoint on ONE thread** — 13 948 244 objects at ~4200/s
on a four-vCPU box, with three vCPUs idle. Stock git acks the same mirror in
~11 minutes and does it on every core.
### The shape
Every pack entry has **at most one** delta base. One parent per node is the
definition of a forest, so the dependents `resolve.rs` has always collected
(`ofs_deps`, `ref_deps`) are not an incidental index — they are the **children
lists**, and two roots' subtrees share no object and no ordering constraint.
[`crate::forest::DeltaForest`] states that: `parent`, CSR child lists, and the
two root sets, built from `PackWalk::entries` and nothing else (LAW 5). One
round of the fixpoint is now a `gatling_run` fan-out over that round's roots
(NEVER rayon — LAW 3), and each unit is a depth-first walk of one subtree.
### What the shape buys besides the cores
The fixpoint held a base from the moment it resolved until its **last
dependent's position in the pack**, a distance the format does not bound — which
is what lane Q's byte ceiling and Belady eviction exist to cap. A depth-first
walk holds the **path from the root to the current node** and nothing else, and
on a path the shallowest frame is *by construction* the one needed last, so
Belady needs no `next_use` table to know what to drop. MEASURED, the kernel
pack: **peak held bases 627 670 217 B (lane Z) → 87 000 000 B at one worker,
145 400 000 B at four.** The ceiling is one atomic across every worker — N
workers do **not** get N budgets.
### Determinism: keyed by ROOT, and the measurement that says why
The stream out is fixed and worker-count independent: roots ascending, and
inside a root the depth-first sequence the ascending children lists determine.
`the_same_pack_resolves_to_the_same_bytes_at_1_2_and_4_workers` compares the
whole tape — rows, batch boundaries, sink calls, in sequence — byte for byte,
and was seen RED by letting the drain take whichever root was at the front of
the buffer instead of the cursor's.
Keying it by **entry** index is the obvious choice and it is unusable. MEASURED
with `examples/forest_stats` on torvalds/linux:
| | subtree **size** | subtree **index span** |
|---|---:|---:|
| p50 | 1 | 0 |
| p90 | 5 | **5 251 404** |
| p99 | 70 | **11 511 634** |
| max | 11 789 | **12 434 450** |
The subtrees are tiny — ideal parallel work — but a five-object subtree
routinely spans five million entries, because `pack-objects` puts a base beside
its first dependent and the rest wherever they fall. An entry-ordered cursor is
therefore pinned by any subtree in flight whose span reaches the top of the
pack. MEASURED, kernel pack, 4 cores: **1 worker 1454.7 s, 2 workers 1572.9 s**
— slower, 1.24 cores of CPU, 286 686 stalls at the lookahead ceiling. Keyed by
root the cursor moves at the rate roots *finish*, and gatling hands roots out in
ascending order, so it follows the workers instead of pinning them: stalls 0.
### The memory budget of the parallel resolver, declared
| what | bound | where |
|---|---|---|
| the walk, the forest, the oid table | **one copy, shared read-only** — N workers do not multiply them | `PackWalk`, `DeltaForest` |
| held delta bases | `ZNIPPY_GIT_BASE_BUDGET_BYTES` — **one atomic for the whole job**, not one per worker | `Round::live_path` |
| resolved-but-unemitted objects | `ZNIPPY_GIT_RESOLVE_LOOKAHEAD_BYTES`, **32 MiB**, charged at what the allocator charges (`ALLOC_OVERHEAD`), with backpressure | `Round::backpressure` |
| per-worker scratch | `2 x workers x SCRATCH_KEEP_BYTES` (1 MiB), shrunk back after every child | `Scratch::{delta,leaf}` |
| the sink's payload copies | only the kinds the sink says it `wants` — under the shipped `Commits` policy, commits and tags only | `PayloadSink::wants` |
`DeltaForest` **replaces** `HashMap<u64, Vec<usize>>` (2.7 M keys, 2.7 M heap
`Vec`s — ~230 MB on the kernel pack) and the fixpoint's `remaining`/`stuck`
index vectors (111 MB each) with three `Vec<u32>` (156.5 MB) and a bitset
(1.7 MB where the round's index list was 56 MB). The parallel resolver's own
footprint is **smaller** than the fixpoint's, and the numbers below say so.
### MEASURED — what it does, and where it does not fit
Same rig, same mirror, same clock as lane Z's `run.sh`: oden KVM, 4 vCPU,
4x75 GiB raid0 xfs `/data`, no swap, 16 GB, gunnar **v1.1.3-207-g83ce408** —
lane Z's own measured build, so the only thing that differs is znippy's
resolver.
| | lane Z (fixpoint) | lane AM (forest) |
|---|---:|---:|
| resolve rate, peak | ~4 200 obj/s | **52 947 obj/s** |
| 0 → 86 % of 13 948 244 objects | — | ~13 min |
| whole index | 3 159 178 ms (52m38s) | **did not finish at 16 GB** |
| peak RSS | 15 078 100 992 B (14.04 GiB) | 13.7 GiB (4 workers, after the scratch cap) |
| watchdog | 0 pauses | pauses at 86 % (4 workers) / 96 % (2 workers), refuses the pack **by name** |
**The honest verdict: the resolve is an order of magnitude faster and the 16 GB
appliance does not have the headroom to let it finish a kernel-sized mirror.**
Lane Z's own run ended with 4.95 GiB available against a 3.36 GiB floor — 1.6
GiB of margin — and the parallel resolver spends it. It is not the resolver's
own structures (peak RSS is *below* lane Z's): what changes is that a
depth-first walk of a subtree whose index span is five million entries touches
the 8.13 GiB mapped pack all over, at four cursors at once, so far more of the
mapping is resident at any moment and `MemAvailable` — which counts only half
of page cache — falls accordingly. The fixpoint walked the pack in entry order
and had perfect locality.
When the floor is breached the box says so and stops, which is lane Z's law
working: `index.gave_up ... refused by name`, the pack's bytes durable, nothing
partial indexed. One give-up now stops the whole round rather than one per
worker — before that fix a two-worker run burned a second, full 300 s deadline
after the first had already given up.
## znippy-plugin-git — what a push costs on DISK (lane Q, 2026-08-30)
Lane P's table above is what the indexer holds in **memory**. This is what the same
push leaves on the **filesystem**, and it is the question Rickard asked when the
appliance filled up: *"we cannot have one repo take all 300 GB disk."*
### The rule
At push time the store writes **the pack verbatim, the index, and the graph**, and
nothing else. The index is `oid → (offset, len, type, base, uncompressed size)`; the
graph is the commit and tag payloads, because those are what
`graph`/`reach`/a tag peel read and none of them may stall. **Trees and blobs — 90 %
of the objects and essentially all of the bytes — are not written at push time at
all.** The first reader of one inflates that one object's delta chain out of the
mmapped pack (`resolve::materialize_at`, `d` inflates and `d−1` delta applications)
and hands it to a write-back thread, which puts it in a budgeted cache for the next
reader.
### The four files
| file | what it is | grows with | evictable |
|---|---|---|---|
| `objects.pack` | the verbatim pushed packs, appended. **The truth.** | what was pushed | never |
| `objects.pack.journal` | one durable row per acked extent | pushes | never |
| `objects.tail` | the `objects` index — the five facts per object | objects | never (it is the addressing) |
| `objects.exploded` | the EAGER pool: commit and tag payloads | commits | never (it is the graph) |
| `objects.exploded.cN` | the LAZY pool: whatever has been read, one file per segment | **reads**, not pushes | **yes — one `unlink` per segment** |
### What an open store costs in descriptors — **3**, and why the number matters
| file | open file descriptions | who holds it |
|---|---:|---|
| `objects.pack` | **1** | the writer arm (`open_blobs`: read/write, `flock`ed), **shared** with `Absorber::reader` and with the index workers |
| `objects.pack.journal` | 1 | the durable arm's append handle — the ack path, untouchable |
| `objects.tail` | 1 | redb |
| `objects.exploded*` | 0 | opened per operation and closed |
It is not a housekeeping number. gunnar's server divides its `RLIMIT_NOFILE` by
exactly it to decide how many repositories it may keep open at once
(`gunnar_server::limits::open_repo_budget` = `min((soft − 256) / fds_per_repo,
MemAvailable × 20 % / 8 MiB)`), so the count is the divisor of an appliance's
concurrent-repository ceiling. Lane O measured the descriptor exhaustion it
causes when it is wrong: on the rig the **202nd** repository touched since boot
failed with `open journal …: No file descriptors available (os error 24)` and
the ssh accept loop died with it.
It was **5** until 2026-08-30 (lane AH): `objects.pack` was opened three times
in one process — the writer's read/write handle, `Absorber::reader`'s read-only
`pread` fallback, and `IndexerPool::archive`'s read-only handle for the index
workers. All three address the file **positionally** (`pwrite` /
`libc::pread` / `read_exact_at`), the file is append-only and `gc` truncates
nothing, so no cursor was shared and no ordering was observable between them:
two of the three bought a kernel object and a page-cache alias per open
repository and no isolation at all. They are now one `Arc<File>` handed out by
`ArchiveWrite::blobs_shared()`, asked for inside `PushPath::with_absorber` so
that no caller can forget it.
**Nothing about durability moved.** The writer still owns the append cursor, the
`flock` and the blob-then-journal fsync order before the ack; sharing an open
file description changes when it is closed, not what is written or in which
order. One thing improved by accident: the `flock` now lives as long as the last
of the three holders, so a reader outliving the writer can no longer drop the
writer lock out from under a store that is still open.
`tests/fd_per_store.rs` counts them off `/proc/self/fd` per store and asserts
the breakdown above — a descriptor is a kernel object, and reading the code
counts the `File::open` calls you can find rather than the handles that exist.
Eviction is a segment `unlink` and not a rewrite, and that is the whole reason the
tier can have a budget at all: the exploded table is one append-only Arrow IPC
stream, so deleting a row in the middle costs a rewrite of up to 17 GB. Cut into
segment files, reclaiming the coldest bytes is `remove_file` — O(1), no rewrite, no
read amplification, and a crash during it leaves a pool that is merely shorter.
### The budget, and the arithmetic that makes it safe
`min(10 % of the filesystem holding the store, 8 GiB)` per repository
(`exploded_cache::default_budget`, `ZNIPPY_GIT_EXPLODE_BUDGET`, or the account's
quota from gunnar's control document). On the shipped 300 GB appliance the ceiling
binds, so **twenty** repositories cost at most 160 GB of cache together — against
141.7 GB for **one** before the cap existed. On a 64 GiB `/data` the percentage binds
instead and each repository gets 6.4 GiB.
Nothing the budget bounds is data. Every row in either pool is re-derivable from the
pack by `materialize_at`, so an eviction is never a loss and never an error a client
can see — it is a later read paying ~0.5–2 ms to inflate its own chain again.
### MEASURED — `torvalds/linux`, oden, 2026-08-30
8 731 794 676 B pack, 13 948 244 objects, 11 215 872 deltas, into a fresh store on
md1; `st_blocks * 512` per file; budget 8 GiB.
| phase | eager (`Full`, lane P2 on the rig) | `Graph` (commits+trees eager) | `Commits` (**shipped**) |
|---|---:|---:|---:|
| at ack | — | **1.00×** the pack | **1.00×** |
| at index-complete | **22.4× at 78 % indexed**, projecting 24–29× | **4.58×** | see the lane Q rows in `nornir-workspace.toml` |
| after every object is read once | — | peak **4.70×**, settles **4.60×** | — |
`Graph`'s remaining 3.2× was one thing and one thing only: `objects.exploded` held
28 244 349 440 B of **tree payloads**. They were eager because `reach::ObjectFacts`
wanted `&HashMap<String, Vec<u8>>` — every tree in the repository, resident, keyed by
a hex `String`, rebuilt on every fold, which is also why peak RSS was **50.9 GiB** on
a box that ships with 16 GB. That map is now `reach::TreeSource`, one lookup method;
the eager arm is still a `HashMap` (it implements the trait and returns
`Cow::Borrowed`, so it copies nothing) and the store's arm resolves the tree the way
any other object is read.
### The plateau — the answer to "does lazy explode take the disk anyway?"
Yes, unless the tier is capped: a client that browses a whole history reads every
object once, and a write-back-on-first-read tier would then hold exactly what the
eager one held. So the budget is not an optimisation, it is the design.
MEASURED, same run, reading **all 13 948 244 objects once**, 390.8 s, 0 missing:
the store stayed between 4.57× and 4.70× the pack for the entire scan while the cache
took **174 412 527 296 B (162 GiB) of write-backs and gave every one of them back**,
324 segments unlinked. Without the budget those 162 GiB would have been on the disk.
`exploded_cache::tests::the_tier_plateaus_at_its_budget_under_a_full_scan` is the
assertion; `examples/lazy_explode_cost.rs` is the harness that produced the table.
### What it costs on the read side
MEASURED, same store, 200 objects sampled evenly across the pack:
| | first read | second read |
|---|---:|---:|
| p50 | 3.9 µs | 3.4 µs |
| p90 | 44.1 µs | 4.6 µs |
The p50 barely moves because most objects sampled are commits and trees; the p90 is
the blob tail, and 44 µs is what one delta chain out of the page cache costs. The
design note's estimate was 0.5–2 ms per object, so the real number is **an order of
magnitude better than budgeted** — the note assumed a disk seek per link and the pack
is mapped.
## znippy-plugin-git: the writer arms (status 2026-08-30)
`archive_write.rs` has three `WriterArm`s with the SAME journal and, for the two
durable ones, the SAME contract — the blob's bytes are durable before the journal
row that references them:
| arm | what `append` does | contract | status |
|---|---|---|---|
| `fast` | pwrite, no fsync, no journal | none (a bench floor) | bench only |
| `safe` | pwrite → fsync → write(journal) → fsync — four blocking syscalls | durable before ack | **`StoreConfig::DEFAULT`** |
| `uring` | the same four as ONE `io_uring_enter`, `IOSQE_IO_LINK` on the first three, the journal row from a one-page registered buffer (`uring_write.rs`) | durable before ack, ordering enforced by the kernel | **what gunnar's appliance medium ships since 2026-08-30** (`[build.installer.data] io = "uring"` → `korp.data.io=uring`); `ZNIPPY_GIT_WRITER=uring` / `with_writer` elsewhere; a kernel without a ring is refused BY NAME, never a silent pwrite |
`uring_write::probe()` is the door for a server that has to choose before its
first store: one `io_uring_setup` of a two-entry ring + the opcode probe, no
registration, no file. gunnar's PID 1 calls it when the medium's
`korp.data.io=uring` token asks for the arm and falls back to `safe` on a row
that names the errno (`ENOSYS` = `# CONFIG_IO_URING is not set`). The crate
itself never falls back — `UringWriter::create` refuses — because a pwrite
number reported as an io_uring number would be a lie. Ceilings the arm carries
(module docs of `uring_write.rs`): one ring + one pinned page per open store,
charged uid-wide against `RLIMIT_MEMLOCK` unless the process has
`CAP_IPC_LOCK` (PID 1 on the appliance has it); `MADV_NOHUGEPAGE` on the
staging page so mimalloc's huge-page arenas are not charged 512× . The
uring-vs-safe numbers on gunnar's appliance rig (same ISO, token flipped):
gunnar `.nornir/best-results-benchmarks.md` §1b.
**The measurement that made `uring` the shipped arm (lane S, oden 2026-08-30).** One
installed image, the `korp.data.io=` token rewritten on its ESP, five runs per arm,
4 vCPU / 8 GB q35/KVM, xfs `/data`. At 1–8 clients the arms are indistinguishable —
sequential push-create 172 ms vs 174, 8-parallel 1145 vs 1175, a 50 MB blob 1848 vs 1808,
clone and fetch a tie — all inside a ±26 % noise floor taken from the three phases that
never reach a writer (sequential clone, parallel clone, incremental fetch). The one row
with a signal is the deepest queue tested, 200 push-creates from 64 clients at once, where
the five run-medians do not overlap: **safe 5130–6149 ms against uring 4433–4682 ms —
1.14× per accepted op, at ±3 % run spread against safe's ±10 %.** Four syscalls were never
the bottleneck below that depth; what the ring buys is the tail and the repeatability.
Two warnings for anyone re-running it. The 24- and 64-parallel wall clocks are **admission
rows, not writer rows**: gunnar admits 19 concurrent transfers and refuses the rest by name,
so at 64 clients only ~250 of 1000 ops reach a writer and a run that refuses more finishes
sooner. The first single run of that matrix read 1.4–1.55× for `uring` purely because the
two arms refused different counts. Compare medians of ACCEPTED ops, never phase wall clock.
**Durability, on the arm that ships**: three `SIGKILL` power cuts in the middle of an
8-parallel push storm, verifying every `(repo, sha)` the client had seen acked —
**133 of 133 present at the acked sha, 0 missing, 0 wrong**, 34 sampled repositories
`clone --bare` + `cat-file -e` + `fsck` clean, XFS log recovery on `/data` at every boot.
**Measured on the KVM rig 2026-08-30 (16 GB, SeaBIOS band, linux mirror):** the push acked at
4144 s (the reader's one scan of the pending pack, `index.scanned` 13 948 244 objects in
2944 s, 0 re-inflations, is what the ~800 ref updates waited on — their own work, not the
worker's); the band's row climbed `19% 2.7M/13.9M resolve 2.9M/s` → `58% 8.1M` → `80%
11.3M paused 1p` while `ram` went 6.4G → 10G → 13G; the watchdog paused at
`3.08G available < 3.13G budget`. The pause held because the readers' scanned table and the
exploded oid tree were not the worker's to drop — they are now (`Absorber::spill` on every
pause). An 8 GB and a 12 GB box died *before* the ack, in gunnar's `read_pushed_pack` `Vec`
(fixed in gunnar: spooled to disk above 64 MiB).
## znippy-plugin-git: the ref log costs one pass per push, not two per push (lane Y, 2026-08-30)
`refs.rs` is an append-only Arrow push log and `fold` replays it into the current namespace:
*"reading the current refs is therefore a scan of a structure sized by pushes, not by
repository size"*. True, and it was being paid **twice for every single reference update** —
`GitOps::update_ref` folded the whole log through `ref_state()` to compare the caller's
expectation, and then `RefLog::push` scanned it again to re-derive `push_seq`. `RefLog::push`
even said so in its own doc, and offered `append_batch` to a server holding its own counter,
which nothing took.
Two `O(pushes)` passes per push means the cost of landing a reference grows with every
reference the repository has ever landed. Measured on oden, release build, one `update_ref`
(fold the namespace, derive the sequence, append the frame), medians over 20:
| prior pushes in the log | before | after | speedup |
|---:|---:|---:|---:|
| 1 | 0.275 ms | 0.093 ms | 3.0× |
| 100 | 1.457 ms | 0.120 ms | 12× |
| 1 000 | 12.864 ms | 0.490 ms | 26× |
| 10 000 | 134.854 ms | 5.704 ms | 24× |
13.5 µs per prior push, with no ceiling. A repository at ten thousand pushes burned **135 ms
of pure rescan** before its next reference could land.
`RefLog` now carries the folded namespace and the counter forward in memory, loaded by one
lazy scan the first time a handle is asked anything. **The log is still the authority and the
frame is still the transaction.** The crash contract is a property of *where* the cache is
advanced: only after `PushLog::append` has returned, i.e. after the frame is on the platter,
so a failed append leaves the namespace and the counter exactly where they were, and a
restarted process rebuilds both with the single scan recovery has always done. The assumption
added is **one writer per store**, which `GitStore` already required — it serialises reference
work behind a process-local `ref_gate`, and two processes over one store were unsound before
this because that mutex could never have seen the other one.
`PushLog::scans` counts the only `O(log)` operation the type has, so the guard asserts
**passes over the file** rather than a wall clock — one pass for a handle that has pushed 1,
100 or 1000 times, where it was two per push, seen red at every N. No push-create benchmark
can see any of this: a create's log is empty, and on the 2 vCPU appliance rig the sequential
push-create row moved 131.4 → 130.6 ms, i.e. not at all (gunnar `.nornir/best-results-benchmarks.md` §1c).
### And the empty store's two barriers, which were 1.5 ms and 0.0 ms
`SafeWriter::create` and `UringWriter::create` fsynced the journal's schema message into a
fresh journal. A store with no journal row has acked no pack, and gunnar's create is
transactional, so a create that does not finish leaves no catalog row pointing at the store;
the first row a writer claims fsyncs the same file and takes the schema with it. What the
barrier bought — that a torn schema could not be inherited as whole — is bought by
`archive_write::journal_head`, one function both arms ask (LAW 5): an empty file gets the
schema, a file that opens with exactly this build's schema message is left alone, anything
else **no longer than** the schema was interrupted before a row could exist behind it and is
written again, and anything else **longer** carried rows and is reported as corrupt rather
than truncated away.
`GitServe::set_head_projected` is the same idea one level up: a caller that keeps `HEAD`'s
authority elsewhere and re-applies it on every open (gunnar's catalog does) may have the
write without the barrier. A detached `HEAD` is derivable from nothing and keeps it.
**Neither barrier was the cost.** On the appliance rig an `fsync` is ~0.22 ms (`4k sync
4517 ops/s`), and removing these two moved their spans by 1.48 ms and 0.28 ms — one fsync
each — against the 8.9 ms and 6.0 ms those *steps* measure. The step is not the barrier.
## znippy-plugin-git — a THIN pack, and why the archive grows one delta link per push (lane AI, 2026-08-30)
### What a thin pack is
A `git push` does not send whole objects when it does not have to. Having read the
server's ref advertisement, the client knows what the server already holds, so it sends a
**thin pack**: entries whose `REF_DELTA` header names, by oid, a base object *that is not
in the pack*. A one-line change to one file is three entries — commit, tree, blob — of
which two are usually deltas against objects the server acked a moment earlier. This is
the ordinary shape of every push after the first, not an edge case.
Stock git resolves that at receive time and then **completes** the pack:
`git index-pack --fix-thin` appends the resolved base objects into the pack it just
received, so what lands on disk is self-contained and its delta chains never leave it —
bounded by the pushing client's own `pack.depth`, 50 by default.
### What §14 does instead, and the consequence nobody had measured
This store appends the client's bytes **verbatim**. That is the whole reason a clone can
be a byte-range copy rather than a re-pack, and it is not negotiable. But it means the
`REF_DELTA` keeps naming a base in an **earlier pack**, forever:
```text
pack 0 blob c1 whole
pack 1 blob c2 REF_DELTA -> c1 depth 1
pack 2 blob c3 REF_DELTA -> c2 depth 2
...
pack N blob cN+1 REF_DELTA -> cN depth N
```
**A repository pushed one commit at a time grows its delta chain one link per push,
without bound.** Nothing repacks; nothing is supposed to.
### What the absorber does about it
`Absorber::absorb_one` resolves each pending pack with `self` as the
[`BaseSource`](../plugins/native/znippy-plugin-git/src/resolve.rs), and a base outside the
pack is answered in this order — cheapest first, and every step is a point lookup:
1. **§14's exploded table** (`ExplodedCache::content`) — resolved content by oid, no
inflate, no chain.
2. **the `objects` index** — the base's archive offset, then `resolve::materialize_at`
follows *its own chain* out of the mapping: `d` inflates and `d − 1` delta
applications, nothing else in the pack touched.
3. **the pending packs, read-only, on this thread** (`Absorber::scan_pending`) — for a
base in a pack that is acked but not yet indexed. Lane Q's re-entrancy rule holds here:
a nested call may LOOK UP an already-built scan and may not BUILD one.
4. **nothing** — and only then is the pack refused by name, as
`ResolveError::missing_base`.
`Ok(None)` from a base source means *absent* and is the only answer step 4 may be reached
from. `Err` means *held and unreadable*, and it travels: it is the store's failure, not
the pushed pack's, and the two must not arrive as the same sentence.
### 🔴 The ceiling that refused the 101st link
`resolve::MAX_DELTA_LINKS = 100` was a **refusal** — "a chain longer than this is not a
deep delta, it is a loop" — and step 2's walk hit it as soon as a repository had been
pushed a hundred and one times. `Absorber::resolved_costed` then fell through to step 3,
found nothing there (the base's pack was long since absorbed), returned `Ok(None)`, and
`BaseSource for Absorber` had already dropped the real error with `.ok().flatten()`. So
the client was told, about a blob this repository was holding:
```text
this pack is thin: it deltas against 509be6a7… which is not in it, and no base source
could produce that object's content — §14's exploded table does not hold it and it
could not be re-derived from the verbatim packs either, so this repository genuinely
does not have that object.
```
MEASURED on oden, 140 sequential single-commit pushes into one repository: **21 refused,
first at push #120**, and then every push after it, forever — because the absorb loops ran
`for job in &jobs { self.absorb_gated(job)?; }` in ordinal order, so the one pack that
could not be indexed was hit first and aborted the loop for all the packs behind it. At
8 / 24 / 64 clients: 168/1120, 504/3360, 1344/8960 — the same 21 per repository, every
time. Lane AD's two-appliance `EventGap` leg never converged for the same reason: the
primary's `GitOps::reachable` failed on its own store and the standby was re-asked every
300 s for 21 minutes.
**A count cannot tell a deep chain from a loop.** The ceiling is a termination *proof*
now: `resolve::CycleGuard` records the archive offsets a walk has stood on past
`MAX_DELTA_LINKS` and refuses a **repeat**, which is what a cycle actually is. It is
strictly stronger — a three-link loop is caught at its first repeat instead of 100 links
later — and it never refuses a legitimate chain. Under the threshold it is two integer
operations and no allocation, so the hot read path is unchanged. One guard, two walkers:
`serve::resolved_type_at` had a hand-written copy of the same ceiling and shares this one
(LAW 5). After: **0 of 140, 0 of 1120, 0 of 3360, 0 of 8960**, and the `EventGap` leg
converges in 4.1 s.
### What bounds the cost, since the depth no longer does
Nothing caps chain depth, so a read of the newest object of an N-push repository is O(N)
inflates **the first time**. What makes that affordable is lane Q's cache, not a ceiling:
`Absorber::keep_for_the_next_reader` writes back what a read cost
(`ReadCost::depth`), a deep chain is `worth_caching` by construction, and the *next*
push's base is that object — answered at step 1, in one lookup, with no walk at all. The
cost is paid once per depth, by the reader that hit it, and never again.
### A pack that cannot be indexed STOPS, and says why
`MAX_ABSORB_ATTEMPTS = 3`. Three and not one, because an absorb can fail for a reason
outside the pack — the memory watchdog gave up, the repository is being deleted, a
transient read error — and one bad moment must not condemn durable bytes. Three and not
thirty, because the resolve is deterministic in the store's own state and every attempt is
paid for by a client waiting on a push.
`Absorber::absorb_each` is the one loop all four entry points run. It **runs to the end**
and reports the first failure afterwards, so a pack that cannot be indexed no longer stops
the push that arrived after it from being indexed and acked; and a pack that has been
given up on is recorded in `Absorber::unindexable` with the backend's own root cause,
logged once as `index.unindexable`, skipped from then on, and readable through
`GitStore::unindexable_packs()`. Its bytes stay durable and untouched, its objects are
genuinely absent from the index, and every read path already knows how to say that: a push
that deltas against one is refused by name at `external_bases_exist`, and a clone that
needs one fails its connectivity check — both immediately, neither after another futile
whole-pack resolve.