Gunnar

nordisk/znippy-zoomies

.nornir/chunk_revolver-unsorted.md

# Magazine → chunk_revolver consolidation (kill the fork)

> znippy's "Magazine" slot pool (`znippy-common/src/slotpool.rs:194`) is a FORK of the
> canonical `gatling::revolver` — its own header says it *"Replaces the per-thread
> ChunkRevolver rings."* Same gatling-style principle: **add the missing mode to the
> canonical engine, then drop the fork.**
>
> **Phase 1 is DONE** (2026-08-25, `gatling/src/revolver/coalesce.rs`). **Phase 2 is
> open** and is znippy-side work.

## The capability Magazine had that chunk_revolver lacked
**Coalescing / shared-slot packing.** Magazine packs many small files into ONE
fixed-size slot as variable-width slices (invariant: a slice ⊆ one file AND ⊆ one
slot), with **per-slot reference counting** (AtomicUsize armed at `publish`, freed
when the last worker `release_one`s) and a **blocking `claim()`** (backpressure).
chunk_revolver was 1-write = 1 fixed-size slot, no ref-count, non-blocking
`try_get_chunk`.

## Phase 1 — coalescing mode in chunk_revolver — **DONE**
`gatling/src/revolver/coalesce.rs`, additive: `ChunkRevolver::new` / `try_get_chunk` /
`return_chunk` / `Chunk` are untouched and `tests/chunk_revolver.rs` still passes.

| Magazine (fork) | canonical |
|---|---|
| `Magazine::new` / `from_plan` / `with_slice_size` | `ChunkRevolver::with_coalescing(num_slots, slot_size, slice_size)` (`coalesce.rs:255`) |
| `Magazine::claim` (blocking) | `claim_slot()` (`:314`); `try_claim_slot()` (`:324`) is the non-blocking twin the fork never had |
| `Clip::{slot_id,remaining,writable,commit_slice}` | `Filler::{slot_id,remaining,committed,writable,commit_slice}` (`:381`, `:393`) |
| `Clip::publish -> Vec<Round>` | `Filler::publish() -> Vec<PackedSlice>` (`:448`), and `publish_into(&mut Vec<_>)` (`:431`), which recycles |
| `Round` + `unsafe as_slice` | `PackedSlice` (`:66`) + `unsafe as_slice` (`:94`), same seven fields |
| `Ejector::release_one` | `Returner::release_one` (`:153`), also `ChunkRevolver::release_one` (`:299`) |
| `Magazine::{returner,slot_size,slice_size,num_slots}` | `returner()` (`:290`), `chunk_size()`, `slice_size()`, `num_slots()` |
| — | `is_coalescing()` (`revolver.rs:144`) |

Three deliberate divergences from the plan sketch and from the fork:

1. **No `slot_ring: Vec<u8>`.** A coalescing slot has no ring affinity worth preserving
   (its consumers are N workers on one global queue, not one worker on one ring), so the
   free list is flat. `arm_coalescing` (`revolver.rs:105`) drains the per-ring free lists
   into it, which is also what makes the two modes mutually exclusive **by construction**
   rather than by a runtime flag: `try_get_chunk` on a coalescing pool can only return
   `None`.
2. **`Mutex<Vec<u32>>` + `Condvar`, not a channel.** Touched once per slot, never per
   slice; keeps crossbeam out of this leaf crate.
3. **`publish_into` recycles the slice buffer.** The fork allocates a fresh `Vec<Round>`
   per claimed slot and grows it one payload at a time; `publish_into` drains into a
   caller-owned buffer and returns its own to a pooled scratch stack.

Proof: `tests/chunk_revolver_coalescing.rs` (5 tests — coalescing lifecycle with the slot
freed only on the Nth release, 4 workers reading published slices against a still-packing
reader, un-published drop returns the slot, empty publish frees immediately, no
cross-mode double-issue). **NOT proven: miri.** The plan asks for a miri run over the
ref-count orderings; this box has no usable nightly (`cargo +nightly miri` → "Missing
manifest in toolchain 'nightly-x86_64-unknown-linux-gnu'"). Orderings are byte-identical
to the fork's (`store(n, Release)` / `fetch_sub(1, AcqRel)`), which is an argument, not a
proof.

Bench: `benches/revolver_coalescing.rs`, best of 9 interleaved rounds over 133 650 real
files (985.7 MiB, mean 7 733 B/file) on an AMD Ryzen AI 7 PRO 360 (8C/16T), 16 workers,
64 MiB reservation for every arm:

| arm | best s | MiB/s | files/s | allocs |
|---|---|---|---|---|
| fixed-slot | 0.171 | 5747.6 | 779 322 | 156 |
| magazine-fork (replica) | 0.143 | 6875.6 | 932 258 | 1319 |
| coalescing | 0.135 | 7290.6 | 988 530 | 1289 |
| coalescing+reuse | 0.137 | 7206.5 | 977 124 | 168 |

Canonical beat the fork replica in all five interleaved runs taken, by 5–15%, and
allocates 7.9× less. **Honest negative:** coalescing is *not* a throughput win over
fixed-slot once the corpus is warm — the pipeline is memcpy-bound and every arm lands
within noise. Cold, coalescing LOSES (1404 vs 1872 MiB/s): 8 MiB slots cycling 64 MiB blow
a cache that 64 KiB slots keep hot. The reason to have the mode is the **capability** —
64 MiB buys 1 024 tiny files in flight fixed-slot, or ~8 700 coalesced — not the speed.

## Phase 2 — switch the znippy consumer + delete Magazine — **OPEN (znippy-side)**

`Magazine` is consumed at **three sites, all in one file**, exactly as the plan predicted
("sole consumer"):

- `znippy-compress/src/slot_packer.rs:660` — `let pool = Magazine::from_plan(plan);`
- `znippy-compress/src/slot_packer.rs:821` — `let pool = Magazine::from_plan(plan);`
- `znippy-compress/src/slot_packer.rs:1119` — `pool: &'p Magazine` (the pack helper)

plus type-only mentions in the same file (`Round` at `:111`, `:197`, `:216`; `Ejector` at
`:153`, `:200`; `Clip` at `:1121`) and one test-side `PoolPlan` import at
`tests/tests/precompressed_big_files.rs:48`.

Mechanical map, then delete `znippy-common/src/slotpool.rs`:

```
Magazine::from_plan(plan)   -> ChunkRevolver::with_coalescing(plan.num_slots,
                                                              plan.slot_size,
                                                              plan.slice_size)
Magazine::claim()           -> ChunkRevolver::claim_slot()
Magazine::returner()        -> ChunkRevolver::returner()
Clip                        -> Filler          (same methods)
Clip::publish()             -> Filler::publish_into(&mut reused)   [preferred]
Round                       -> PackedSlice     (same seven fields, same unsafe as_slice)
Ejector                     -> Returner
Ejector::release_one        -> Returner::release_one
```

`PoolPlan` **stays in znippy**: `DEFAULT_SLOT_SIZE` / `DEFAULT_NUM_SLOTS` /
`slice_size_for` are znippy's output-affecting cut policy, not pool mechanics, and their
tests live there. Keep the type, move only the pool it constructs. Add
`znippy-zoomies` (or `rotaryengine` directly) to `znippy-compress`'s manifest if absent.

Gates, unchanged: round-trip BYTE-IDENTICAL
(`fallback_byte_identical_to_io_uring` etc.), and a before/after RELEASE bench of
small-file archiving (`xtask nornir-bench archive_*`) — revert on >1–2% regression. The
zero-copy pwrite path must stay identical-or-better.

## §2 of the slot-pool memory plan — disk-backed slots — still proposed
`.nornir/slot-pool-memory-plan.md:91` proposes a disk-backed slot mode (tmp-file-backed
slots that can be evicted under pressure) on top of §1's `PoolPlan`. Untouched by Phase 1
and independent of it: it is a change of where a slot's bytes live, not of how slots are
claimed, packed or released. If it is ever built, it belongs in `coalesce.rs` behind the
same `Filler`/`PackedSlice` surface so no consumer changes.