# PLAN — bound the compress slot-pool memory (Njord backup, issue #23)
**Status:** §1 IMPLEMENTED (2026-07-22, oden) · §2 (disk-backed slots) still proposed
> **2026-08-25 — the pool moved out from under this document.** Everything below
> that names `Magazine` / `slotpool.rs` describes a fork znippy no longer owns:
> the slot pool is now `gatling::revolver` in coalescing mode
> (`znippy_zoomies::chunk_revolver`), and only the CUT POLICY stayed, as
> `znippy-common/src/poolplan.rs`. `PoolPlan::plan` and every bound §1 describes
> are unchanged and still hold — read `Magazine::new` as
> `ChunkRevolver::with_coalescing` and `slotpool.rs` as `poolplan.rs` throughout.
>
> §2 is UNAFFECTED and still open. Disk-backed slots change *where a slot's bytes
> live*, not how slots are claimed, packed or released, so the collapse neither
> blocks nor advances it — but it must now be built behind the upstream
> `Filler`/`PackedSlice` surface, and therefore belongs upstream too, so no
> consumer changes.
## What landed
The 1.6 GiB unconditional reservation is gone. `slot_packer.rs` no longer holds
`SLOT_SIZE`/`NUM_SLOTS` consts; each pass plans its pool with
`slotpool::PoolPlan::plan(input_bytes, num_workers, budget)`, which reserves the
smallest of three bounds — **the old default** (never more than
`8 × 200 MiB`), **the input** (`ceil(input / slice_size)` slices; a 5 MiB tree
reserves 7 MiB, not 1.6 GiB), and **the budget**
(`common_config::slot_pool_budget_bytes()`: `ZNIPPY_SLOT_POOL_BYTES`, else 50 %
of the **cgroup** headroom, else 50 % of host available memory). Floor: one slot
of one slice, so the pipeline always makes progress.
Deviation from the plan below, and the reason: the plan proposed shrinking
`slot_size`, from which `Magazine::new` derives `slice_size`. **`slice_size` is
output-affecting** — it is the big/small partition threshold *and* the big-file
cut length, so it lands in `chunk_seq`, `fdata_offset` and every chunk checksum.
Shrinking it under memory pressure would have made the archive depend on how much
RAM the pod had. So `Magazine` gained `with_slice_size`/`from_plan`, `slice_size`
stays pinned at `200 MiB / num_workers` regardless of the plan, and only the
reservation moves. `slot_size × num_slots` affects nothing but how many slices
coalesce per buffer and how many buffers are in flight; the ordered sink writes in
producer order either way.
Measured on oden (32 cores, 29 workers → `slice_size` 7 231 558 B), debug build:
| input | budget | pool reserved | peak RSS |
|---|---|---|---|
| 2.4 GB | default | **8 × 209 715 200 = 1.6 GiB** (the historical geometry, exactly) | 2 741 MB |
| 765 MB | default | 8 × 94 010 254 = 752 MB (input-bound) | 1 852 MB |
| 765 MB | `ZNIPPY_SLOT_POOL_BYTES=64MiB` | 8 × 7 231 558 = 57.9 MB | 1 165 MB |
| 765 MB | `ZNIPPY_SLOT_POOL_BYTES=16MiB` | 2 × 7 231 558 = 14.5 MB | 1 130 MB |
| 5 MB | default | 1 × 7 231 558 = 7.2 MB | 61 MB |
No fat-path regression by construction: when neither the input nor the budget
binds, `plan` early-returns `PoolPlan::default_for(num_workers)` — the same
`8 × 200 MiB` the consts gave — as the 2.4 GB row shows.
**Still open after this.** The slot pool is no longer the RAM floor, but it was
not the only one: the 765 MB run still peaks at ~1.13 GB with a 14 MB pool. The
residual is the per-worker codec state — 29 gatling workers × a zstd level-19
context (`CONFIG.compression_level = 19`) plus `zstd_output_buffer_size` scratch.
On a small pod `max_core_in_flight` scales down with the core count so this
shrinks with it, but if a backup pod is still tight, that is the next term to
bound, not the slot pool.
**Original plan follows.**
**Status (original):** proposed (not yet implemented)
**Driver:** Njord issue #23 "Bygg en stabil IT-arkitektur 2.0". The self-built
backup/restore (`tillsynia-admin`) seals its staging tree with `znippy` as a lib
(`znippy_compress::compress_dir`). Ops flagged that the backup "has previously
needed a lot of memory because the snapshot is handled in memory". The Njord
side is now fully streaming (S3 blobs → disk, DB COPY chunked, uploads
multipart), so the remaining RAM floor is **znippy's slot pool**.
## The problem
`znippy-compress/src/slot_packer.rs` hardcodes:
```rust
const SLOT_SIZE: usize = 200 * 1024 * 1024; // 200 MiB
const NUM_SLOTS: usize = 8;
```
`Magazine::new` (`znippy-common/src/slotpool.rs`) allocates the whole pool
eagerly, up front, regardless of input size:
```rust
.map(|_| vec![0u8; slot_size].into_boxed_slice()) // × num_slots
```
So **every** `compress_dir` call reserves `8 × 200 MiB = 1.6 GiB` of heap before
touching a single file, plus per-worker zstd/OpenZL scratch buffers. On a backup
pod with a modest memory limit (e.g. 512 MiB–1 GiB request) this OOM-kills or
forces a large, wasteful reservation. The pool size is **independent of the data
volume** — a 5 MiB staging dir still reserves 1.6 GiB.
Crucially, `StrategicConfig` (`znippy-common/src/common_config.rs`) already
computes `max_mem_allowed` and `min_free_memory_ratio` from detected RAM — but
`SLOT_SIZE`/`NUM_SLOTS` **ignore it**. The knobs exist; the slot pool just
doesn't read them.
## Goals
1. Cap slot-pool RAM to a caller/config-chosen budget (target: a backup can run
comfortably under ~256 MiB total).
2. Offer a **disk-backed** slot mode ("use tmp disk rather than RAM") for
memory-constrained hosts, trading throughput for a tiny RSS.
3. No throughput regression on the fat/bench path (Loki, Odin) when memory is
plentiful — the current 1.6 GiB behaviour stays the default there.
4. Preserve the zero-copy skip/incompressible path and the two-pass
(big/small) design.
## Proposed changes (in priority order)
### 1. Make the pool config-driven (cheap, high value)
- Add `slot_size: usize` and `num_slots: usize` to `StrategicConfig`, derived
from `max_mem_allowed` with a hard floor/ceiling. Default keeps today's
`200 MiB × 8` when RAM is plentiful; under a cgroup limit it shrinks so
`slot_size × num_slots ≤ max_mem_allowed × (1 − min_free_memory_ratio)`.
- `slot_packer.rs` reads `CONFIG.slot_size` / `CONFIG.num_slots` instead of the
two `const`s. `slice_size` math is unchanged (`slot_size / num_workers`).
- Env overrides for ops: `ZNIPPY_SLOT_SIZE` (bytes/`MiB` suffix) and
`ZNIPPY_NUM_SLOTS`, parsed in `common_config`. This lets the Njord backup pod
set e.g. `ZNIPPY_SLOT_SIZE=32MiB ZNIPPY_NUM_SLOTS=2` → 64 MiB pool.
- **Note:** the big/small partition threshold is `slice_size = slot_size /
num_workers`. Shrinking `slot_size` pushes more files into the *big* pass
(sequential chunked reads — already bounded memory), which is exactly the
behaviour we want when RAM is tight. Verify the big pass's per-worker chunk
buffer is itself `≤ slice_size`.
### 2. Disk-backed slots (tmpfs/tmp-disk mode)
- New `SlotBacking` enum in `slotpool.rs`: `Heap` (today) | `MmapFile(dir)`.
- `MmapFile` mode `mmap`s each slot from a file under a caller-supplied dir
(`ZNIPPY_SLOT_DIR`, default `$TMPDIR`). Pages are backed by the page cache and
can be evicted under pressure → RSS stays low, spillover goes to disk instead
of OOM. Use `memmap2` (already in the constellation? confirm; else add).
- `compress_dir` gains an optional `SlotBacking` param (or reads
`ZNIPPY_SLOT_DIR`); Njord's backup points it at the pod's ephemeral
`/tmp` (or a mounted scratch PVC), matching the constellation build-target
doctrine (heavy work on fast scratch disk, not RAM).
- Keep zero-copy: pwrite-from-slot still works from an mmap'd region.
### 3. Report the reservation
- `CompressionReport` (or a `log::info!`) states the chosen `slot_size ×
num_slots` and backing mode, so a backup run records its actual memory
envelope — feeds the issue #23 requirement to "document actual RPO/RTO and
resource use after each drill".
## Tests (constellation testable-code law)
- `slotpool`: a unit test asserting `Magazine::new(n, s, w)` allocates exactly
`n` slots of `s` bytes and that a config-shrunk pool reports the smaller size
(red-when-broken: hardcode reintroduced → size wrong).
- `common_config`: env-override parse test (`ZNIPPY_SLOT_SIZE=32MiB` →
`slot_size == 32 * 1024 * 1024`), and the `max_mem_allowed` clamp
(pool ≤ budget).
- `slot_packer`: seal a small dir with a tiny pool (`num_slots=2,
slot_size=1MiB`) and assert the archive round-trips (decompress → identical
tree) — proves correctness is independent of pool size. Extend to
`MmapFile` backing once (2) lands.
- Bench guard: a HEAVY bench row before/after on Loki confirming no throughput
regression at the default (plentiful-RAM) pool size.
## Out of scope
- Changing the Arrow-IPC index format or the on-disk archive layout — this is
purely about the *transient* compress-time buffers.
## Njord side — already done (no znippy dependency)
`tillsynia-admin` now streams everywhere it previously buffered:
`ObjStore::get_to_file` / `put_file` / `get_stream` (multipart up, chunked
down), restore DB load via chunked `COPY FROM STDIN`, and the admin-server
snapshot up/download endpoints stream instead of `read`-into-`Vec`. That removes
the per-object and per-bundle RAM spikes; this plan removes the last floor.