<!-- nornir:class: design -->
# ldeflate — parallel DEFLATE/zlib compressor
**Opened 2026-08-03.** The write side of `linflate`. Before this crate there was **no
DEFLATE compression code anywhere in the znippy-zoomies workspace** — every codec here is
decompress-only, and the only compressor in the tree is the vendored C zstd in
`zstd-sys-rs`. Independently confirmed: every `GzEncoder` / `compress_to_vec` in the
workspace is inside `#[cfg(test)]`.
Companion: [`gatling-tricks.md`](gatling-tricks.md), which records the engine patterns
independently of this consumer.
---
## What it is, in three sentences
The entropy coder is **`zlib-rs` via `flate2`, reused unchanged** — pure Rust, no C, no
cmake, no `build.rs`, no build-time network. What is new is the **scheduling**: a two-queue
planner that coalesces small objects into batches and splits large ones at
`Z_FULL_FLUSH` boundaries, all on one no-barrier `gatling` pass. Output is ordinary
RFC-1950 zlib and is verified by handing it to the real `git` binary.
## Machine and provenance of every number below
| | |
|---|---|
| host | oden — AMD Ryzen Threadripper PRO 3975WX, **32 cores**, `nproc` 32, 1 thread/core |
| date | **2026-08-03** |
| build | `--release`, `CARGO_TARGET_DIR=/home/rickard/scratch/cargo/zoomies-deflate` |
| harness | `ldeflate/examples/measure.rs`, min-of-N **round-robin** across configurations |
| **box state** | **SHARED.** Load swung between **3 and 84** during the session; `korp-server` alone held ~19 cores for part of it. Every table below states the load it was taken at. |
> ### ⚠ Read this before quoting a throughput number
>
> The **headline scaling and sustained numbers (Findings 4 and 5) were taken on a QUIET box
> (load 3–5)**: 28.30x at 32 threads, 30.79 of 32 cores held. **A 7-run re-measurement on
> 2026-08-03 (Finding 4b) showed that 32-thread speedup is a property of machine occupancy,
> not of this code — it spanned 10.38x–23.20x across those runs.** Quote the CPU-seconds
> figure (1.025x–1.054x vs serial at 32 threads, reproducible to 2.8%) when you want a claim
> about the crate; quote a speedup only together with the cores it actually held.
>
> The **threshold and window sweeps (Findings 2 and 3) were taken while the box was loaded
> (load 73–84)**. Those sweeps are decided on **CPU-seconds per GiB**, not wall time,
> precisely because CPU-seconds consumed by our own process is far less sensitive to other
> tenants than wall clock is; the wall MB/s columns in those tables are indicative only and
> visibly noisy. Every table states the load it was taken at, and every configuration in a
> sweep was sampled **round-robin** so a load spike hit all rows equally.
>
> Tables report *cores actually held* (`cpu-seconds / wall`) beside every speedup, so
> efficiency can always be read against the machine we got rather than the one we asked for.
## Which workload each measurement models
Every arm models a **BATCH write**: `repack()`, a loose-object write burst, a client-side
pack write, or a cold-tier re-compress. `P-004` rejects per-*request* fan-out on a loaded
serving tier (1000 concurrent clones × 32 workers = 32 000 threads); it does not reject
fan-out inside a batch operation, and that distinction is the whole reason this crate has a
place.
**No arm models serving a clone, deliberately.** On a clone the right answer is pack-copy
(`P-001`) — measured 25.30 ms against 28 609 ms for per-object zlib over 74 523 objects,
and *smaller output too*, because git's delta encoder already extracted the redundancy.
**No compressor beats not compressing.** An arm that implicitly modelled a clone would be
measuring a path `P-001` deletes.
---
## The corpus, stated up front
A uniform-size corpus would invalidate this entire design, so here is the real one:
**48 917 objects, 1666.8 MiB**, read with `git cat-file --batch-all-objects --batch` from
eight real repositories on this box (nornir, dwarves, korp, znippy, skade, edda, holger,
znippy-zoomies).
| statistic | value |
|---|---|
| mean | 35 743 B |
| p10 / **p50** | 125 / **1 014 B** |
| p90 / p99 | 36 325 / 578 292 B |
| p99.9 / max | 4 203 350 / 55 420 968 B |
| objects under 1 KiB | **50.4%** |
| objects under 4 KiB | 69.8% |
| objects under 64 KiB | **93.8%** |
| **BYTES** in objects under 64 KiB | **14.8%** |
That last row is the tension the whole design sits on. **93.8% of the objects hold 14.8% of
the bytes.** Per-object fixed costs are set by the first number; throughput is set by the
second. A design tuned on either alone gets it wrong.
The **large / LFS-shaped** corpus is 4 × 64 MiB objects, in two flavours: text-like
(compressible) and high-entropy (standing in for the video/image/archive content that is
*why* a file is large enough to be in LFS at all).
A **uniform 1 MiB × 256 control** exists in the harness (`measure -- uniform`) purely so
that shape is on the record as *not* git-like and never quoted as a git result.
---
## Finding 1 — the per-object cost is zlib's state reset, not the scheduler
This is the load-bearing measurement and it contradicts the premise the design started from.
**Dispatch cost (32 workers, empty units):**
| units | claim 1 | claim 4 | claim 16 |
|---|---|---|---|
| 10 000 | 290.6 ns | — | 288.3 ns |
| 100 000 | 66.9 ns | — | 31.7 ns |
| 1 000 000 | **62.2 ns** | — | **27.7 ns** |
**Work cost, one deflate at level 6, one core, min-of-9:**
| size | ns/object | MB/s |
|---|---|---|
| 64 B | 4 966 | 12.3 |
| 256 B | 4 956 | 49.3 |
| 1 KiB | 5 350 | 182.5 |
| 4 KiB | 5 993 | 651.8 |
| 16 KiB | 8 863 | 1 763.0 |
| 64 KiB | 18 863 | 3 313.4 |
| 256 KiB | 64 917 | 3 851.0 |
**A 64-byte object costs the same as a 256-byte one.** There is a hard ~5 µs floor
independent of payload size, and it is **80× larger than the gatling dispatch**. So
"dispatching a task to deflate 200 bytes costs more than the deflate" is *false here* — the
deflate costs vastly more, and the cost is a fixed one.
Isolating it (standalone probe against `flate2` + `zlib-rs` directly):
| operation | ns |
|---|---|
| `Compress::new(L6)` — build a compressor | **4 082** |
| `reset()` alone, any level (L0/L1/L6/L9) | **1 567 / 1 570 / 1 567 / 1 564** |
| `compress()` of 64 B with no reset | **95** |
| `reset` + `compress` 64 B at L0 / L1 / L6 / L9 | 1 706 / 1 902 / 4 391 / 4 502 |
`reset()` is **level-independent** — the signature of a fixed table clear, sized by zlib's
`memLevel`. The actual coding of a small object is 95 ns. Everything else is setup.
**Consequence for the design:** the coalescing queue's win is **reusing one compressor
across many objects** (`reset`, 1567 ns) instead of building one per object
(`Compress::new`, 4082 ns) — about **2.5 µs saved per object**. Amortising the 62 ns
gatling claim was never the point.
**Consequence for gunnar.** `crates/gunnar-pack/src/write.rs:121` builds a fresh
`ZlibEncoder` per object. `.nornir/performance-design.md` Measurement 4 found reusing it
bought "no difference beyond noise" and warned against selling it as a throughput fix.
**That measurement stands, and its corpus explains it**: mean 19.2 KB, where 2.5 µs is ~1%
of the work. Real git objects have a **median of 1014 B**, where 2.5 µs is a *quarter* of
the per-object cost. The reuse win scales inversely with object size, and it was measured on
a corpus 19× larger than git's median.
## Finding 2 — a falsified hypothesis: shrinking the LZ77 window does not help
If the floor is a hash-table clear, shrink the table. `window_bits` was added to test it.
**Wrong, measured:**
Cost of one deflate at L6 by window, ns/object:
| size | wb9 | wb10 | wb11 | wb12 | wb13 | wb14 | wb15 |
|---|---|---|---|---|---|---|---|
| 64 B | 5 014 | 5 005 | 5 001 | 4 998 | 4 987 | 4 968 | **5 001** |
| 256 B | 5 033 | 5 020 | 5 016 | 5 006 | 4 966 | 5 041 | 5 047 |
| 1 KiB | 8 687 | 5 430 | 5 376 | 5 386 | 5 390 | 5 355 | 5 374 |
| 4 KiB | 22 085 | 10 955 | 7 849 | 6 756 | 6 173 | 6 144 | 6 192 |
| 16 KiB | 77 325 | 30 412 | 18 389 | 12 627 | 9 977 | 8 875 | 8 962 |
The floor is **identical** at every window. zlib's hash table is sized by `memLevel`, which
is independent of `windowBits` and which `flate2` does not expose — this knob cannot reach
it. And a small window is actively harmful on real data (45 876 real git objects < 64 KiB):
| window | cpu-s/GiB | vs wb15 | ratio | **output size vs wb15** |
|---|---|---|---|---|
| 9 | 17.02 | 0.87× | 2.033 | **+37.1%** |
| 12 | 14.11 | 1.05× | 2.598 | +7.3% |
| 14 | 14.45 | 1.02× | 2.761 | +1.0% |
| **15** | 14.78 | 1.00× | **2.788** | — |
**Default stays 15.** The API is kept — it is cheap, correct, and honest — but the claim
attached to it is now a negative result. Reducing the floor would need `memLevel`, i.e. an
upstream change to `flate2` or dropping to `zlib-rs` directly. Not done here; recorded as a
concrete future item worth roughly **1.5 µs per object**.
## Finding 3 — the coalescing threshold: monotone, no knee, worth 15%
Round-robin sweep, min-of-7, decided on CPU-seconds (the load-robust metric).
**On the sub-64 KiB majority — 45 877 real git objects, 246.7 MiB:**
| coalesce_below | units | cpu-s / GiB | vs one-unit-per-object |
|---|---|---|---|
| off | 45 877 | 16.94 | 1.00× |
| 256 B | 43 299 | 16.98 | 1.00× |
| 1 KiB | 32 370 | 16.44 | 1.03× |
| 4 KiB | 20 340 | 15.86 | 1.07× |
| 16 KiB | 9 887 | 15.32 | 1.11× |
| **64 KiB** | 482 | **14.78** | **1.15×** |
| 256 KiB | 482 | 14.78 | 1.15× |
**On the whole repository — 48 917 objects, 1666.8 MiB:** off 14.09 → 256 KiB 13.77,
i.e. **1.02×**. Only 2%, because 85% of the bytes live in objects too big to coalesce.
**How the threshold was found, and what it is.** It is **not a knee** — the curve is
monotone and saturates only when the threshold covers every object. That follows directly
from Finding 1: if the win is per-object compressor reuse, more objects per batch is always
better, and the only limit is load balance (fewer units → less for LPT to work with) and
the RAM bound on live output. **So the answer is "coalesce everything you reasonably can",
and the default is `coalesce_below = 64 KiB`, which covers 93.8% of real git objects.**
**Is ljar's batch-of-64 our number? No — and it is not ljar's either.** ljar's 64
(`ljar/src/reader.rs:20`) is a **RAM bound** ("only one batch of decompressed output is
live at a time"), not dispatch amortisation, and ljar never coalesces at all: it dispatches
one gatling unit per entry at claim 1 (`parallel.rs:32`, `batch.rs:109`). Compression's cost
profile moves the answer a long way — our batch closes at **512 KiB of source or 256
objects**, which on the measured corpus is ~95 objects per batch, and the binding constraint
is keeping ~15 units per worker for LPT rather than any per-batch count.
**Claim size measured irrelevant.** 1 / 4 / 16 landed within **0.3%** (16.94 / 16.89 /
16.89 cpu-s per GiB). The house note that claim 1 is a 2× regression comes from `stree.rs`
on far cheaper units; on a per-object deflate it does not transfer. Default 4, because it
costs nothing.
**The ratio is byte-identical across every configuration above (2.788 / 3.107).** Coalescing
changes not one output byte — asserted as a test, not just observed.
## Finding 4 — scaling: the reproducible number is ~5% CPU overhead, NOT a speedup
> **⚠ Re-measured 2026-08-03 (later session), n = 7 independent runs.** The wall-clock
> speedup at 32 threads is **not a property of this code** and does not reproduce: across
> seven runs it spanned **10.38x to 23.20x — a 123.5% spread** — driven entirely by how many
> cores the box would give us. What *does* reproduce, to within 2.8% over the same seven
> runs, is the **CPU-seconds the schedule costs versus serial: 1.025x–1.054x at 32 threads.**
> Read [the re-measurement](#finding-4b--what-reproduces-and-what-does-not) below before
> quoting any number from the table that follows.
Git corpus, 1670.9 MiB, level 6, min-of-5 round-robin. **Box QUIET: load 5 → 3.**
| threads | MB/s | speedup | efficiency | cores actually held |
|---|---|---|---|---|
| serial (no gatling) | 80.1 | 1.00x | 100% | 1.00 |
| 1 | 79.5 | 0.99x | 99% | 1.00 |
| 2 | 155.5 | 1.94x | 97% | 2.00 |
| 4 | 299.0 | 3.73x | 93% | 3.99 |
| 8 | 594.9 | 7.43x | 93% | 7.94 |
| 16 | 1 170.7 | 14.62x | 91% | 15.73 |
| 24 | 1 720.6 | 21.49x | 90% | 23.40 |
| **32** | **2 266.6** | **28.30x** | **88%** | **30.79** |
**28.30x on 32 hardware threads at 88% efficiency, holding 30.79 of 32 cores.** The curve
is still rising at 32 and does not plateau — the efficiency loss from 4 threads (93%) to 32
(88%) is 5 percentage points across a factor of eight.
**Compare `P-004`**, which measured per-object deflate at **20.3x / 63% at 32 threads** and
*worse* at 64. Both numbers are real and they do not contradict each other; the difference
is the scheduler. `P-004` used a **static stride partition across `std::thread`** — a fixed
carve-up with no work stealing. On a corpus whose objects span 125 B to 55 MB, a static
partition is as slow as its unluckiest slice. This crate uses LPT (heaviest-claimed-first)
over a shared atomic cursor, so a worker that draws cheap objects simply draws more of them.
**8 points of efficiency and 1.4x of speedup came from the schedule, not the codec.**
The 64-thread regression `P-004` found is not re-tested here — this box has 32 hardware
threads and no SMT, so there is nothing above 32 to measure. **32 remains the ceiling on
this machine, and that ceiling is not reached.**
The gatling pool itself costs essentially nothing: 1 worker is 0.99x of a no-gatling serial
run.
> **Earlier in the same session, on a LOADED box (load 39–56, `korp-server` holding ~19
> cores), the same benchmark read 17.24x / 54% at 32 threads — while our process could only
> obtain 17.43 cores.** Both runs are in this document deliberately. Efficiency against
> *cores actually held* was 99% in the loaded run and 92% in the quiet one; the headline
> efficiency figure moved by 34 points purely from machine occupancy. **On a shared box,
> quote the load average with the number or do not quote the number.**
## Finding 4b — what reproduces and what does not
**Re-measured 2026-08-03**, `f736163`, corpus **49 497 objects / 1684.7 MiB** (the corpus is
read live from eight working repos, so it had grown 1.2% from the 48 928 objects of the run
above — the two tables are therefore *not* on identical bytes). **n = 7 independent runs**,
each itself min-of-5 round-robin, so 35 samples per thread count. Every run under
`sudo -n systemd-run --scope -p MemoryMax=32G`, run strictly serially.
**The box was never quiet, and could not be made quiet.** Other tenants held **7.2 to 32.0
of 32 cores** across the batch (measured per run as a 10 s `/proc/stat` window immediately
before it); load average ranged **7.4 to 166**. Two long-running jobs from other lanes
(`errcode_raise_sites` at ~394%, `osmium extract` at ~122%) never finished, and a third lane
saturated all 32 cores partway through. That variation is what makes the table below
informative rather than merely noisy.
| threads | speedup min–max | spread | cores actually held | eff vs cores HELD | **cpu-s vs serial** |
|---|---|---|---|---|---|
| 1 | 0.99–1.00x | 1.0% | 1.00 | 99–100% | **1.001–1.014x** |
| 2 | 1.96–2.01x | 2.6% | 2.00 | 98.0–100.5% | **0.997–1.018x** |
| 4 | 3.87–3.94x | 1.8% | 3.98–3.99 | 97.2–99.0% | **1.009–1.029x** |
| 8 | 7.55–7.74x | 2.5% | 7.88–7.94 | 95.7–97.8% | **1.022–1.045x** |
| 16 | 13.76–15.19x | 10.4% | 14.32–15.69 | 95.4–96.9% | **1.032–1.048x** |
| 24 | 14.65–21.94x | 49.8% | 15.25–23.09 | 95.0–97.3% | **1.027–1.052x** |
| 32 | 10.38–23.20x | **123.5%** | 10.86–24.33 | 94.9–97.6% | **1.025–1.054x** |
**Read the first column against the last.** The same seven runs of the same binary over the
same corpus produced 32-thread speedups from 10.38x to 23.20x, and CPU costs from 1.025x to
1.054x. The speedup column measures *the box*; the CPU column measures *the code*. Speedups
at 1–8 threads reproduce fine (≤2.5%) only because ≤8 cores were always free; the spread
grows with thread count precisely as the request starts to exceed what was available.
**The engine takes every core it can get.** Cores held tracks `32 − other tenants` almost
exactly — e.g. run 1 measured 7.2 cores of other load and held 24.33 (32 − 7.2 = 24.8), and
efficiency *against cores actually held* stayed in a **94.9–97.6%** band at 32 threads across
every occupancy from 10.86 to 24.33 cores held. Under-subscription is not a scheduling bug;
there was nothing else to take.
### Does 28.30x survive?
**Not reproduced, and not refuted — the condition never recurred.** The best occupancy in
seven runs was **24.33 of 32 cores held**; the original run's 30.79 was never approached, so
this session could not put 28.30x to the test directly.
What can be said is that 28.30x is **consistent with, and slightly conservative against**,
these measurements. The original table implies its own CPU cost: 22.70 cpu-s at 32 threads
against a 20.86 cpu-s serial baseline = **1.088x**, a *higher* overhead than any of the seven
re-runs (1.025–1.054x). Carrying the re-measured overhead forward:
- at 30.79 cores held (the original run's occupancy) it predicts **29.2–30.0x** — above the
28.30x actually recorded;
- at a genuinely idle 32 cores it predicts **30.4–31.2x**.
So the headline number is not too *high*. **What was wrong is the framing**: 28.30x was
presented as a property of the crate when it is a property of that run's machine occupancy,
and it is the one number here that cannot be reproduced on a shared box. The durable claim
is the CPU one.
**Unresolved — stated as unresolved.** The original run's 1.088x overhead sits outside the
1.025–1.054x band of all seven re-runs. Two explanations fit and **this session could not
separate them**: (a) memory-bandwidth saturation that only bites above ~25 cores held, which
no re-run reached, or (b) residual contention in the original "quiet" run, which held 30.79
of 32 and so had ~1.2 cores of something else on it. **No memory-bandwidth counter was read
— option (a) is inferred from the shape of the curve, not counted.** Settling it needs a
genuinely idle box, which oden was not on 2026-08-03.
## Finding 5 — sustained load: median window fully saturated, with a per-pass dip
30-second run, 39 passes over the full 1.67 GiB corpus, 100 ms per-core `/proc/stat`
sampler on a `gatling::background::Job`. **Box quiet (load 3).**
```
39 passes over 30.5s -> 2136.2 MB/s SUSTAINED
cores >50% busy: mean 29.0 / 32, worst window 1, best window 32
summed utilisation: p50 32.0 / 32 cores, p5 12.0 (the dip floor)
per-core mean over the run: min 90%, max 98% (spread 8 pp)
OUR PROCESS held 28.90 of 32 cores over the whole 30.5 s
```
**What sustains and what does not.** The **median 100 ms window is fully saturated
(32.0 / 32 cores)**, and over the whole run **no core sat out** — the per-core mean ranges
only 90% to 98%. Throughput held at 2136 MB/s across 39 consecutive passes with no decay,
which is the "sustained, not a burst" property.
**But there is a real dip, and it should not be papered over.** p5 is 12.0 cores and the
worst single window is 1. That is the **serial tail of each pass**: `compress_objects`
materialises a `Vec<Vec<u8>>`, so between passes one thread reassembles the results and then
frees ~500 MB of output. At 782 ms per pass the dip is brief, which is why the median is
still 32.0 — but it is there, and it is exactly the cost `compress_objects_to_writer` (the
31+1 streaming path) exists to remove: that path never materialises the whole output and has
no per-pass teardown. **A real `repack()` should use the streaming entry point, and this dip
is the measurement that says why.**
## Finding 6 — the LFS answer: skip compression, the win is not what you think
Large-object corpus, 4 × 64 MiB.
**Compressible (text-like) — segment sweep, 32 workers:**
| segment | MB/s | ratio | output vs unsplit |
|---|---|---|---|
| none (unsplit) | 400.1 | 63.003 | 1.0000 |
| 64 KiB | 4 859.2 | 49.340 | +27.7% |
| 256 KiB | **5 128.2** | 58.328 | +8.0% |
| **1 MiB** | 4 126.0 | 61.629 | **+2.2%** |
| 4 MiB | 3 905.9 | 62.646 | +0.6% |
| 16 MiB | 1 390.6 | 62.930 | +0.1% |
Splitting is worth **10.3× wall throughput for a 2.2% ratio cost** at 1 MiB segments. (The
16 MiB row collapses because 4 × 64 MiB / 16 MiB = 16 units on 32 cores — too few to fill
the box, a direct demonstration of Trick 1's floor.) **Defaults set from this table:
`split_above = 4 MiB`, `segment = 1 MiB`** — `segment` must stay ≤ `split_above` or
splitting never fires at all.
**Incompressible (video/archive-shaped) — the actual LFS case:**
| segment | MB/s | **ratio** |
|---|---|---|
| none | 48.6 | **1.000** |
| 64 KiB | 593.1 | 1.000 |
| 1 MiB | 555.0 | 1.000 |
**Level-6 DEFLATE achieves a compression ratio of 1.000 — literally nothing — at 48.2 MB/s
on one core.** Splitting makes it 11× faster at achieving nothing.
**Is the split-path win compression, hashing, or I/O? Measured separately, one 64 MiB
object, one core:**
| content | deflate L6 | SHA-1 | hash ÷ deflate |
|---|---|---|---|
| compressible | 398.4 MB/s | 2 021.1 MB/s | **5.1×** |
| **incompressible** | **48.2 MB/s** | **2 027.3 MB/s** | **42.1×** |
> ### The design conclusion, stated plainly
>
> **For large already-compressed objects the right behaviour is to skip compression
> entirely.** Not to parallelise it. A deflate that returns ratio 1.000 while running 42×
> slower than the hash over the same bytes is pure waste, and making it 11× faster is
> making waste cheaper, not making progress. The incompressible-detection gate (magic-byte
> probes plus a caller hint, being fixed in znippy) must run **before** this compressor
> sees the content.
>
> **And "parallelise the hash instead" is not available for a single object.** SHA-1 and
> SHA-256 are inherently serial over one stream; there is no split. What you can do is
> parallelise *across* objects, or move to a tree hash (BLAKE3). At 2 GB/s serial you
> probably do not need to.
>
> **So for LFS the parallel win is neither compression nor hashing — it is I/O**, which is
> the 31+1 shape (Trick 6): one core keeping the device saturated while the rest compute.
> Someone will otherwise benchmark this compressor on LFS-shaped data, find it does
> nothing, and conclude the design failed when it is working exactly as intended.
**Does one threshold serve both distributions?** Yes, and that is a real result arguing the
design is right. Git objects are 93.8% under 64 KiB, so they take the coalescing queue and
the split queue never fires. LFS objects are megabytes to gigabytes, so they take the split
queue and the coalescing queue never fires. The **same `Config`** serves both without a
caller doing anything, because the classification is per-object, not per-call. The one thing
a caller must configure is `split_above` if their "large" is much smaller than 4 MiB.
## Where the within-stream split axis does *not* apply
**Git streams contain no flush boundaries.** Independently measured over 87 709 real git
streams: 7 candidate `00 00 FF FF` markers, and **0 genuine ones** after validating each by
`inflate_segment` against a byte-exact suffix. So `linflate::deflate_scan` /
`ljar::chunk::split_chunk` are **inert on git data** — the *read*-side split has no purchase
on packs.
The asymmetry worth noting: this crate is on the **write** side, so it *creates* the
boundaries rather than hunting for them, and it can split any input it likes. But that does
not make the axis useful for git — git objects are too small to be worth splitting, which
is the same conclusion by a different route. **The split axis is for the large-object /
LFS / cold-tier corpus, and the coalescing axis is for git.**
---
## Guarantees, and how each is proven
| guarantee | proof |
|---|---|
| **Stock `git` reads it** | `tests/stock_git.rs`: a repository whose *entire* loose object store this crate wrote passes `git fsck --full --strict` (and fsck is asserted to have *seen* all 411 objects, not walked an empty store), every object is byte-compared through `git cat-file`, and a real pack-v2 assembled from our streams passes **`git index-pack --strict`** plus `git verify-pack`. |
| — and the check can fail | Two negative controls: a corrupted loose object makes `fsck` complain; a corrupted pack payload makes `index-pack --strict` exit non-zero. Without these the acceptance tests would prove nothing. |
| **Pure Rust, no C, no cmake** | `tests/no_c_toolchain.rs` scans the resolved `cargo tree -e normal,build` for `cc`, `libz-sys`, `libz-ng-sys`, `*-sys`, `cmake`, `bindgen`. Tree is exactly `ldeflate → flate2 → zlib-rs` plus `anyhow`/`rotaryengine`. |
| — and *that* check can fail | The same scan is run against `zstd-sys-rs` in this workspace, which vendors ~25 `.c` files and uses `cc` + `bindgen`, and must fire. If it ever goes quiet the guard is blind. |
| **Deterministic** | The work plan is a pure function of `(object sizes, Config)` and never reads the core count. `parallel_equals_serial_at_every_worker_count` asserts byte-identical output at 1/2/3/8/32 workers against a no-gatling serial oracle — and `segment_size_actually_changes_the_bytes` proves that assertion is not vacuous by showing a different config *does* change the bytes. |
| **rayon-free (LAW 3)** | The workspace guard covers `ldeflate` automatically (it reads the `members` line). Proven red by injecting `rayon::current_num_threads()` into `ldeflate/src/lib.rs` — the guard failed and named the file and line. |
**Ratio cost of the whole design, on the real corpus: zero.** Coalescing and claim size
change no output bytes at all. Splitting costs 2.2% at the default segment size, and only on
objects ≥ 4 MiB.
## Open items, honestly labelled
1. **The scaling ceiling is not established.** Re-run Finding 4 on a quiet box. Today's 32-
thread row measures oden's occupancy, not ldeflate.
2. **`memLevel` is unreachable through `flate2`** and is worth ~1.5 µs per object on
git-median data (Finding 1/2). Either an upstream `flate2` API or a direct `zlib-rs`
binding.
3. **`compress_objects_to_writer` (31+1) is correctness-tested but not perf-measured.**
Its benefit is peak-RSS and device overlap, which needs a real NVMe target to show.
4. **No delta encoding.** This compresses whole objects. `P-009` notes gitoxide has no
delta encoder either; that is a much larger piece of work and is what makes pack-copy so
hard to beat.
---
## Formats: what was added after the DEFLATE core, and what it cost
`ldeflate` now writes **zlib, gzip, ZIP and JAR**. Only the first needed an
engine; the rest are framing, and that is the result worth recording — the
two-queue planner, the coalescing, the split at `Z_FULL_FLUSH` and the
no-barrier pass are **reused verbatim** by every format (LAW 5). Total new
scheduling code across gzip, ZIP and JAR: **none**.
| format | what was actually written | reference tool that judges it |
|---|---|---|
| zlib (RFC 1950) | the engine | `git index-pack --strict`, `git fsck --strict` |
| gzip (RFC 1952) | 10-byte header, CRC-32, ISIZE | `gzip -t`, `zcat`, `pigz -d` |
| ZIP | local headers, central directory, EOCD | `unzip -t/-p`, Python `zipfile` |
| JAR | ZIP + manifest-first ordering | Python `zipfile` (no JVM on this host) |
### gzip has two shapes and they are not interchangeable
**One member, split internally** (`gzip_compress`) — the body is cut at
full-flush boundaries and reassembled under one header and one *combined*
CRC-32. This is what `pigz` emits. Every tool reads it; **no tool can decompress
it in parallel**, because the boundaries are invisible from outside.
**Many members, concatenated** (`gzip_compress_members`) — RFC 1952 §2.2 makes a
gzip file a series of members, and each is independently decodable. This is the
shape **`lgz`, the workspace's own parallel gzip reader, can fan out over**. It
closes a loop: the repo's bench harness currently builds its multi-member corpus
with `split -b 8M | xargs -P N gzip -6 | cat`, and this is that pipeline
in-process, without `split`, `xargs`, `gzip` or a temp directory.
The member shape costs ratio, because independent members cannot share an LZ77
window — and that cost is asserted as a **test**, because if members were *not*
larger than one stream they would be sharing a window and would not be
independent. The cost proves the property.
### The strongest evidence in the whole crate is four bytes
`tests/stock_gzip.rs::our_header_is_structurally_what_stock_gzip_writes`
compares **our combined CRC-32 — folded out of per-worker checksums through
`crc32_combine` — byte-for-byte against the CRC stock `gzip` computed serially
over identical input.** Two independent implementations, same four bytes.
That is worth more than any round-trip in this repository. A round-trip through
our own trailer checks our arithmetic against our arithmetic; `flate2` shares
`zlib-rs` with us, so it would agree with a shared misreading of the spec. Only
an outside implementation computing the same checksum a different way is
evidence about the wire.
### ZIP: entries parallelise first, splitting second
A real JAR is hundreds of tiny `.class` files plus a few large ones — the same
skew as a git repository (median 1014 B, 93.8% under 64 KiB). So entries go
straight into the existing planner: small ones coalesce into batches, and an
entry only gets split internally when it is over `split_above`. **Both axes
apply to one archive**, which is why a single 200 MB entry cannot serialise a
worker. Container assembly stays serial — a few dozen bytes of header against
kilobytes of payload, memcpy-class.
**Entries that do not compress are STORED (method 0).** That is Finding 6
applied per entry, not a nicety: level-6 DEFLATE over already-compressed content
measured ratio 1.000 while running 42x slower than a hash over the same bytes.
Python's `zipfile` confirms method 0 on the incompressible entry and method 8 on
the compressible one from *outside*, so the decision is not merely asserted
internally.
**Refusals, not silent truncation.** No ZIP64: over 4 GiB or over 65 535 entries
is an error naming what was exceeded. A writer that ignores this wraps a 32-bit
field and produces an archive no tool can diagnose. Entry names that are
absolute, drive-lettered, or contain a `..` component are refused too — this
crate will not be the thing that produces a zipslip payload.
### Why Python's `zipfile` and not only `unzip`
`unzip` will fall back to scanning local file headers when the central directory
disagrees with them. Python's `zipfile` indexes **through the central directory**
and nothing else. A writer that computes central-directory offsets wrongly
therefore **passes `unzip` and fails Python** — which is precisely the bug a
single-reader test would ship. Two readers that disagree about where to look is
worth more than two readers that agree.
`jar` and `java` are **not installed on this host**, so the JAR arm verifies the
structural contract `java.util.jar.JarInputStream` depends on (manifest first,
correctly terminated, `Main-Class` present) rather than claiming a JVM read it.
**A real `jar tf` / `JarInputStream` arm is an open item**, not a completed one.
---
## The OOM of 2026-08-03, and what it changed
At 12:32 UTC a test built from this crate reached **497 GB anon RSS on a 499 GB
box**. The OOM killer fired and took down a shared tmux service and three
unrelated sessions.
**Cause:** two tests crossed a *degenerate chunk parameter* with a *large input* —
`len = 3_000_000` against `member = 1` (three million one-byte gzip members), and
`len = 1_000_000` against `seg = 1`. `cargo test` ran them concurrently.
**The test was the trigger; the bug was that the library said yes.** Both
parameters are amplifying: the caller passes two small integers and the crate
allocates a structure proportional to their quotient. That is now refused —
`limits::MAX_PIECES = 2^18`, an `Err` naming the request and the ceiling, never a
clamp (a silent clamp would change the segment plan, therefore the output bytes,
therefore the determinism guarantee, *without the caller learning*).
Three rules came out of it and they hold for every format here:
1. **Degenerate parameters are tested against SMALL inputs only.** `m = 1` and
`seg = 1` are exactly the boundaries worth testing — at 8 KiB, where a million
pieces cannot arise. Large inputs get realistic chunk sizes. The reasoning is
in a comment at both sites so nobody re-crosses them.
2. **Every test and bench runs under `(ulimit -v 25165824)`.** No exceptions.
3. **The library refuses absurd requests rather than attempting them** — piece
counts, ZIP entry counts, entry sizes, archive size.
Current whole-suite cost: **67 tests, 23 s, 2.0 GB peak RSS** under the 24 GB cap.
---
## Costing the `memLevel` floor (open item 2, now priced)
Finding 1 measured a per-object floor of ~4.4 us at level 6, of which **1567 ns is
`reset()` alone, level-independent** — a fixed table clear sized by zlib's
`memLevel`. Finding 2 established that `windowBits` cannot reach it.
**What a direct `zlib-rs` binding would buy.** `memLevel` defaults to 8, giving a
hash table of `2^(memLevel+7) = 32768` entries x 2 bytes = **64 KiB cleared per
stream**. At `memLevel = 4` that is 4 KiB — a 16x smaller clear. Extrapolating
from the measured 1567 ns as essentially all memset, the floor would fall to
roughly **100-200 ns**, saving **~1.4 us per object**.
**What that is worth, and to whom.** On the measured git corpus (48 928 objects)
it is ~68 ms per full pass against a 737 ms parallel pass — **~9%**. On the
sub-64 KiB majority it is proportionally larger. On the LFS corpus it is zero.
So it is a small-object optimisation and only worth doing if small objects are
the workload.
**What it costs.** `flate2` exposes no `memLevel`, so this means either an
upstream `flate2` API addition (cheap to write, unbounded to land) or dropping to
`zlib-rs`'s own `DeflateStream` API and giving up `flate2` as the compatibility
layer. The second is maybe a day and adds a direct dependency on a
lower-stability API surface.
**Recommendation: not yet.** 9% on one corpus does not justify leaving `flate2`,
and a smaller `memLevel` also costs compression ratio (fewer hash chains, worse
match finding) which has not been measured. Revisit if a small-object workload
becomes the dominant one, and **measure the ratio cost first** — the window-bits
result (Finding 2) is the cautionary tale: it looked like a free win and cost 37%
in output size.
---
# 2026-08-04 — the git bake-off, and what it says about `gunnar gc`
**Opened because the ask was "run the ldeflate vs zlib-rs bench and make ldeflate
faster".** The first half of that ask needed correcting before it could be
answered, and the correction is the most useful thing in this section.
## The framing correction: `ldeflate` IS `zlib-rs`
`ldeflate/Cargo.toml` pins `flate2 = { default-features = false, features =
["zlib-rs"] }`. The entropy coder on **both sides** of any "ldeflate vs zlib-rs"
comparison is therefore the same code. What differs is scheduling. So:
> **Single-stream, `ldeflate` cannot beat `zlib-rs`. It can only lose by its own
> overhead.** A benchmark that reports otherwise is measuring a difference in
> *what work was done*, not in how fast it was done.
That is not hypothetical — it happened twice in this session's own harness and
both times the guard caught it:
1. **`Config::default()` skips already-compressed objects.** The first control
run had `ldeflate` at **0.924× the CPU of raw zlib-rs** and looked like a win.
It was not: the planner had *stored* rather than deflated the objects that
probe as compressed, emitting **+0.56% bytes** for that CPU. Real behaviour,
real trade, but not a scheduling result. Every arm below therefore runs
`compress_everything()`.
2. **`compress_everything()` alone is still not enough** — it leaves splitting
on, and cutting an object at `Z_FULL_FLUSH` costs ratio. Arm 2's
byte-equality assert fired on the difference. Every arm below also runs
`without_splitting()`.
**A benchmark of a compressor that does not assert output-byte equality is not a
benchmark.** Both of the above are invisible in a wall-clock table.
## Corpus — the 19 `workspace_nordisk` repositories
Read with `git cat-file --batch-all-objects --batch` and **frozen to one file on
md1**, because a corpus read live from working repositories grows between runs —
Finding 4b above records a 1.2% drift that made two of its own tables
incomparable.
| statistic | value |
|---|---|
| objects | **76 616** |
| total | **2251.5 MiB** |
| mean | 30 814 B |
| p10 / **p50** | 126 / **1130 B** |
| p90 / p99 | 33 266 / 509 749 B |
| p99.9 / max | 3 278 452 / 55 420 968 B |
| under 1 KiB | 47.9% |
| under 64 KiB | **94.2%** |
| **BYTES** under 64 KiB | **17.1%** |
Same shape as the eight-repo corpus above: 94.2% of the objects hold 17.1% of the
bytes.
## Finding 7 — a third `Compress::new` was being paid where nothing needed one
Finding 1 established that `Compress::new` costs **4082 ns** and `reset()` costs
**1567 ns**, so reusing an encoder saves ~2.5 µs per object. `Encoder` did that
*within* one unit of work. It could not do it *across* units, because every
scheduler here hands its map closure a `Fn`, not an `FnMut` — there is nowhere to
keep a `&mut Encoder`. So both schedulers constructed inside the closure:
| site | what it built | how often |
|---|---|---|
| `plan::compress_objects` | **two** encoders, zlib-wrapped and bare | **per unit** |
| `stream::compress_objects_to_writer` | one encoder | **per OBJECT** |
Two things are wrong in that table.
**The bare encoder was dead code on the git path.** `run_unit`'s inner closure
picks the wrapped encoder when the container is zlib-wrapped and the bare one
otherwise — but the "wrapped" pool is built with `zlib_wrapped =
container.encoder_is_zlib_wrapped()`, so on the gzip/raw containers it *already
is* the bare encoder. The second encoder was only ever reachable from
`Unit::Segment`. Every `Batch` and `Whole` unit paid 4082 ns to construct a
`Compress` it never touched.
**The streaming path paid construction per object.** On a git-median 1130 B
object the whole deflate is about 5 µs, of which 4.1 µs was `Compress::new`. That
is the path this document's own Finding 5 names as what "a real `repack()` should
use".
`EncoderPool` (in `unit.rs`) fixes both: a `Mutex<Vec<Encoder>>` a worker pops
from and returns to on guard drop. Not a `thread_local!`, because that would make
the crate non-reentrant — a `compress_objects` called from inside another one's
map closure on the same thread would hit a `RefCell` double-borrow panic. Both
checkouts are lazy, so the git path constructs the bare encoder **zero** times.
### The reuse guard was blind first, and that is the point
The first version of the guard asserted on `parked()`, the free-list length.
**It passed with reuse deliberately broken.** A `checkout` that pops an encoder,
throws it away and builds a fresh one leaves the free-list length identical at
every observation point — the state round-trip LAW 2 warns about, built while
looking for exactly that failure mode.
The fix is an **applied output**: `EncoderPool::constructed()` counts
`Compress::new` calls. Re-broken the same way, the guard now fires with
`left: 2, right: 1`. `parked()` is kept, with a doc comment saying not to assert
reuse on it.
### What the pool actually bought — measured, and it is small
**A/B'd round-robin** (base, opt, base, opt…) so a load spike hits both arms
equally, two binaries built from an identical harness, `min` per binary.
**The noise floor is stated first, because it decides everything below.** The
harness's `raw zlib-rs` row is *literally the same source in both binaries*, so
its opt/base ratio is pure noise. It read **0.9717× CPU** on the whole-repo
corpus and **0.9843×** on the small-object one. Anything inside about ±3% is
therefore nothing.
**Whole repository — 76 616 objects, 2251.5 MiB, n = 3 rounds, load 5–17:**
| row | opt/base CPU | verdict |
|---|---|---|
| *raw zlib-rs (identical code — the noise floor)* | *0.9717×* | *—* |
| `compress_objects` workers=1 | 0.9894× | **inside the floor** |
| crossover, workers 1/2/4/8/16/32 | 0.98–1.02× | **inside the floor** |
| `compress_objects_to_writer` 1/8/31 | 0.98–1.00× | **inside the floor** |
**No measurable effect at all.** The arithmetic says why, and it is the trap
Finding 1 already named: this corpus is **bytes**-dominated while the cost the
pool removes is **count**-dominated. 5 200 units × 8.2 µs is 43 ms of a 33 000 ms
pass (**0.13%**); even the per-object streaming path is 76 616 × 4.08 µs = 313 ms
of 33 500 ms (**0.93%**). The mean object here costs 430 µs, so 4 µs is
invisible. **The first A/B of this change was designed without doing that
arithmetic and it measured nothing, as it had to.**
**Small objects only — `LDEFLATE_MAX_OBJ=65536`, 72 137 objects (94.2% of them),
386.1 MiB (17.1% of the bytes), n = 5 rounds, quiet box (load 4–5):**
| row | opt/base CPU | **net of the 0.9843× floor** |
|---|---|---|
| *raw zlib-rs (the floor)* | *0.9843×* | *—* |
| `compress_objects` workers=1 | 0.9553× | **≈2.9% faster** |
| `compress_objects_to_writer` workers=1 | 0.9355× | **≈5.0% faster** |
| `compress_objects_to_writer` workers=8 | 0.9231× | **≈6.2% faster** |
| `compress_objects_to_writer` workers=31 | 1.0241× | inside a 9.5% spread — nothing |
> **The honest headline: the encoder pool is worth ~3–6% CPU on the small-object
> majority and is INVISIBLE on a whole-repository pass.** It is kept because it
> is also strictly less work — an unconditional construction of an encoder the
> git path never touches is not defensible at any measured size — and because
> the `/2` reserve that came with it makes the output byte-identical to raw
> `zlib-rs`. It is not a throughput fix and must not be sold as one.
## Finding 8 — the reserve was a systematic under-guess, and it changed the BYTES
`Encoder::drive` reserved `src.len() / 3` for the output. The measured ratio on
this corpus is **2.825** (2251.5 MiB in, 796.9 MiB out at level 6), i.e. output
is 0.354 × input — *above* a third. So the majority of the bytes forced a
`reserve` + memcpy inside the drive loop. Now `src.len() / 2`.
**The part worth recording is not the memcpy.** Running out of output space
mid-stream makes `deflate()` return and resume, and the resulting bitstream is
**not always identical** to the one an uninterrupted call produces. The diff arm
found **3 of 76 616 objects** (665 KB – 1.7 MB) where the two call sequences give
the same output *length* but different bytes near the end. Both are valid
DEFLATE, both inflate to the object — they are simply different encodings.
Consequences, stated plainly:
- The crate's determinism guarantee is **worker-count independence**, which still
holds exactly. It was never byte-stability across builds, and this is the proof
that the distinction is real.
- After the `/2` change, `compress_objects` output is **byte-identical to a plain
one-shot `zlib-rs` deflate over the whole 76 616-object corpus** (835 639 104 B
both sides). Before it, the two differed by 160 B. That equality is now the
harness's apples-to-apples check and it prints which of the two it got.
## Finding 9 — the parallel ceiling on this corpus is ~10×, and it is not the cores
Arm 2, `compress_everything().without_splitting()` so every row is the same
computation as the serial baseline. **Box QUIET for once: load 7–14.**
| workers | speedup vs serial zlib-rs | cpu vs serial | **cores actually held** |
|---|---|---|---|
| 1 | 1.00× | 1.003× | 1.00 |
| 2 | 2.01–2.08× | 0.96–1.00× | 1.99 |
| 4 | 3.84–3.93× | 1.02–1.05× | 3.98 |
| 8 | 7.75–8.15× | 0.98–1.02× | 7.80–7.91 |
| 16 | **9.91–10.87×** | 0.99–1.01× | **9.93–10.74** |
| 32 | **9.14–10.28×** | 1.04–1.07× | **9.66–10.68** |
**It scales cleanly to 8 and then stops, and 32 is WORSE than 16.** Efficiency is
97–99% up to 8 workers and collapses to ~31% at 32. The engine is not failing to
take cores it could get: at 16 workers it holds 10.7 of the 16 it asked for, on a
box whose load average was 9.9.
The cap is the **serial tail of `compress_objects`** — it materialises a
`Vec<Vec<u8>>`, so one thread reassembles 76 616 results and then frees ~800 MB.
Finding 5 already identified this dip from the other direction (p5 of 12.0 cores,
worst window 1) and predicted it; this is the same thing seen as a ceiling. At a
33 s serial baseline and a 3.1 s parallel wall, a ~1 s serial tail caps the
speedup at ~10-11×, which is what the table reads.
**So the honest headline is 10×, not 28×.** Finding 4's 28.30× was taken on a
corpus and a machine occupancy this session could not reproduce, and Finding 4b
already retracted its framing. This measurement, on a *quiet* box, says the
ceiling is a property of the API shape.
## Finding 10 — `gunnar gc` cannot be moved by any compressor
This is the result that decides whether any of the above matters to gunnar, and
it is a negative one.
`repack()` runs gitoxide's `Mode::PackCopyAndBaseObjects`, so an object whose
pack entry can be carried across is a memcpy and **never meets a compressor**
(`P-001`). `RepackOutcome` reports both counts and nothing had read them.
`gunnar/crates/gunnar-store/examples/gc_ratio.rs` reads them, on bare
repositories built by **real pushes** — not `git clone --bare`, which repacks on
the way in and would have measured an already-consolidated store.
**oden, 2026-08-04.** Two real server shapes:
| shape | repos | objects | copied from pack | **recompressed** | **share** |
|---|---|---|---|---|---|
| `transfer.unpackLimit=1`, 5 pushes | 6 | 39 115 | 36 165 | **2 950** | **7.54%** |
| `transfer.unpackLimit=100` (git's default), 40 small pushes | 3 | 13 478 | 8 284 | **5 194** | **38.54%** |
The first reproduced to **0.05 percentage points** across two runs at load 25 and
load 11.
**And gc holds 1.74 of 32 cores** (11.79 cpu-s over 6.764 s wall, unpinned, load
11), so it is not compressor-bound; it is bound by the object walk, delta
resolution, index and multi-pack-index writing, and SHA-1.
Costing the deflate directly on the loose-heavy shape, where it is largest: the
3 759 loose objects hold **61.3 MB**, which at this box's measured level-6 rate of
**70.4 MB/s** on real git objects is **0.87 cpu-s** — against a **4.66 cpu-s** gc.
> **≈19% of gc CPU is deflate, in the shape most favourable to a compressor, and
> 7.5% of objects in the shape a busy server actually has. A compressor that cost
> ZERO could not do better than that, and `ldeflate` is not free.**
**Recommendation: do NOT wire `ldeflate` into `gunnar gc`.** Beyond the Amdahl
ceiling, the deflate in `repack()` happens *inside* `gix_pack::data::output::
entry::iter_from_counts`, which also owns pack-copy, delta handling and entry
generation. Reaching it means replacing gitoxide's pack generator, not swapping a
compressor call. That is a large piece of work for ≤19% of a 1.74-core job.
**What could be worth doing instead** — stated as an untested hypothesis, not a
result: gc holds 1.74 of 32 cores, so its headroom is in the walk and the index,
not the codec. That is a `gix-pack` question.
## What was NOT determined
- **No per-phase attribution inside `repack()`.** The 19% figure is the deflate
cost of the loose objects computed from their byte volume and this box's
measured deflate rate; it is not a profile of gitoxide, and it excludes the
~1 435 recompressions that come from broken delta chains (whose byte volume was
not measured). It is a bound, not a breakdown.
- **The ~10× ceiling is attributed to the serial reassembly by elimination**, not
by a profiler: cores held tracks the request up to 8 and stops, on a box with
22 idle cores. A `perf` run would settle it and was not done.
- **`compress_objects_to_writer` is still the weak path.** It holds under 3 cores
at 31 workers (183 MB/s against `compress_objects`' 728 MB/s at 32). The
encoder pool removed 4082 ns per object from it and it is still bottlenecked
somewhere else — the ordered sink's single collector, most likely, since it
does the write, the bookkeeping and the drop of every output buffer on one
thread. Not chased.
- **No SIMD was attempted, deliberately.** gunnar's `P-022` recorded an AVX2
dispatch reverted with 4/4 pairs against, because `#[target_feature]` functions
cannot inline and the portable loop auto-vectorises after inlining. The same
trap applies here.
## The `lbzip2` one-engine violation, found in passing
`tests/rayon_free_law.rs` was **run, not inspected**. Three of its four tests
pass; `only_the_gatling_leaf_crate_owns_a_worker_pool` **fails on `origin/main`**:
```
lbzip2/src/limits.rs:162: private pool `thread::spawn` — hs.push(std::thread::spawn(move || {
```
Verified pre-existing by running the same test on a pristine `origin/main`
worktree: identical file, identical line. Not introduced here and not fixed here
— `lbzip2` is another lane's crate — but it is live and LAW 3 says it should not
be.
The rayon scan itself was proven **red on `ldeflate` specifically**: injecting
`rayon::current_num_threads()` into `ldeflate/src/lib.rs` made
`whole_workspace_is_rayon_free` fail and name the file and line.
## Finding 11 — LFS: the split axis DOES pay on real LFS, and the ratio cost is ~0
Raised as a caveat to be measured rather than assumed: real LFS payloads are
usually already-compressed media, so `SkipPolicy::resolve()` might store them
raw, in which case the split path would fire on *chunking* and Finding 6's 10.3×
would not transfer.
**Measured on the 98 REAL LFS objects on this box** — `.git/lfs/objects` of
nornir, korp, edda, facett, holger, skade — **247.1 MiB**, not a synthetic
corpus. oden 2026-08-04, level 6, min-of-3, load 61–67 (noisy, but every
comparison below is between rows of the *same* run).
**The caveat does not hold here:**
| question | answer |
|---|---|
| objects the magic probe calls already-compressed | 43 of 98 (43.9%) |
| **BYTES** the probe would store rather than deflate | **15.6 of 247.1 MiB — only 6.3%** |
| objects ≥ `split_above` (4 MiB), eligible to split | 19 |
The 43 objects that probe as compressed are the *small* ones; **93.7% of the
bytes are genuinely deflated, at ratio 1.705**. This is not the ratio-1.000
incompressible case Finding 6 warned about.
| configuration | wall s | MB/s | **ratio** | cores held |
|---|---|---|---|---|
| compress_everything, **split OFF**, 1 worker | 4.060 | 63.8 | 1.706 | 1.00 |
| compress_everything, **split OFF**, 32 workers | 1.001 | 258.7 | 1.706 | **4.04** |
| compress_everything, **split ON**, 32 workers | **0.438** | **591.3** | **1.705** | **9.86** |
| default (skip ON, split ON), 1 worker | 4.571 | 56.7 | 1.685 | 0.84 |
| default (skip ON, split ON), 32 workers | 0.517 | 501.4 | 1.685 | 7.66 |
> **The within-stream split is worth 2.29× wall at the SAME worker count
> (1.001 s → 0.438 s at 32 workers) for a 0.06% ratio cost (1.706 → 1.705), and
> 9.27× against a serial baseline.** The mechanism is visible in the last
> column: without splitting, 98 objects over 32 workers can only hold **4.04
> cores**, because the plan has 98 units of wildly uneven weight and the biggest
> one serialises the tail. Splitting turns 19 large objects into ~1 MiB segments
> and the same box holds **9.86**.
**This is the one place in gunnar where `ldeflate` has a clear, measured case**,
and it is a much better one than Finding 6's synthetic corpus suggested: the
ratio cost there was 2.2%, here it is **0.06%**, because real LFS objects are
large enough that a 1 MiB segment barely dents the LZ77 window's usefulness.
Two honest limits:
- **The `skip` default costs ratio on this corpus**: 1.685 with the probe on
against 1.706 with it off, i.e. **1.2% larger output** for 11% less wall. The
probe is deciding on magic bytes and calling 43 objects compressed that still
had 1.7× of redundancy in them as a set. Worth a look; not chased here.
- **Cores held stops at 9.86 of 32**, the same ~10 ceiling Finding 9 found on
the git corpus. 98 objects is a small unit count and the serial reassembly is
the same one. The split axis raised the ceiling from 4 to 10; it did not
remove it.