# dwarves — agent-found problems
Real problems found while working this repo that were **not** actioned in the change that
found them, plus the measurements that rule other explanations out. Sister file to the
constellation's other `<repo>_agents_problems.md`.
Reporting shape follows the workspace LAW: **col-1** a 1–3 word unique subject, **col-2**
status (`done` · `new` · `as dåligt` · `as bra`), **col-3** the technical detail.
---
## 2026-08-18 — `dwarves-server` OOM-killed the 62 GB `dwarf` box
Eleven kills over 2026-08-16/18, the last four at 22:43 · 23:23 · 00:20 · 01:15, each at
`anon-rss` 49–53 GB / `total-vm` 61–73 GB / `pgtables` 106–127 MB, followed by 152 ×
`systemd-journald: Under memory pressure, flushing caches` and the host dying at
02:17:31 without completing a journal write.
| subject | status | detail |
|---|---|---|
| mimalloc purge | done | **Term 1 of the OOM, and the one this repo owns.** `mimalloc = "0.1"` compiles the **v3** C sources (`libmimalloc-sys/build.rs` picks `v3` unless the `v2` cargo feature is set; nothing in the constellation sets it), and v3 raised the `purge_delay` default from v2's 10 ms to **1000 ms**. At 1000 ms freed memory is never handed back: MEASURED on `dwarf`, a 2 GB heap of ~1 KB `String`s dropped three times over stays at 2006 MB RSS forever; `MIMALLOC_PURGE_DELAY=10` → 156 MB, `=0` → 7 MB. RSS therefore stopped meaning "in use" and became the running MAXIMUM of every transient the process ever built. Fixed in `crates/dwarves/src/alloc.rs`, called from `server::serve_all`; guarded by `crates/dwarves/tests/mimalloc_transient_peak.rs`. On a REAL embedded-nornir republish (`deep_scan = true` over dwarves' own Cargo.lock closure — 357 dep crates, 13 344 files, 12 023 symbol scans) the tuning cuts PEAK RSS **12–20 %** — 3.08 → 2.71 GB and 3.38 → 2.72 GB across two independent pairs — which is the number that matters, because production died AT its peak (`anon-rss` == systemd's `memory peak` at all eleven kills). The settled floor moved only 2.53 → 2.46 GB: see the next row for why. |
| deepscan batch | new | `nornir/src/deepscan.rs:409` `const BATCH: usize = 1_000_000;`, used at `:424` as `present.chunks(BATCH)`. The comment above it says *"Per batch of crates (bounds peak memory)"* — with 2 265 crates in the nordisk closure there is exactly **one** batch, so it bounds nothing. Three full copies of the corpus are live at the flush: `CollectSink.files: Vec<CrateFile>` (`:310`, 66 478 files decompressed into `content: String`), `WarehouseSink.pending: Vec<SymbolScan>` (`:326`, the 60 467 scans the log prints), and the Arrow arrays `ingest_symbol_scans_async` clones every string into (`nornir/src/warehouse/iceberg.rs:3432`, accumulated in `tables` at `:3451` before one `ingest_multi` at `:3717`) — plus `sort_groups` (`:1138`) making a sorted copy while the original is alive. **This is the transient the allocator fix now returns; shrinking it is still worth doing.** A real batch size (crates per flush, sized off available memory) belongs here. |
| working-set cache | new | **Term 2 of the OOM, and it is NOT reachable from this repo.** MEASURED: after one heavy republish the settled floor is ~2.5 GB and is *insensitive* to the allocator setting (2.53 GB as-shipped vs 2.46 GB tuned) — so that floor is LIVE memory, and at production scale it is very likely the larger of the two terms. `nornir/src/warehouse/working_set.rs:88` `PREVIEW_CACHE` and `:90` `ARROW_CACHE` are process-global `Mutex<Vec<…>>` statics warmed at the END of every republish (`nornir/src/monitor.rs:1557` → `warm_hot_tables`, `working_set.rs:139`). `WORKING_SET_CAP = 64` (`:49`) bounds **entry count, not bytes**, and there are only 18 hot keys — so the cap never fires and all 18 decodes stay resident indefinitely. `HOT_PREVIEW_LIMIT = 5_000_000` (`:85`) rows × 15 `HOT_LENS_TABLES` (`:57`) of `TablePreview.rows: Vec<Vec<String>>` (one heap `String` per cell) is tens of GB on its own, and `arrow_cached` → `scan_arrow` → `load_and_read_all` (`nornir/src/warehouse/iceberg.rs:1529`) has **no limit at all** while `HOT_ARROW_TABLES` (`:80`) includes `embeddings` — the entire 768-float vector table for the workspace, held in an `Arc` in a static. `preview_cached` (`:107`) also builds the new decode *before* evicting the old, so each warm peaks at 2×. Cap it on BYTES. `nornir` deliberately has no crate dep on dwarves (`nornir/Cargo.toml:671-680` — that edge was removed so the forge could compile nornir IN), so nothing in this repo can bound it; it has to land in nornir. |
| cache key identity | new | `nornir/src/warehouse/working_set.rs:97,107,119` — the cache keys are `(table, snapshot_id, limit)` with **no warehouse/workspace identity**. With several served workspaces they evict each other every sweep, so the same tables are re-decoded from scratch on every sweep of every workspace. |
| deepscan idempotency | new | `nornir/src/monitor.rs:1133` calls `deepscan::deep_scan(&dirs, wh)` with no SHA/skip ledger, unlike the knowledge scan at `:504-590` which prints `knowledge unchanged, skip scan`. So the whole dep closure is re-appended under a fresh snapshot id on every republish and `symbol_facts`/`call_edges`/`feature_gate_facts`/`test_inventory` grow monotonically on disk — which is also what feeds the preview cache above. |
| embed pre-read | new | `nornir/src/monitor.rs:1806-1828` builds `inputs: Vec<RepoInput>` over **all** present members in one gatling pool, each holding full file contents (`nornir/src/vector/store.rs:505`). The doc comment at `:1786-1789` claims *"each member's sources are DROPPED after its commit, so peak resident text no longer grows with member count"* — the `into_iter()` drain at `:1840` is correct, but the peak already happened at `:1806`, before the first member is embedded. The doc and the code disagree; the code is what runs. |
| scip result vector | new | `nornir/src/monitor.rs:847-856` — the SCIP fan-out is bounded (`scip_fanout`, `clamp(cores/8, 2..=8)`) but `resolved: Vec<Result<ScipScan>>` is not: every resolved member's full occurrence row set (`ScipRow`, 6 `String` fields) is held until the serial pass drains it at `:861`. Inside a worker, `ingest_index_file` (`nornir/src/knowledge/scip.rs:171`) holds the raw `index.scip` bytes **and** the parsed protobuf `Index` **and** the emitted rows simultaneously. |
| republish locals | new | `nornir/src/monitor.rs:593` `symbol_scans` and `:607` `git_heats` are locals of `republish_with` that are never scoped or dropped, so every rescanned member's symbol tables stay resident across the SCIP phase, the index build, the (measured 1021 s) embed, deep-scan and arch-generate — they are only needed by the append loop at `:626-647`. |
| arch phase | new | `nornir/src/monitor.rs:1243` `arch_computed` holds every member's full `ArchGraph` **and** its rendered SVG `String` at once; `:1267` `all_access` accumulates every member's `fn → table` edges for the whole workspace before one append at `:1310`. |
| unit blast radius | new | The unit `dwarves-server --install` renders declares no `MemoryAccounting=`/`MemoryHigh=`/`MemoryMax=` and no `OOMPolicy=`, so a runaway in the fat process is a **global** OOM: the kernel picked victims box-wide (`tailscaled` and `tmux: server` both invoked the OOM killer, journald flushed caches 152 times) and the host died rather than the unit. `Restart=on-failure` with no `StartLimitBurst` then re-launched it straight back into the same climb — restart counter reached 11. NOT actioned because `skidbladnir::service::ServiceSpec` (`Skidbladnir/src/service.rs:350`) has no field for cgroup resource limits; adding one is a skidbladnir change and every nordisk service wants it, so it should land there once rather than as a dwarves-local drop-in. |
| health full scan | new | `crates/dwarves/src/server.rs:262-265` — `Dwarf.Health` calls `store::read_runs(&self.warehouse)` and uses it only for `.len()`, i.e. it parses the **entire** run history off disk on every liveness poll. `recent_runs` (`:245-256`) has the same shape: read everything, `row_to_pb` everything, `reverse()`, then `truncate(cap)`. Harmless today (the live warehouse has no `brokkr/` at all) and it is a read, not a retention — but it is O(history) per poll for a UI that polls, and it will be the next thing to hurt once pipelines actually run. |
| action cache growth | new | `crates/dwarves/src/cache.rs:128,149,168` — the action cache is an append-only JSONL: `insert` appends a line even when the `input_id` already exists (later lines win on reload), and nothing ever compacts it. In-memory size is bounded by DISTINCT input ids, so this is not the OOM, but the file grows without limit and `ActionCache::load` — called once per dispatched run — parses all of it. |
| proxy dial churn | new | `crates/dwarves/src/warehouse_proxy.rs`, the `proxy_impl!` macro: every proxied RPC does `self.0.channel().await?`, i.e. a **fresh** `Channel::connect` (new TCP + HTTP/2 handshake) to the embedded nornir per call, rather than one channel cloned per handler. MEASURED not to leak — 4 500 proxied `Warehouse.Tables` calls left RSS plateaued at ~80 MB and the fd count constant at 13 — so this is a latency/TIME_WAIT problem, not a memory one, and it was left alone as out of scope. |
| cron store rows | new | `crates/dwarves/src/server_cron.rs:78-89` — `CronStore::save` inserts and never deletes, so a pipeline that is de-registered keeps its `cron_last` row forever. Bounded by the number of pipelines ever registered; noted for completeness, not a risk. |
| assemble cell red | new | `crates/dwarves/tests/assemble_container_cell.rs` does not compile on clean `main` (`E0063: missing field 'params' in initializer of 'BuildArgs'`, twice) — verified by stashing all local work and building the target at `cf72fdf`, and still red at `fb5a752`. So a bare `cargo test -p dwarves` cannot even finish linking on `main` today. PRE-EXISTING and unrelated to the allocator work; not touched, because "fixing" a red I did not cause by editing someone's in-flight surface is how a real failure gets papered over. |
| silenced funnel | done elsewhere | `crates/dwarves/tests/silenced_tests_guard.rs` was RED at `cf72fdf`: `funnel_persist_live.rs` carries `#![cfg(feature = "funnel-action")]`, which is not in the `default` closure and had no declared arm, so 2 test fns compiled to an empty binary and printed `running 0 tests ... ok`. Verified pre-existing by moving the new allocator cell off disk (identical single finding over 46 targets instead of 47). **Fixed upstream while this work was in flight** — `cc8528a` ("testmatrix arms: release_registry_robot + funnel_persist_live rescued") declares both arms, and the guard is green again on `fb5a752`. Recorded because the finding was real and the record of who fixed it is worth having. |
| variant roster | new | `crates/dwarves/tests/variant_drive.rs:911` `the_drive_roster_is_declared_not_invented` is RED on clean `main` (both at `cf72fdf` and at `fb5a752`): `.nornir/build-variants.toml` declares SIX atoms (`dwarves-server`, `dwarves`, `dwarves-cli`, `dwarves-ui`, `dwarves-ui-gpu`, `dwarves-solo`) and the drive roster covers five. PRE-EXISTING and byte-identical with the new cell removed. The guard is right — an uncovered atom is a silent cap — and the missing entry is a declaration only its author can make honestly. |
| floors unresolvable | new | `main` at `fb5a752` **does not resolve at all** in the documented side-by-side checkout layout, so no `cargo` command runs on it: `46b99a5` ("floors raised to the published API") set `facett-pipeline = "0.1.16"` (`crates/dwarves/Cargo.toml:635`) and `nornir-service = "0.1.7"` (`crates/dwarves/Cargo.toml:700`, `crates/nornir-funnel-embed/Cargo.toml:54`, `crates/nornir-warehouse-embed/Cargo.toml:45`), while the sibling checkouts are at facett `0.1.15` and edda's nornir-service `0.1.6`. A path dep whose sibling is BELOW the declared floor is a resolution error, not a warning: `error: failed to select a version for the requirement 'facett-pipeline = "^0.1.16"'`. PRE-EXISTING and reproduced on `fb5a752` with zero local changes. NOT touched: those floors were raised deliberately to match crates.io, so lowering them would revert someone's decision, and the honest fix is to bump the siblings. The allocator change was verified twice around it — fully green at `cf72fdf` (the tip when the work started) and again on top of `fb5a752` with ONLY those three floors relaxed locally and never committed. |
| host disk full | new | Not a code bug, but it belongs next to this incident: `/` on `dwarf` is **100 % full** (1.9 T, 12 MB free when hit on 2026-08-18). `cargo` target dirs across the constellation account for essentially all of it — `nornir/target` 273 G, `holger/target` 210 G, `korp/target` 174 G, `knut/target` 102 G, `dwarves/target` 100 G, and eight more. A release build of the fat server failed with `No space left on device (os error 28)` mid-link. Worth connecting to the incident: the last thing before the host died at 02:17:31 was journald flushing caches 152 times and then **not completing a journal write** — an out-of-space `/var/log/journal` produces exactly that, so the memory ratchet may not have been the only thing squeezing the box that night. Reclaimed only `dwarves/target/debug/incremental` (16 G, pure regenerable cache) to finish the verification; **no other repo's target tree was touched** — that is the operator's call, not an agent's. |
| constellation-wide | new | The same v3 `purge_delay` default applies to **every** nordisk binary that installs mimalloc: `nornir/src/bin/nornir-server.rs:18` and `nornir/src/bin/nornir.rs:11` both declare `#[global_allocator] mimalloc::MiMalloc` with no option tuning. A standalone `nornir-server` running the same monitor sweep will ratchet exactly the same way. The fix wants lifting into a shared leaf (edda) so one call site serves the whole constellation instead of being re-implemented per repo. |
### What was ruled out, and how
The hypotheses that looked most likely up front were all eliminated by measurement, not
by reading:
* **sindri / the watch loop.** `/home/dwarves/.dwarves/warehouse/` contains exactly one
file, `cron.redb` — there is **no `brokkr/` directory**, so `store::append_history` has
never been called and not one pipeline run has ever executed. The in-memory
`Registry` (`crates/dwarves/src/server.rs:52,864`) starts empty at every boot and is
only ever populated by the `RegisterPipeline` RPC, so `server_cron::sweep_cron` is
skipped on every tick (`server.rs:895`, `if !regs.is_empty()`). The "watch loop" churn
in recent `git log` (`a6e0892`, `e80629a`) is in `dwarves-gui` — the UI — not in the
server. **Also proves the `--real`/`cargo build` and run-history hypotheses dead**: no
run means no `RealRunner`, no `InMemBuildInfoSink`, no history rows, no bench rows.
* **gRPC / grpc-web subscribers.** 8 500 grpc-web `Dwarf.Health` calls and 4 500 proxied
`nornir.v1.Warehouse/Tables` calls, each on a **fresh connection**, against a live fat
server: RSS 68 → 80 MB and then flat, thread count constant at 14, open fds constant
at 13. No per-connection retention.
* **Unbounded channels.** There are none — `grep -rn "unbounded_channel\|unbounded("`
over `crates/` returns nothing; the only channels are `std::sync::mpsc` shutdown
signals in test/embedding helpers.
* **Overlapping sweeps.** `nornir/src/server_grpc.rs:5932` awaits its `spawn_blocking`
inside the tick loop and sets `MissedTickBehavior::Skip`, and `ctx.republish_lock`
(`:6117`) is a second belt, so a 20-minute sweep cannot stack behind a 60 s tick.
`nornir/src/monitor.rs:2415` `spawn_poll_loop` is dead code — nothing wires it.
* **Captured child output.** rust-analyzer's SCIP run pipes **stderr only**
(`nornir/src/knowledge/scip.rs:265`, `.output()`); the index itself goes to a temp
*file* via `--output` (`:250`) and is read back at `:171`. The enormous payload never
passes through a pipe.
* **What the reproduction does and does not show.** The allocator term is demonstrated
three ways and all three agree: a 2 GB standalone probe (0 % returned as shipped,
100 % returned at `purge_delay=10`), the regression cell in-process (443 MB peak,
**0 KiB** returned as shipped vs ~300 MB returned tuned), and a real embedded-nornir
deep-scan republish (peak 3.08 → 2.71 GB and 3.38 → 2.72 GB). What is NOT
demonstrated is that the allocator term accounts for the whole 50 GB — the settled
floor after a heavy sweep is the same either way, which is what points at the
`working-set cache` row above as the larger term. Said plainly, because a fix that
measurably lowers the ceiling is worth having and is still not the same thing as a
fix for everything.