mikronova/modgunn

.nornir/cve-warehouse-ingest.md

<!-- nornir:class: reasoning -->

# CVE warehouse ingest — all major vuln DBs → Iceberg → modgunn scan

> Plan for getting **every major vulnerability database** into the shared nornir
> Iceberg warehouse (`cve_advisories`), airgap-compatible, so modgunn Engine A can
> render a Pass/Warn/Block verdict over the **full dependency closure** of any repo
> in any ecosystem — fully offline. Extends [`modgunn-master-plan.md`](modgunn-master-plan.md)
> §1 (Engine A), §7.3 (adopt-list) and §8 (open decision: advisory-DB sourcing in airgap).

---

## 0. Goal & the one decision that shapes everything

**Goal:** one warehouse-resident advisory corpus, refreshed by sealed-znippy
generations, that covers all the big feeds and all ecosystems holger serves
(cargo · pip · npm · maven · go · nuget · gem · **deb · rpm · apk** · conda ·
composer · helm · docker base images), matched precisely against real versions.

**The decision: store ranges, match at scan time — not expand-at-import.**

Today `cve_advisories` stores `affected_versions` as **exact versions** and
`scan::Advisory::matches` ([`scan.rs:127`](../src/scan.rs)) does **exact equality**
(empty list ⇒ *all* versions). To import a range-only OSV/NVD/GHSA entry you must
expand the range to exact versions ([`versions::expand_range`](../src/versions.rs)),
which needs the *full published version universe per package*. Over all DBs × all
packages that:
- explodes row counts,
- is lossy (a version published *after* import is silently "not affected"),
- couples the advisory feed to a registry-version catalog you don't have offline.

**Recommendation:** evolve to **range-native** matching — persist the OSV range
events (`introduced` / `fixed` / `last_affected`) and match with
[`versions::affected_by(eco, events, version)`](../src/versions.rs) against the
artifact's *actual* version at scan time. Expansion becomes lazy (only against the
versions you actually scan) or unnecessary. This is the single highest-leverage
change and **P0** below; everything else layers on it.

---

## 1. The sources (the "big CVE databases"), by format

| Source | Format | Ecosystems | modgunn status |
|---|---|---|---|
| **OSV.dev** aggregate (per-ecosystem `all.zip` exports) — federates GHSA, PyPA/PYSEC, RUSTSEC, Go vulndb, npm, Maven, NuGet, Hex, Pub, … | **OSV** (lingua franca) | nearly all package ecosystems | ✅ parser (`osv.rs`) |
| **GitHub Advisory DB (GHSA)** (`github/advisory-database` git, OSV files) | OSV | npm/pip/maven/nuget/rubygems/go/cargo/composer | ✅ reuse `osv.rs` (just another dir) |
| **CISA KEV** | JSON | (CVE-keyed, cross-eco) | ✅ `feeds::parse_kev_json` |
| **FIRST EPSS** | CSV | (CVE-keyed) | ✅ `feeds::parse_epss_csv` |
| **OpenSSF Malicious Packages** | OSV-ish JSON | npm/pip/… | ✅ `feeds::parse_malicious` |
| **NVD (NIST)** — CVE JSON 2.0, CVSS vectors + CPE | NVD JSON 2.0 | product-centric (CPE), weak pkg map | ❌ **new `nvd.rs`** |
| **Distro trackers** — Debian Security Tracker, Ubuntu USN/OVAL, Red Hat CSAF/OVAL, Alpine secdb, SUSE CSAF, GitLab Advisory DB | mixed (JSON/YAML/OVAL/CSAF) | **deb · rpm · apk** | ❌ prefer OSV.dev distro exports; native adapters for gaps |
| **VEX / CSAF** (not_affected suppression) | CSAF | cross-eco | 🚧 partial (`provenance.rs`) |

**Coverage strategy:** OSV.dev is the workhorse — its per-ecosystem exports already
fold GHSA/RUSTSEC/PYSEC/Go/npm/Maven/NuGet *and* several distro feeds (Debian,
Alpine, Ubuntu, Rocky, AlmaLinux, SUSE). Lead with it (P1) and only write native
adapters where OSV.dev's coverage is thin (NVD CVSS backfill = P3, distro gaps = P4).

---

## 2. Schema v2 — `cve_advisories` (additive, db_rev-versioned)

Iceberg appends are additive and `str_col` reads are drift-tolerant
([`warehouse.rs:98`](../src/warehouse.rs)), so we add columns without breaking the
existing reader. The new source-of-truth is `ranges`; `affected_versions` stays as
an optional pre-expanded cache for tiny/explicit-version advisories.

| Column | Type | Note |
|---|---|---|
| `id` | Utf8 | primary advisory id (prefer CVE, else GHSA/RUSTSEC) |
| `aliases` | Utf8 (CSV) | **dedup key** — all ids this advisory is known by (CVE↔GHSA↔RUSTSEC) |
| `ecosystem` | Utf8 | canonical modgunn label (`security::ecosystem_for`) |
| `package` | Utf8 | |
| `ranges` | Utf8 (JSON) | **new source of truth** — `[{introduced, introduced_exclusive, fixed, last_affected}]` per `versions::RangeEvent` (`IntroducedExclusive` = NVD `versionStartExcluding`, a *strictly greater than* lower bound) |
| `affected_versions` | Utf8 (CSV) | optional pre-expanded cache (""=all) — kept for back-compat |
| `severity` | Utf8 | info/low/medium/high/critical |
| `cvss_vector` | Utf8 | **new** — from NVD/OSV `severity[]` |
| `cwe` | Utf8 (CSV) | **new** |
| `epss` | Float64 (null) | exploit probability (FIRST) |
| `kev` | Boolean | confirmed exploited (CISA) |
| `withdrawn` | Boolean | **new** — OSV `withdrawn` ⇒ never matches |
| `source` | Utf8 | **new** — `osv|ghsa|nvd|debian|alpine|redhat|…` (provenance + per-source toggles) |
| `published_micros` / `modified_micros` | Int64 | **new** — cadence/anomaly (A20, A33) + audit |
| `db_rev` | Utf8 | import generation; latest rev wins on read |

**Code touch points:**
- `scan::Advisory` ([`scan.rs:83`](../src/scan.rs)) — add `ranges: Vec<RangeEvent>`,
  `aliases`, `cvss_vector`, `cwe`, `withdrawn`, `source`, timestamps.
- `scan::Advisory::matches` — range-aware: `withdrawn` ⇒ false; else
  `affected_by(ecosystem, &ranges, version)` OR the exact-list fallback.
- `warehouse::append_advisories` / `load_advisory_db` / `CVE_COLS`
  ([`warehouse.rs:156,202,513`](../src/warehouse.rs)) — round-trip the new columns.

---

## 3. Pipeline — internet-side build → airgap carry → secure-side import

Mirrors holger's airgap model (build a sealed `.znippy`, carry it, import it).

```
 INTERNET SIDE (agent / world connector)            — LUFTGAP —     SECURE SIDE (modgunn + skade)
 ┌───────────────────────────────────────────┐                   ┌────────────────────────────────┐
 │ 1 fetch  OSV.dev all.zip · GHSA git · NVD  │   2 seal          │ 4 verify sig (blake3)          │
 │          JSON · KEV · EPSS · distro feeds  │ ─► .znippy bundle ─►  5 append_advisories(db_rev,…) │
 │ 3 normalize → Vec<Advisory> (ranges) +     │   db_rev =        │    → cve_advisories generation │
 │   enrich(EPSS,KEV) + dedup-by-alias        │   date+srchash    │    load_advisory_db() = latest │
 └───────────────────────────────────────────┘                   └────────────────────────────────┘
```

1. **Fetch** (internet side): pull each feed snapshot to staging, pinned by date.
   Reuse holger-agent's `world` connector pattern; no live access on the secure side.
2. **Normalize**: each source → `Vec<Advisory>` with ranges. Reuse `osv.rs`/`feeds.rs`;
   new `nvd.rs` + distro adapters. **Dedup/merge by alias** — collapse the same
   vuln across CVE/GHSA/RUSTSEC into one row (alias union, range union, max
   severity, best CVSS, OR of kev). `feeds::enrich` folds EPSS/KEV by CVE id.
3. **Seal**: write the corpus as a **signed `.znippy`** stamped
   `db_rev = <ISO-date>-<sha of source set>` (master-plan §8). The carriable unit.
4. **Carry** across the gap (USB/diode), no return channel — file self-verifies.
5. **Import** (secure side): `modgunn warehouse import <bundle.znippy>` → verify →
   `Warehouse::append_advisories(db_rev, &advisories)` ([`warehouse.rs:156`](../src/warehouse.rs)).
   Append = new generation; `load_advisory_db` already returns latest `db_rev`.
   Idempotent re-import is safe (a re-imported rev just re-wins identically).

---

## 4. The scan path — "scan all dependencies"

1. **Dep closure.** nornir captures the resolved transitive closure per repo into
   `sbom_components`; modgunn reads it via `read_sbom_components(repo)`
   ([`warehouse.rs:263`](../src/warehouse.rs)). Multi-ecosystem resolvers are the
   master-plan §6 cutover (cargo-metadata exists; add pip/npm/maven/go/…). Holger's
   per-archive listing (`list_archive_files` → `security::parse_coords`) is the
   lighter path already wired in Tool #3.
2. **Join.** For a single repo / promotion gate: `load_advisory_db(Some(eco))` per
   ecosystem (cheap, partition-scoped). For a **fleet / whole-warehouse** scan:
   push the `sbom_components × cve_advisories` join down to skade/DataFusion
   (skade `sql` feature) — match `(ecosystem, package)` then `affected_by(version)`
   — instead of loading the whole DB into RAM.
3. **Verdict.** `OsvScanner` (range-aware) → per-component `Verdict` → `ScanReport`
   (Pass/Warn/Block) with KEV→Critical, EPSS≥τ→High, VEX `not_affected` suppression.
   Cache on `(blob_sha256, policy_version, db_rev)`.
4. **Surface.** holger Tool #3 / `holger-server scan`; nornir viz Security pane.
   `holger-server scan` swaps `demo_scanner()` for a warehouse-backed scanner when a
   catalog root is configured.

---

## 5. Phases (each independently shippable)

| Phase | Deliverable | Repos |
|---|---|---|
| **P0** ✅ | Schema v2 + **range-native matching** — DONE. `Advisory` gained `ranges`/`aliases`/`withdrawn`/`source`; `matches` is range-native via `affected_by`; `merge_by_alias` dedup; `cve_advisories` v2 columns (ranges JSON, aliases CSV, withdrawn, source) round-trip drift-tolerantly. 4 new tests (range match, withdrawn, alias dedup, warehouse v2 round-trip) + holger's 11 security tests green. | modgunn |
| **P1** ✅ | **OSV.dev full ingest** — `osv.rs` parses range-native (carries OSV ranges + `aliases` + `withdrawn`); `parse_and_merge` normalizes+dedups across feeds; `modgunn warehouse import`/`stats` CLI; **internet-side fetcher** `tools/fetch-advisories.sh` (OSV.dev per-ecosystem `all.zip` + NVD + KEV + EPSS → staging dir → the documented `import` invocation). *Remaining:* only the agent's sealed-znippy wrapping of the staging dir (carry step). | modgunn (+ agent fetch) |
| **P2** ✅ | Exploit + malicious enrich — DONE (offline). EPSS + KEV wired into `import` (`--epss`/`--kev` → `feeds::enrich`); OpenSSF Malicious-Packages wired via `--malicious <dir>` (`feeds::parse_malicious` → Critical, `source = openssf`, folded into the corpus with `merge_by_alias`). Smoke-tested (OSV + malicious in one import). | modgunn |
| **P3** 🔄 | **NVD** — *enrichment DONE.* `nvd.rs` parses CVE JSON 2.0 (CVSS v3.1/v3.0/v2 → severity + vector, CWE ids, description); `enrich_with_nvd` folds them onto advisories by id OR alias (sets CVSS vector + CWE, lifts severity), wired into `import` as `--nvd <dir>`. `Advisory`/warehouse gained `cvss_vector` + `cwe` (schema). 3 enrichment tests. **CPE backfill DONE:** `parse_nvd_advisories` emits standalone advisories from vulnerable application CPEs (product → package, `versionStart*`/`versionEnd*` → ranges — `versionStartExcluding` maps to the EXCLUSIVE `RangeEvent::IntroducedExclusive`, never the inclusive `Introduced`, or the boundary version false-positives; an upper bound with no lower bound is anchored `Introduced("0")`), tagged `ecosystem = "cpe"` (separate namespace, won't false-match OSV) + `source = "nvd"`; CLI `import --nvd-backfill <dir>`. CVEs absent from OSV now land with coords. | modgunn |
| **P4** 🔄 | **Distro trackers** (deb/rpm/apk) — *version grammars DONE:* `versions.rs` gained dpkg `deb_cmp` (epoch/upstream/revision, `~` ordering) + `rpm_cmp` (rpmvercmp: `~`/`^`, numeric-vs-alpha, leading-zeros), wired into `compare`/`affected_by` for `deb`/`rpm`/`apk`; `normalize_ecosystem` maps OSV distro labels (Debian/Ubuntu→deb, Alpine→apk, Red Hat/Rocky/Alma/SUSE→rpm) so OSV.dev's distro exports ingest with the right comparator. So deb/rpm advisories now range-match correctly via the existing OSV path — **no native adapter needed** for the OSV-covered distros. **apkvercmp DONE:** `apk_cmp` implements Alpine's grammar (numeric dot-parts, trailing letter, `_alpha`/`_beta`/`_pre`/`_rc`<release<`_p` suffix ranks, `-rN` revision) — `apk` no longer borrows dpkg. 4 tests. *Remaining:* native adapters only for the rare feeds OSV.dev doesn't carry. | modgunn |
| **P5** 🔄 | **Whole-warehouse scan** — *in-memory variant DONE (dep-neutral):* `scan_fleet` joins ALL `sbom_components` (every repo) × the latest `cve_advisories` in one sweep; `scan_sbom_delta(from)` is the **incremental CDC** path — only the components added since a snapshot, off skade's `table_delta`, returning the next watermark (poll `commit_seq`). CLI `warehouse scan [--since <snapshot>]`. Tolerant of an absent SBOM table. 2 tests + CLI smoke. Components assume `cargo` until nornir adds an ecosystem column (§6). *Variant B (pushdown) DONE:* `scan_fleet_sql` behind the opt-in `warehouse-sql` feature (`skade/sql`) runs the `name=package` equi-join in DataFusion, then range-matches + latest-rev in Rust on the narrowed candidates; CLI `warehouse scan --sql`. A test proves it returns exactly the in-memory result. The DataFusion tree is **isolated behind the feature** — the lean gate/build never pulls it. | modgunn (+ skade) |
| **P6** 🔄 | **VEX/CSAF suppression DONE:** `vex.rs` parses CSAF `known_not_affected` → suppressed CVE ids; `warehouse scan --vex <dir>` drops those findings (reports the suppressed count). 2 VEX tests. **Scorecard storage DONE:** `scorecard_signals` table + `append_scorecard`/`load_scorecard` (latest-per-subject). **SBOM-drift A30 DONE:** `sbom_drift(snap_a, snap_b)` reports `(entered, left)` `(name, version)` sets via the `snapshot_id` column. 2 more tests. | modgunn |

---

## 6. Cross-repo dependencies & risks

**Needs from skade:** confirm/enable the `sql` (DataFusion) feature for the join;
verify large-append throughput (OSV.dev all-ecosystems is hundreds of thousands of
records); consider **partitioning `cve_advisories` by ecosystem** so the gate read
`load_advisory_db(eco)` stays a single-partition scan. skade is **not checked out
here** (`../skade/skade`, path dep) — clone from `codeberg.org/nordisk` before P0.

**Needs from nornir (master-plan §6 cutover):** multi-ecosystem `components`
resolvers feeding `sbom_components`; retire the rust-only rustsec mirror
(`NORNIR_ADVISORY_DB`, [`security.rs:483`](../../nornir-src/src/security.rs)) in
favour of the warehouse table.

**modgunn surface to add:** `nvd.rs` + distro adapters; alias-merge in normalize;
CLI verbs `modgunn warehouse build-bundle | import | scan` (modgunn is lib-mostly
today — add the bin surface or expose via holger/nornir CLIs).

**Risks**
- Exact→range matching is a behaviour change — keep both columns through P0 and
  prove the existing `scan.rs` tests still pass before removing the exact path.
- NVD CPE↔package mapping is lossy — treat NVD as enrichment; OSV is the authority.
- Range-grammar coverage: `versions.rs` is semver (cargo/npm/go/nuget/composer)/PEP440-lite/
  maven-lite/gem/loose today;
  distro **epoch/revision** versions (deb `1:2.3-4`, rpm `1.2-3.el9`) need comparators (P4).
- Single-writer: modgunn import vs a live `nornir-server` holding the redb lock —
  schedule imports when nornir-server is down, or add the read-only copied-snapshot
  path (master-plan §8). Cadence: KEV/EPSS daily, OSV weekly, NVD weekly — each a
  new `db_rev`; old generations stay for time-travel/audit (A31/A34).