# feeds — guide
> **Class:** `guide` · audience: someone importing advisory feeds or building an `AdvisoryDb`
> from disk. The *what* is in [`feeds-design.md`](feeds-design.md); the *why* in
> [`feeds-reasoning.md`](feeds-reasoning.md).
## f0 — import a generation from the CLI
The secure-side operator path (needs `--features warehouse,scan`):
```sh
modgunn warehouse import --warehouse <root> --rev 2026-07-04-a \
--osv advisories/osv/ \
--malicious advisories/openssf/ \
--nvd advisories/nvd/ \
--epss advisories/epss.csv \
--kev advisories/kev.json \
--exploit-poc advisories/poc-in-github/ \
--metasploit advisories/metasploit_modules.json \
--nuclei advisories/nuclei-cves.json \
--exploitdb advisories/files_exploits.csv \
--crates-yanks advisories/crates/ \
--npm-deprecations advisories/npm/ \
--pypi-yanks advisories/pypi/ \
--go-retract advisories/gomod/
```
It parses every `*.json` under each dir, merges + dedups by alias, enriches with EPSS/KEV,
and appends the corpus as a new `db_rev` generation. The summary prints counts of KEV /
EPSS-scored / public-exploit / CVSS-vectored advisories, the registry yanks + npm
deprecations, and any skipped source files. The network fetch that fills those dirs runs
internet-side (`tools/fetch-advisories.sh`) and hands a sealed znippy across the airgap — this
verb is offline.
**The sources (each optional; at least one required):**
- `--osv` / `--malicious` / `--nvd` / `--nvd-backfill` — the CVE/GHSA/RUSTSEC + OpenSSF-malicious
+ NVD CVSS/CWE + CPE-backfill advisory corpus (see [`feeds-design.md`](feeds-design.md) f1).
- `--epss` / `--kev` — EPSS probability + KEV flag *enrichment* folded onto matched advisories.
- `--exploit-poc <dir>` (PoC-in-GitHub) / `--metasploit <file>` / `--nuclei <file>` (NDJSON template
index) / `--exploitdb <files_exploits.csv>` (the Exploit-DB index — a *published, downloadable*
exploit, the strongest of the four) — public-exploit availability. The four **union** into ONE
exploit-CVE set and a matched CVE is lifted to ≥High and summary-tagged `[public-exploit]`,
matched by id **or alias** so a GHSA-primary merged row still gets the lift.
- `--crates-yanks <dir>` / `--npm-deprecations <dir>` / `--pypi-yanks <dir>` / `--go-retract <dir>` —
registry yank/deprecation/retraction rows (`MODGUNN-YANK-*` / `MODGUNN-DEPRECATED-*`); crates/npm/pypi
exact-version keyed, Go `retract` supports an inclusive `[lo,hi]` range. `--go-retract` reads
`go.mod`/`*.mod` text (the only Go yank signal — Go has no central yank API).
**Anti-silent-empty.** A *per-doc* parse failure (a truncated OSV/registry doc) is named on
stderr + counted, never silently dropped; a well-formed doc that yields a real 0 (nothing
yanked) is fine; but an *enrichment* feed (`--epss`/`--kev`) or the public-exploit union
(`--exploit-poc`/`--metasploit`/`--nuclei`/`--exploitdb`) that was provided yet parsed to 0 rows is a
truncated catalog that would silently disable its escalation — so the import **bails** rather than
write a fake pass. `--exploitdb` adds one more edge: a CSV whose header carries no `codes` column is
not an Exploit-DB index at all (a wrong file, or a fetched HTML error page), so it is rejected outright
rather than read as “no exploits”.
## f0 — build an AdvisoryDb in code (no warehouse)
```rust
use modgunn::{osv, feeds};
use std::collections::{HashMap, HashSet};
// 1 · parse + dedup an OSV directory's worth of docs
let mut advisories = osv::parse_and_merge(osv_json_strings);
// 2 · enrich with exploit intelligence
let epss: HashMap<String, f64> = feeds::parse_epss_csv(&epss_csv);
let kev: HashSet<String> = feeds::parse_kev_json(&kev_json)?;
feeds::enrich(&mut advisories, &epss, &kev);
// 3 · hand to Engine A
let db = modgunn::scan::AdvisoryDb::new("2026-07-04-a", advisories);
```
## f1 — per-source recipes
- **OSV / GHSA** — `osv::parse_doc(doc)` auto-detects single vs batch; `osv::parse_and_merge`
over a whole dir is the usual call (it dedups).
- **RUSTSEC `.md`** — `osv::parse_rustsec_md(md)` (feature `rustsec`); its `patched`/`unaffected`
front-matter is inverted into affected ranges by `osv::rustsec_affected_ranges`.
- **NVD** — `nvd::parse_nvd(doc)` → `Vec<NvdEntry>`, then `nvd::enrich_with_nvd(&mut advisories,
&entries)` to fold CVSS vector + CWE + severity by id/alias. For CVEs OSV misses entirely,
`nvd::parse_nvd_advisories` emits `ecosystem = "cpe"` standalone advisories (`--nvd-backfill`).
- **VEX** — `vex::collect_suppressions(docs)` → the id set; pass it as `--vex <dir>` to
`warehouse scan`, or load it into `Policy::vex_not_affected` for Engine A.
## f2 — version matching
To test whether a concrete version is hit by an advisory's ranges:
```rust
use modgunn::versions::affected_by;
let hit = affected_by("cargo", &advisory.ranges, "1.2.3");
```
`Advisory::ranges` is a **disjunction** — one event list per OSV range object — so the
scan-time call is `affected_by_any`, which ORs them. `affected_by` evaluates ONE range; never
concatenate two ranges' events into a single list, because the walk is a monotone on/off sweep
and the narrower range's `Fixed` would cancel the wider one's window.
```rust
use modgunn::versions::affected_by_any;
let hit = affected_by_any("cargo", &advisory.ranges, "1.2.3");
```
The comparator is picked from the ecosystem automatically — semver for
cargo/npm/go/nuget/composer, PEP440 for pip, RubyGems segments for gem, dpkg/rpm/apk for distro
packages. Anything else (notably NVD's `cpe`) falls back to a loose dotted-numeric compare that
is NOT prerelease-aware. Feed OSV distro labels through `versions::normalize_ecosystem` first so
`Debian`/`Alpine`/`Red Hat` map to `deb`/`apk`/`rpm`.
## f4 — score a package's install behaviour (A23, no CVE needed)
To flag a *malicious* package that has no advisory yet, fold its `package.json` text into the
A23 behaviour-feature vector (needs `--features scan`; offline, no code run):
```rust
use modgunn::behavior::{parse_npm_package, parse_pypi_setup, parse_pyproject_toml, parse_setup_cfg};
// npm — reads scripts{}, no exec (Result: the JSON can be malformed)
let b = parse_npm_package(package_json_text)?;
if b.network_in_install {
// dropper pattern: an install hook fetches stage-2 from the network
}
// PyPI setup.py — scans the whole file text, no Python run (infallible text fold)
let p = parse_pypi_setup(setup_py_text);
// PyPI pyproject.toml — SECTION-AWARE: only build/install-active sections scanned
let q = parse_pyproject_toml(pyproject_toml_text);
// setuptools setup.cfg — SECTION-AWARE INI: only build/install-active keys scanned
let c = parse_setup_cfg(setup_cfg_text);
for f in b.findings().iter().chain(p.findings()).chain(q.findings()).chain(c.findings()) { // Vec<core::Finding>
println!("{} · {} — {}", f.severity, f.id, f.summary);
}
```
**npm** scores only the auto-run hooks (`preinstall`/`install`/`postinstall`/`prepare`) — a
`test`/`build` script the user invokes on purpose is not the malware vector and is ignored.
**PyPI**'s `setup.py` runs *in full* at `pip install` from an sdist, so `parse_pypi_setup` scans
the whole file; its install-hook signal is a `cmdclass` install/develop override, and it extracts
the same features (a whole-file `urllib.urlopen` + `os.system` still trips the network finding
with no `cmdclass`). A **`pyproject.toml`** is *declarative* TOML, so `parse_pyproject_toml` is
**section-aware** (NOT a whole-file scan): it folds only the build/install-active sections
(`[build-system]` / a PEP 517 in-tree `backend-path` / a `cmdclass` table / a `[tool.*.build.hooks…]`
build-hook) and skips the declarative `[project]` deps & URLs — so an honest project's `urllib3`
dependency or `https://` homepage does **not** false-positive, while a build-hook that `curl … | bash`es
a payload IS flagged High. It uses a tiny line-scanner (no `toml` dep) so it stays in the lean `scan`
gate. A **`setup.cfg`** is *declarative* INI with the same hazard, so `parse_setup_cfg` is likewise
**section-aware** (no `ini` dep): it folds only the `[options]` `cmdclass` / `dependency_links` keys
and a build-active `[options.entry_points]` group, and skips `[metadata]` (incl. `url`) and every
declarative dep key (`install_requires` / `extras_require` / …) — so a benign `install_requires`
`urllib3` or a `[metadata] url = https://…` does **not** false-positive, while a `dependency_links`
payload fetch IS flagged High. All four land in the SAME `PackageBehavior`. The strongest finding is **High**
`MODGUNN-BEHAVIOR-NETWORK-IN-INSTALL` (a `curl`/`wget`/`fetch(`/`urllib`/URL fetch); the
`sensitive_apis` set (npm `child_process`/`eval(`/`atob(`; Python `os.system`/`marshal.loads`/…)
and a high-entropy base64 blob are Medium; merely declaring an install hook is Low.
### f4 — from the CLI (no code)
Vet a single package on a laptop, offline, without writing Rust:
```sh
# ecosystem inferred from the basename
# package.json → npm · setup.py → pypi · pyproject.toml → pyproject · setup.cfg → setupcfg
modgunn behavior --path ./suspect/package.json
modgunn behavior --path ./suspect/setup.py
modgunn behavior --path ./suspect/pyproject.toml
modgunn behavior --path ./suspect/setup.cfg
# force the parser for an oddly-named file, and rename the subject
modgunn behavior --path ./blob.json --ecosystem npm --subject suspect-pkg
# gate a promotion: exit 3 if ANY finding fires
modgunn behavior --path ./package.json --fail-on-finding
# PERSIST the analysis into the warehouse (verdict ledger), then read it back
modgunn behavior --path ./package.json --warehouse ./wh # append vector + verdict
modgunn warehouse behavior-log --warehouse ./wh # latest per (ecosystem, subject)
```
It prints the feature vector (`hooks=… network_in_install=… urls=… entropy=… apis=[…]`) then one
line per finding (`High MODGUNN-BEHAVIOR-NETWORK-IN-INSTALL — …`) and a `worst <sev>` footer. It
folds through the SAME `parse_npm_package`/`parse_pypi_setup`/`parse_pyproject_toml`/`parse_setup_cfg` the library exposes — no
network, no code executed. Without `--fail-on-finding` it always exits 0 (a report); an unknown
basename with no `--ecosystem`, or a missing/malformed file, is a usage error (non-zero).
Add `--warehouse <root>` to **persist** the ecosystem-tagged feature vector + its derived verdict
(worst severity + finding ids) into the `behavior_features` table via the SAME warehouse-persist
path `scorecard --warehouse` uses (no second seam). `warehouse behavior-log --warehouse <root>
[--ecosystem <eco>] [--subject <name>]` reads the latest analysis per `(ecosystem, subject)` back
(worst-severity first) — the append→load round-trip. The persist leg needs a `--features warehouse`
binary; a plain `behavior` (no `--warehouse`) touches no warehouse code.
## Gotchas
- Ranges are the source of truth; don't pre-expand into `affected_versions` unless you need a
cache — a version published after expansion would read "not affected".
- `merge_by_alias` collapses CVE/GHSA/RUSTSEC rows that share an alias — expect fewer rows out
than docs in, and a union of their ranges.
- Enrichment is by CVE id: an advisory whose primary id is a GHSA still gets EPSS/KEV if a CVE
alias matches.