> **RULE: Never add Co-Authored-By or any AI attribution to commits. Ever.**
# korp-collectors
The standalone LEAF of korp's open-data collector layer — one object-safe
`Collector` trait + per-access-METHOD collectors (DCAT-AP-SE, PxWeb/JSON-stat2,
ENTSO-E, generic REST/JSON, Trafikverket, OSM) parsing pure-Rust → Arrow, an
Iceberg `BatchSink` (via skade), and the `RegionSource` fetch seam.
## Why it is a leaf
It depends on **neither `korp` nor `facett`**, so BOTH `korp` and `facett-demo`
can depend on it. That breaks the `facett ⇄ korp` dependency cycle: `facett`
never depends on `korp`; both consume this leaf, which depends on neither.
Keep it a leaf. Allowed deps: `arrow-array`/`arrow-schema` 58, `parquet` 58,
`anyhow`, `async-trait`, `reqwest` (rustls), `serde`/`serde_json`, `governor`,
`bytes`, `tokio`; feature-gated: `osm-katana` (`osm`), `skade` (`skade-sink`),
`nornir-testmatrix` (`testmatrix`). **No `facett-*`, no `korp` dep — ever.**
## Features
`parsers` (default) · `osm` (osm-katana + RegionSource) · `skade-sink` (Iceberg
runner) · `testmatrix` (nornir functional-status rows; off by default).
## Laws
* Every crate builds AND has real inject/assert tests (no "didn't panic").
* RAGNARÖK: a collect landing 0 rows is RED unless `CollectStats.reason` says why.
* The `RegionSource` fetch test uses REAL OSM bytes (osm-katana parses ≥1 way),
an in-process server, and a request counter — deterministic, no live network.
* Tests emit `nornir_testmatrix::functional_status(...)` rows under
`--features testmatrix` (feature OFF = no-op, no nornir dep). Use the crate
wrapper `korp_collectors::functional_status(...)`; `emit_registry_descriptors()`
emits one `descriptor_declared` row per collector (discovery-complete law).
* Every `Collector` DECLARES a typed `source_target()` → `SourceTarget` (the
open-data endpoint it pulls FROM + where rows land: warehouse table / Arrow /
a facett map scene). `registry_descriptors()` enumerates `(meta, source→target)`
so the test-matrix iterates `(collector, source, target)` tuples.
* Real-or-fake samples: every collector has a `sample::*_try_get_real_sample_data()`
that ATTEMPTS a short-timeout live fetch (validated against its own parser) and
FALLS BACK to an embedded `Synthetic` fixture — tests + benches never hang and
never fail for want of a network. `KORP_COLLECTORS_OFFLINE=1` forces synthetic.
* Benches (`benches/collectors_parse.rs`, criterion, `harness = false`) measure
per-collector parse/normalize throughput over that same offline-safe sample.
## Direction taxonomy
Every `Collector` is `Direction::Source`; the `BatchSink` is `Direction::Sink`.
This axis is orthogonal to `SourceKind` (the transport/format family).