# Bifröst — tables → graph, reasoning
> **Class:** `reasoning`. This file holds the *why*, the trade-offs, **every design we
> rejected**, and the honest limitations. The descriptive shape we run is in
> [`bifrost-design.md`](bifrost-design.md), which by rule mentions none of this. If you want
> to know "why not X?" — X is here.
> Reuse: `bifrost.uses = [falkordb, graphar]`.
## Why the fabric at all
Bifröst exists to make lakehouse tables *queryable as a graph* continuously, without a batch
export step. The whole design serves four goals, and every law below is in service of them:
1. **Arrow end to end.** Spark Connect and skade both hand back Arrow `RecordBatch`es; they
flow through transforms and into the FalkorDB loader untouched. No row-object marshalling,
no intermediate serialization.
2. **Declarative first, code when needed.** ~90% of table→graph mappings are "this table is
a label, these two columns are an edge" — those are YAML `vertex`/`edge`/`fk_edges` steps.
The remaining 10% get a full Rust `Transform` with the whole Arrow API. The config stays
declarative; the Rust stays out of config files (`custom: name` + a registry).
3. **Exactly-once-ish.** Per-table progress is a cursor, data ships once, replays are safe.
4. **Source-agnostic.** One `TableSource` trait; Spark, Spark-free Iceberg, and file drops
are all just implementors.
## Why state-after-sink (exactly-once-ish)
The cursor is written **only after** the sink round succeeds. A crash between sink and cursor
replays the whole round — and because the sink is `MERGE` keyed on a stable id, a replay is
idempotent (the second MERGE is a no-op). This is cheaper and simpler than a two-phase commit
and gives at-least-once delivery with idempotent effect = effectively exactly-once. The
alternative — advancing the cursor first, or transactionally — either drops data on a crash
or couples bifröst to sink-side transactions it does not own.
## Why MERGE both endpoints on every edge
Tables sync in **arbitrary order**, so an edge can land before the table that owns one of its
endpoints. If the sink only matched existing nodes, that edge would be dropped. Instead every
edge MERGEs *both* endpoints, creating a placeholder node on the id key; when the owning
table later syncs, its vertex MERGE on the **same id** enriches that placeholder in place. No
ordering constraint on the mapping, no lost edges.
## Why `REFRESH TABLE` before each Spark poll
Spark caches Iceberg metadata per session. A long-lived poller that never refreshes would
keep seeing its own session's stale snapshot and miss every other writer's commits. The
Spark source issues `REFRESH TABLE` before reading the snapshot list so it sees the true
latest commit each round. (The Spark-free `IcebergSource` reopens the catalog per poll, so it
has no equivalent staleness.)
## Why backoff only on a no-progress round (idle ≠ failure)
An idle round — the source simply had no news — is the *normal* steady state of a poller, not
a failure. Backing off on idle would slow a healthy quiet system to a crawl. So a round is a
failure (`RoundReport::is_failure`) only when ≥1 table was attempted and *every* attempt
failed; an idle round (nothing synced, nothing failed) keeps base cadence, and a partial
round (some tables synced, some failed) counts as **progress** and resets the schedule.
Consecutive failures double the sleep to a cap; the first progressing round resets it.
## Why declarative-first, code-when-needed (not one or the other)
A pure-YAML DSL would eventually grow a scripting language to express the 10% hard cases —
reinventing Rust badly. A pure-Rust API would push the trivial 90% into code that a data
engineer shouldn't have to touch. The split keeps each in its lane: YAML for shape, a `custom`
Rust `Transform` for logic, joined by name through the registry.
## Rejected: a plugin system (why a compile-time registry, for now)
The fabric is a **compile-time** `TransformRegistry`: you depend on `knut-bifrost` and hand
it your `Arc<dyn Transform>` impls before the listener starts. We did **not** build a plugin
loader. Compile-time registration covers the embedding use case fully, with zero dynamic-load
surface. When out-of-process transforms *were* needed, the answer was not a plugin ABI but
two implementors of the **same** `Transform` trait that keep `GraphMutation` as the contract:
`WasmTransform` (a sandboxed `wasmi` guest, Arrow IPC in/out) and `FlightTransform` (a
polyglot Arrow-Flight `DoExchange` service, any language, out of process). Both are just
`custom` transforms — no new plugin machinery, same currency.
## Deferred / future sources
- **Iceberg REST catalog** (via `iceberg-rust`) — a third `TableSource` for shops that run a
REST catalog instead of skade's embedded one. Slots behind the same trait; not yet built.
- **Debezium CDC** — a Debezium topic → batches → the same fabric, as an alternative CDC feed
to the Iceberg changelog path. Deferred; the `SyncMode::Cdc` routing already exists to
receive it.
(Both the Spark-free Iceberg source and Airflow AIP-82 emission started life on this list and
are now shipped — see [`bifrost-design.md`](bifrost-design.md) f1/f6.)
## The honest limitation: iceberg-rust has no incremental scan
**iceberg-rust 0.9.1 (what skade pins) does NOT expose an incremental / changelog scan.** Its
`TableScanBuilder` has `snapshot_id(id)` (time-travel to one snapshot) but no
`from_snapshot_id`/`to_snapshot_id` and no `create_changelog_view` equivalent — that
procedure is Spark-only. So the Spark-free delta had to be **built from the manifest layer by
hand** (skade's `Table::read_delta`): walk the snapshot lineage back to the cursor, classify
each `Added` manifest entry (`Data` → inserted parquet, `EqualityDeletes` → a delete file),
and read *only* those files. That manifest diff is the reason a Spark-free delta exists at
all, and its two honest edges:
- **Equality deletes, not position deletes.** A `Delete` snapshot with **equality**-delete
files (delete by business-key value) is a first-class CDC delete (`_change_type = DELETE` →
`VertexDelete` → `DETACH DELETE`), exercised by a real unit + container test. **Position**
deletes (delete by file + row offset) can't be mapped to a graph identity, so they set
`needs_full_reload` and the whole target snapshot is re-read — correctness over speed.
- **`needs_full_reload` is un-exercisable through this stack.** iceberg-rust 0.9.1's
`Transaction` only does `fast_append`, so skade cannot *produce* an overwrite or a
position-delete to stage a test for it. The branch is defensive for a table some external
engine (Spark, PyIceberg) overwrote; it is covered by the code path and this doc, not a
live test.
## Why the spark-connect fork is vendored in-tree (off-box buildability)
The `spark` source needs `spark-connect`, but upstream 0.2.2 (the newest published
release) still pins **arrow 56** — which dragged a whole second arrow-56 tree into
knut just for the Spark path, forcing an IPC round-trip bridge to hand its batches
to the arrow-58 workspace. The fix was a fork that bumps only the arrow version
(56 → 58, *no source changes* — the crate's Arrow-IPC surface is API-compatible
across the bump), so the entire dependency graph stays arrow-58-only.
We **vendored** that fork into `vendor/spark-connect-arrow59/` rather than pointing
the patch at an external sibling checkout. The reason is off-box buildability: a
`[patch.crates-io]` at a `../sibling` path means a bare `git clone knut` cannot
build the `spark` feature at all. In-tree, with every proto committed and a
vendored protoc in `build.rs`, the Spark path builds from a clone alone — no
sibling repo, no system protoc, no network. It reverts to the crates.io crate the
moment upstream ships an arrow-58 release (see the header of
`vendor/spark-connect-arrow59/README.md`).
This closed the spark-connect leg of off-box build. The remaining bare-clone
blockers — the root `[patch.crates-io]` on `skade`, plus sibling `path` deps on
`draupnir`, `nornir`, `znippy-zoomies`, `facett-pop` and `facett-pipeline` — are
now **gone too**: every one of those crates had published a version carrying the
API knut needs, so each is a plain crates.io dependency. `skade`'s
equality-delete surface (`DeltaPlan::delete_files`, `read_equality_deletes`,
`Table::delete_equality`) shipped in **skade 0.5.0**, which retired the patch
entirely. A bare `git clone knut && cargo build --workspace` now resolves.
Two properties of cargo make this worth stating plainly, because both were
mis-assumed in this tree at one point:
* A `path` that escapes the repo breaks resolution for the **whole workspace**,
not just the crate declaring it — manifests are read before features are
considered.
* Neither `optional = true` nor an accompanying `version = "…"` protects you.
The first blocker a bare clone actually hit here was `draupnir`, which was
both optional *and* versioned.
`crates/knut-tracing/tests/no_sibling_path_deps.rs` walks every manifest in the
repo and fails if any `path` resolves outside the repo root, so this cannot
silently come back. In-repo relative paths and `vendor/` are what it permits —
vendoring in-tree, as spark-connect is, remains the sanctioned escape hatch for
a crate that genuinely cannot be published.
## Known gaps
Composite / multi-label MERGE keys and edge-only changelog deletes remain open. TLS/auth on
the `/metrics` endpoint is out of scope — front it with a reverse proxy if you expose it
beyond a private scrape network.