# Arrow → FalkorDB bulk load — reasoning
> **Class:** `reasoning`. This file holds the *why*, the trade-offs, and the gotchas. The
> descriptive shape we run is in [`falkordb-design.md`](falkordb-design.md), which by rule
> mentions none of this. Reuse graph: `falkordb.uses = [graphar]` — the `_gar_id` identity is
> defined in [`graphar-reasoning.md`](graphar-reasoning.md).
## Why UNWIND batches, not row-by-row
The naïve loader issues one `CREATE` per row. That's one Redis round trip per node — for a
120M-node graph, 120M round trips, each dominated by network latency, not by the work. So we
render a whole batch of rows as a single `UNWIND [ {..}, {..} ] AS r CREATE …`: one round
trip amortises `DEFAULT_BATCH_SIZE` (512) rows, and FalkorDB plans the CREATE once for the
whole list. The batch size is the tuning knob (`with_batch_size`); the shape — one statement
per batch — is the law (F-1).
## Why the hot path is column-resolved (H1)
The obvious way to build a Cypher map is per cell: match on the Arrow `DataType`, downcast the
array, format the value into a fresh `String`, join. For a batch of 512 rows × M columns
that's 512·M downcasts and as many temporary `String`s — the downcast and the allocation, not
the formatting, dominate. `ColRenderer::resolve` downcasts each **column once** (M downcasts
per batch, not 512·M), and `write_row_map` renders every cell straight into one pre-sized
buffer — no per-cell downcast, no intermediate `String`. This is the §H1 win the
`knut.cypher_render` bench measures; it's why the renderer is column-shaped, not row-shaped.
## Why Arrow over CSV
FalkorDB ships a `falkordb_bulk_loader` that eats CSV. We don't use it and we don't emit CSV:
- **Typed, columnar, no text parse.** Arrow batches are already the typed columnar form the
renderer wants; CSV would mean serialize-to-text then parse-back, throwing away the types on
the way out and re-guessing them on the way in.
- **The same batches persist elsewhere.** An Arrow batch loaded into FalkorDB is byte-for-byte
the same batch that persists as [GraphAr/parquet](graphar-reasoning.md) or an Iceberg table,
read back zero-copy. A CSV intermediate is a dead-end artifact — strictly worse than the
Arrow we already hold.
## Why knut never shells out to `falkordb_bulk_loader`
Shelling out to the external bulk loader would mean: materialize CSV to disk, spawn a Python
process, and hand it a *second* connection with its own auth, batching, and error surface that
knut can't observe. Instead every knut bulk path — `FalkorDbLoader`, parallel loaders, Flight
SQL DoPut, Bifröst's `FalkorDbSink` — goes over the same Redis/RESP wire through the same
`insert_*` methods. One code path, one error type (`FalkorError`), one place to tune, and the
loads compose (Flight and Bifröst *call* the loader rather than reinventing it).
## Why explicit-and-total value mapping (loud, not silent)
`arrow_value_to_cypher` maps a fixed set of Arrow types and returns
`FalkorError::UnsupportedType` on anything else. The tempting alternative — fall back to
`format!("{:?}")` for unknown types — is silent data corruption: a type we never validated
gets a debug-string rendering that *looks* like it loaded and is wrong. Making the map total
means an unmapped type is a **visible gap that fails the load**, not corrupt data sitting in
the graph. Adding a type is a deliberate act (extend the match), which is exactly the friction
we want on the correctness-critical seam. `Date`/`Timestamp`/`List` rendering was added this
way (done 2026-06-11).
## Why edges MERGE on `_gar_id`, index first
Edges reference endpoints by graphar's `_gar_id`, and both endpoints are matched before the
relationship is created. Without an index, each endpoint match is a label scan — O(nodes) per
edge, quadratic over a load. So the workflow is **`create_index(label, "_gar_id")` before any
edge MERGE**: build the range index while the graph is still just vertices, and every endpoint
lookup becomes a point read. This ordering is a workflow rule, not something the loader can
enforce — hence it lives in the guide and here, loudly.
## Gotcha: FalkorDB on Valkey 9
The FalkorDB module gates on redis-compat ≥ 8.0, but Valkey 9 reports `redis_version` as
7.2.x — so the module refuses to load on an otherwise-fine Valkey 9. The
`containers/falkordb-valkey9` Containerfile lowers that gate to 7.0 at build time so the module
loads. Live tests are container-gated (`--ignored`) and run against **both** stock FalkorDB
and FalkorDB-on-Valkey 9 (`KNUT_FALKORDB_IMAGE`), asserting node/edge counts and properties —
so this compatibility path is actually exercised, not assumed.
## Why a native GRAPH.BULK fast lane (and why it's not the default path)
`UNWIND … CREATE` makes the server parse + plan + execute Cypher per batch. For an *initial*
population that planning is pure overhead — the shape is always the same CREATE. FalkorDB's
`GRAPH.BULK` protocol skips the planner and ingests a typed, columnar node/relation payload
directly. knut already holds the typed Arrow columns, so a native Rust encoder (`src/bulk.rs`)
emits that payload with **zero CSV, zero Python, zero shell** (L9) over the same RESP wire and
the same `FalkorError` — the reasoning doc's "one code path over the wire" law holds, we just
add a faster verb on it. Measured ≈48× the parallel-UNWIND path and ≈108× the single-connection
baseline at 200k nodes (debug).
It is **not** the default because `GRAPH.BULK` `CREATE`s and cannot MERGE — it is safe only on a
fresh/empty graph. So `choose_bulk` gates it on `graph_is_empty()`, and the incremental/upsert
loads (bifrost sink, re-populate) stay on `UNWIND`. The result is provably identical: the
container-gated parity test reads both graphs back and asserts byte-equal node/edge rows.
**Gotcha discovered live (the BEGIN/EXISTS dance).** `BEGIN` creates the graph and errors if the
key already exists; the tokenless append errors if the key is absent. But the auto-select probe
(`graph_is_empty`, a `GRAPH.QUERY`) *materializes* an empty key — so after probing, `BEGIN`
would fail "key already exists". The loader therefore reads a non-mutating `EXISTS` and sends
`BEGIN` iff the key is absent, appending otherwise. Both branches are exercised by the live test
(single-shot on a probed key = append; streamed on a fresh key = BEGIN-then-append).
**Gotcha: the two blob-count tokens.** The wire is
`GRAPH.BULK <key> [BEGIN] <nodes> <rels> <label_blob_count> <reltype_blob_count> <blobs…>` — the
two entity counts are followed by the *count of node blobs* and *count of relation blobs*
(FalkorDB's "number of node descriptor tokens"). Omitting them makes the server read a blob as
that count and fail; this is only discoverable against a live server, hence the container gate.
## Deferred: true multi-graph-key sharding + offline RDB
`bulk_insert_streamed` chunks one load into ordered appendable calls (bounded memory) but still
targets one graph key. Splitting a load across *several* graph keys for parallel ingest needs a
correct cross-shard-edge story (edges whose endpoints land in different shards) plus a
merge/alias step — undocumented and module-version-specific in FalkorDB — so it is deferred to a
version-pinned spike. Likewise the genuine "server-off" RDB-assembly path: the serialized graph
form is module-version-specific and not appendable; `GRAPH.BULK` (fast + appendable + safe) is
the shipped answer.
## Note: composite keys + multi-label, done
Composite MERGE keys and multi-label nodes landed 2026-06-11 (plan §8). The design point worth
remembering: the single-key / single-label path is kept **byte-identical** to the historical
one — `insert_vertices_multi` with a one-label slice renders exactly `` `Label` ``, and
`insert_edges_keyed` with `["_gar_id"]`/`["_gar_id"]` still stages the legacy `src`/`dst`
carriers. New capability, zero regression on the old path.