# Native GRAPH.BULK fast-lane — design
> **Class:** `design` · **Function axis:** `f6` native bulk encoder · `f7` auto-select · `f8` streamed/appendable sharding · `f9` chunked multicore encode.
> Extends [`falkordb-design.md`](falkordb-design.md) (the UNWIND load bridge). Reuse graph: `falkordb.uses = [graphar]` — endpoint identity is graphar's `_gar_id`.
> Strict: this file states *only the shape we run*. The *why* is in [`falkordb-reasoning.md`](falkordb-reasoning.md).
## f6 — the native `GRAPH.BULK` encoder (the fast lane)
Alongside the `UNWIND … CREATE` path, `graphar-falkordb` speaks FalkorDB's **binary
bulk-insert protocol** natively (`src/bulk.rs`) — the same wire the `falkordb_bulk_loader`
uses, but emitted in pure Rust straight from Arrow columns: **no CSV, no Python, no shell**
(L9). `GRAPH.BULK` bypasses the Cypher parser/planner entirely, so it is the **initial /
empty-graph population** fast lane; `UNWIND` stays the incremental/upsert path.
**The wire** (mirrors FalkorDB `src/bulk_insert/bulk_insert.c`, verified live):
```
GRAPH.BULK <key> [BEGIN] <node_count> <relation_count> <label_blob_count> <reltype_blob_count> <node_blob…> <reltype_blob…>
```
`BEGIN` **creates** the graph (first call to an absent key; errors if the key already exists);
the tokenless form **appends** (errors if the key is absent). Node global ids are assigned in
blob order — the first node of the first node blob is id `0` — which is exactly how a relation
blob's `uint64` endpoints reference nodes.
| blob | layout |
|---|---|
| **node** | `label\0` · `u32 prop_count` · (`propname\0`)* · then per node `prop_count` typed values |
| **relation** | `reltype\0` · `u32 prop_count` · (`propname\0`)* · then per relation `u64 src_id` · `u64 dst_id` · `prop_count` values |
**Typed value** = 1 type byte + payload: `NULL`(0)→∅ · `BOOL`(1)→1 byte · `DOUBLE`(2)→f64-LE ·
`STRING`(3)→bytes+`\0` · `LONG`(4)→i64-LE · `ARRAY`(5)→`u64-LE len`+that many values. Integers
and floats are little-endian (the server reads them with a host-order pointer cast).
| law | statement | enforced by |
|---|---|---|
| **B-1** | Arrow → bulk value is total; an unmapped type is a loud error | `encode_cell` → `FalkorError::UnsupportedType` (`src/bulk.rs`) |
| **B-2** | A node's props are `_gar_id` (absolute 0-based row index in its group) then every column — identical to what `insert_vertices` writes. **If the batches already carry a `_gar_id` column** it is used verbatim (no duplicate, stored as-is), so a caller with a *global/offset* `_gar_id` (knut-popviz) stays byte-identical to its UNWIND load | `encode_node_blob` |
| **B-3** | Edge endpoints are per-endpoint-label `_gar_id`s translated to global node ids by the endpoint group's base offset; out-of-range / unknown-label endpoints are a loud error (the format can't MATCH-fail) | `build_bulk_payload` / `resolve` |
Non-finite floats and nulls both encode as `NULL` — exactly what the UNWIND path renders as
Cypher `null` — so the two paths store the same value (B-2).
## f7 — auto-select (initial → bulk, incremental → UNWIND)
`choose_bulk(graph_is_empty)` returns the fast path **only** for an empty graph (bulk `CREATE`s,
it can't MERGE onto existing nodes). `FalkorDbLoader::graph_is_empty()` probes `count(n)`;
`bulk_insert` then reads `EXISTS` (a non-mutating check) to decide `BEGIN` (absent key) vs
append (a key a probe already materialized).
| fn | signature | does |
|---|---|---|
| `bulk_insert` | `bulk_insert(&mut self, nodes, edges) -> (usize, usize)` | encode + ship the whole graph in one `GRAPH.BULK` round trip |
| `bulk_insert_streamed` | `bulk_insert_streamed(&mut self, nodes, edges)` | one appendable call per node/reltype blob (f8) |
| `graph_is_empty` | `graph_is_empty(&mut self) -> bool` | the auto-select signal |
| `choose_bulk` | `choose_bulk(empty) -> bool` (free fn) | pure policy: bulk iff empty |
| `build_bulk_payload` | `build_bulk_payload(&[BulkNodeGroup], &[BulkEdgeGroup]) -> BulkPayload` (free fn) | the pure, DB-free encoder core |
**Consumer — `knut-popviz` auto-selects bulk.** The population demo's `load_falkor`
(`crates/knut-popviz`) now takes the same fork: after the `--drop` `DETACH DELETE`, an empty
graph auto-selects the `GRAPH.BULK` streamed lane (`IngestMode::Auto` → `choose_bulk(graph_is_empty)`),
a non-empty one keeps the parallel-`UNWIND` path. Two reconciliations make popviz's graph
byte-identical either way: its vertex batches carry a **global, offset-based `_gar_id` column**
(reused verbatim by B-2's supplied-column mode — no duplicate, no 0-based flip), and the bulk
lane regenerates edges with **zero offsets** so the encoder's `base + position` (B-3) reproduces
the same global endpoints the UNWIND path MATCHes. Node groups are passed in `LABELS` order, so
each group's base offset equals popviz's `offsets[li]`. Proven live by
`knut-popviz`'s `popviz_bulk_matches_unwind_byte_identical` (byte-identical across all 12 labels
+ 11 rels) and pinned DB-free by `bulk_groups_reconcile_gar_id_and_edges`.
## f8 — streamed / appendable sharding
`bulk_insert_streamed` sends each node group, then each edge group, as its **own** appendable
`GRAPH.BULK` call (`BEGIN` only on the first to an absent key) instead of one giant command —
bounding per-command size/memory for very large loads. It is **ordered on purpose**: the
server assigns global node ids in call order, and every node must exist before its edges, so
the streamed result is byte-identical to the single-shot `bulk_insert`. This is the seam for
future multi-graph-key sharding (split disjoint subgraphs across keys, then alias/merge) —
deferred until a live-DB `GRAPH.RESTORE`/merge story is proven (see reasoning).
## f9 — chunked, multicore encode (no single-core stall, no broken pipe)
At demo-zone max scale the dominant label (a 60M-row `TaxRecord`) used to encode its whole
blob on **one core** and then ship it as a single `GRAPH.BULK` command whose payload blew
past FalkorDB's `proto-max-bulk-len` (512 MB default) — the server closed the socket
mid-write and the fill died with `Redis error: Broken pipe`. Fixed in `build_bulk_payload`:
each node/edge group is split into ≤ `MAX_ROWS_PER_BLOB` (2M) blobs and the blobs are
encoded **in parallel across all cores via gatling** (LAW #0: gatling, never rayon), laid
down in order. Because blobs stay ordered — every chunk of label `li` before `li+1`, chunks
in row order — the server assigns the same contiguous global ids, so the graph is identical
byte-for-byte; only the wire is split (and progress reporting gets finer for free).
`encode_node_blob` gained a `gar_base` so a split label keeps assigning contiguous
synthesized `_gar_id`s (byte-identical to the one-blob path; popviz's supplied `_gar_id`
column is still passed verbatim). Every command now stays well under the wire limit.
Guarded by chunk-window splitting, `gar_base` continuity across a split, and
chunked-vs-single-blob graph equivalence (edges resolve across the boundary). Measured
live (`e0c0378`, scale 0.3, oden): **46 GRAPH.BULK blobs encoded across up to 32 cores in
3.65 s** (36,360,000 nodes + 33,060,000 relations; each blob < `proto-max-bulk-len`) → fill
OK in 31 s, no broken pipe — the exact scenario that crashed before now saturates the box.
## Parity + numbers
The container-gated `bulk_matches_unwind_byte_identical` proves a `GRAPH.BULK` load (single-shot
**and** streamed) yields the byte-identical graph — same nodes, `_gar_id`, properties, and
relationships (incl. edge props) — the `UNWIND` path produces from the same Arrow batches. DB-free
byte-exact tests in `src/bulk.rs` guard the encoder (header layout, type bytes, endianness,
`_gar_id` scheme, endpoint translation, out-of-range errors) with no server.
Live bench (`bench_bulk_vs_unwind`, 200k nodes, debug, Loki, one FalkorDB container):
| path | nodes/s |
|---|---|
| single-connection `UNWIND` (baseline) | ~25k |
| parallel `UNWIND` (`insert_vertices_par`) | ~56k |
| **native `GRAPH.BULK`** | **~2.69M (≈48× the parallel path, ≈108× the baseline)** |
End-to-end **consumer** number — `knut-popviz`'s whole 12-label population (nodes + edges +
`_gar_id` indexes) via `bench_popviz_bulk_vs_unwind` (60,600 vertices, debug, Loki 12c, one
container): parallel-`UNWIND` **16.1k nodes/s** → streamed `GRAPH.BULK` **356k nodes/s = ≈22×**
(lower than the raw ingest ratio because it wraps edge load + index creation, the honest demo-fill wall).