# GraphAr gar/v1 substrate — reasoning
> **Class:** `reasoning`. This file holds the *why*, the trade-offs, and the gaps. The
> descriptive shape we run is in [`graphar-design.md`](graphar-design.md), which by rule
> mentions none of this. If you want to know "why not X?" or "what doesn't it model yet?" —
> it's here.
## Why five interchangeable formats
The `gar/v1` layout is format-agnostic on purpose: a property group is *a chunk*, and the
chunk's encoding is a storage decision, not a data one. We support all five — `parquet`,
`csv`, `orc`, `json`, `arrow-ipc` — because each earns its place downstream: parquet for
compact analytic storage (the default), orc for Spark/Hive interop, arrow-ipc for zero-copy
hot paths, csv/json for human-readable fixtures and debugging. The law that makes this safe
is G-1: every codec round-trips the **same Arrow schema**, so the format is swappable per
property group without touching a single consumer. If two formats disagreed on schema, this
whole property would collapse — which is what the round-trip conformance test guards.
## Why arrow 58 pinned workspace-wide (one tree)
Arrow is not just a dependency here, it's the *lingua franca* between graphar, orc-rust,
spark-connect, Flight SQL, and FalkorDB load. If two of those linked different arrow major
versions, an `arrow::RecordBatch` produced by one would be a different, incompatible type to
the other — no zero-copy hand-off, silent version-mismatch errors at the seam. So the whole
workspace pins **one arrow tree (58)**: orc-rust and spark-connect are held to the same
version graphar reads, and a batch flows edge-to-edge without a re-encode. The cost is that
bumping arrow is a workspace-wide event, not a per-crate one — a deliberate trade for one
coherent columnar type across every crate.
## Why `_gar_id` is the identity
A property graph needs a stable internal handle for endpoints. We use GraphAr's `_gar_id` —
a vertex's absolute 0-based row index across its chunks — rather than any user property,
because it is dense, contiguous, and assigned by the writer, so it's always present and always
unique even before the user declares a primary key. Edges carry `src`/`dst` as `_gar_id`s
(`Edge::new(i64, i64)`), which is exactly the key [FalkorDB MERGE](falkordb-reasoning.md) and
the Flight SQL vertex/edge schemas build on. A user-property identity would be optional,
possibly non-unique, and typed — three ways to be missing when an edge needs to resolve an
endpoint.
## Why object storage is first-class, not bolted on
The `ChunkStore` is generic over `object_store` from the start rather than being a
local-filesystem type with an S3 adapter grafted on later. A graph the same size as the bench
fixtures lives happily on disk; a 120M-node population graph does not, and the moment you need
it on S3/GCS/Azure the *entire* read/write path — not just a copy step — has to speak the
remote dialect. Making `ChunkStore` `Arc<dyn ObjectStore>`-generic means local and remote are
the same code path (G-3); the only backend-specific thing is the constructor.
**Why retry and multipart are first-class too.** Remote object stores fail transiently
(throttling, connection resets) and reject single-`put` uploads above a size limit — so a
substrate that's "remote-capable" but retries nothing and can't upload a big chunk is
remote-capable in name only. We front `object_store`'s native retry with the typed
`RetryOptions` (so callers tune backoff without importing `object_store`'s config type) and
route large writes through `put_multipart`. The multipart part size is kept ≥ 5 MiB because
S3/GCS/R2 all reject smaller non-final parts; any error `abort()`s the upload so those stores
don't leak orphaned parts. Default local / in-memory traffic stays under the 16 MiB threshold
and never goes multipart, so the common case pays nothing.
*Gotcha:* `from_url_with_retry` can't use `object_store::parse_url_opts` — that entry point
won't thread a typed `RetryConfig` through — so it dispatches on `ObjectStoreScheme` and
applies `with_retry` on the per-scheme builder instead. Failure-injection against real S3
stays container-gated; the unit tests only cover the config plumbing.
## Corpus model-gaps (validator skips, not failures)
Validating against the real `apache/incubator-graphar-testing` corpus surfaced two gaps that
are **skips, not bugs**:
- **`list<float>` parameterized type** — knut's `DataType` enum has no parameterized list
type, so the one corpus graph using `list<float>` is skipped rather than mis-modelled.
Adding it means teaching `DataType` about parameterized element types.
- **The `java/` subtree** — uses a *different, non-`gar/v1`* info-file schema (`label`
instead of `type`). It isn't a gar/v1 graph, so the corpus test skips it by design rather
than reporting spurious errors.
To read the upstream info files at all, `GraphInfo.prefix` had to become `#[serde(default)]`
— the corpus `*.graph.yml` omit it. During development the corpus test validated **12 of the
corpus's gar/v1 graphs cleanly** with those two skips.
## Why round-trip conformance always, corpus opt-in
Two tiers, deliberately asymmetric:
- **`round_trip_conformance`** runs **always**: it builds a graph with
`VerticesBuilder`/`EdgesBuilder`, dumps it, asserts the exact spec paths exist, and checks
`validate()` is error-free. This is the gate that keeps G-1 honest and catches a codec or
layout regression on every CI run, with zero external dependencies.
- **`corpus_validates`** is `#[ignore]`d and gated on `GRAPHAR_TESTDATA` pointing at a local
checkout of the upstream corpus (validated with `ValidateOptions::corpus()` for
extensionless chunks). It never fetches over the network. It's opt-in because it needs a
multi-hundred-MB checkout no CI runner should clone on every build — it's the conformance
proof you run deliberately, not the regression guard you run constantly.
## Notes / measured
- Bench targets `knut.format_{parquet,csv,json,orc,arrow_ipc}`, `knut.builder_dump`,
`knut.merge_property_groups` — see [`benching-benchmarks.md`](benching-benchmarks.md). Fixtures are synthetic and
on `/tmp`, never the T9 USB drive (external IO skews throughput).
- Object-store round-trips (SeaweedFS / RustFS) are container-gated `--ignored` tests, not in
the release gate. Validation is read-only and never touches the network.