# GraphAr gar/v1 substrate — design
> **Class:** `design` · **Function axis:** `f0` (the chunked-file substrate) + `f1` formats/io · `f2` metadata/info-files · `f3` readers/builders · `f4` ChunkStore/object-store · `f5` validation.
> Strict: this file states *only the shape we run*. Every *why*, every rejected alternative, lives in [`graphar-reasoning.md`](graphar-reasoning.md); the workflow is in [`graphar-guide.md`](graphar-guide.md).
## f0 — the chunked-file substrate (the whole)
A graph is a **directory of chunks**: column-oriented files of a fixed row count, described
by YAML info-files, storing one Apache GraphAr `gar/v1` property graph. graphar reads and
writes those chunks as Arrow `RecordBatch`es, in five interchangeable file formats, over
local disk or object storage (`crates/graphar/src`, re-exported from `src/lib.rs`).
Three laws hold across the whole substrate:
| law | statement | enforced by |
|---|---|---|
| **G-1** | Every chunk round-trips the same Arrow schema regardless of format | `io::write_chunk` / `io::read_chunk` dispatch on `FileType` (`src/io/mod.rs:22`,`:12`) |
| **G-2** | `_gar_id` is the vertex identity; edges reference endpoints by it | `Edge::new(src, dst)` takes `i64` gar-ids (`src/writer/edge_writer.rs:27`) |
| **G-3** | Storage is `object_store`-generic — one API over local/S3/GCS/Azure | `ChunkStore` wraps `Arc<dyn ObjectStore>` (`src/storage.rs:246`) |
**Reuse map (what the graphar subsystem `uses`):** it depends only on `arrow` (58, one tree
workspace-wide), `object_store`, and the format codecs (`parquet`, `orc-rust`, `arrow-ipc`).
It is used *by* [`graphar-falkordb`](falkordb-design.md) (bulk load), `graphar-flight` (Flight
SQL), `knut-bifrost`, and the bench suite — graphar is the columnar floor they all stand on.
## f1 — formats & io
`io::write_chunk(path, &[RecordBatch], &FileType)` (`src/io/mod.rs:22`) and
`io::read_chunk(path, &FileType)` (`src/io/mod.rs:12`) are the file-level seam. `FileType`
(`src/types.rs:50`) is the five-way format enum, each with a fixed extension via
`FileType::extension()` (`src/types.rs:60`):
| `FileType` | extension | codec |
|---|---|---|
| `Parquet` (default) | `parquet` | inline `io/parquet.rs` |
| `Orc` | `orc` | `io/orc.rs` (orc-rust) |
| `Csv` | `csv` | inline `io/csv.rs` |
| `Json` | `json` | `io/json.rs` |
| `ArrowIpc` | `arrow` | `io/ipc.rs` |
`write_chunk` creates parent dirs then matches the `FileType` to the codec's `write`; each
codec has a mirror `read`. Value types the schema may carry are `DataType` (`src/types.rs:8`)
and edge topology is `AdjListType` (`src/types.rs:73`).
## f2 — metadata / info-files
The `gar/v1` graph is described by YAML info-files, modelled in `src/metadata/`:
| struct | file | describes |
|---|---|---|
| `GraphInfo` | `src/metadata/graph_info.rs:10` | `*.graph.yml` — the graph, its vertex/edge lists, `prefix` (`#[serde(default)]`) |
| `VertexInfo` | `src/metadata/vertex_info.rs:13` | `*.vertex.yml` — label, `chunk_size`, property groups; `validate()` (`:92`) |
| `EdgeInfo` | `src/metadata/edge_info.rs:32` | `*.edge.yml` — src/type/dst triple, adjacency lists |
| `AdjListInfo` | `src/metadata/edge_info.rs:13` | one adjacency list (ordered/unordered, by-src/by-dst) |
| `PropertyGroup` | `src/metadata/property.rs:43` | a column group → its own chunk dir |
| `Property` | `src/metadata/property.rs:8` | one column: name, `DataType`, primary flag |
A vertex label's columns are partitioned into **property groups**, each group stored as its
own `vertex/<label>/<group>/chunk0..chunkN` run; edge groups are partitioned per source/dest
chunk. This is the layout `f5` validates and `f3` reads.
## f3 — readers & builders
**Builders** (write side, `src/writer/`) accumulate rows then dump chunks:
- `VerticesBuilder::new(info, base_dir)` (`src/writer/vertex_writer.rs:66`) →
`add_vertex(Vertex)` (`:74`) → `dump() -> u64` (`:79`, returns vertex count). A `Vertex`
(`:24`) carries `PropertyValue`s (`src/writer/vertex_writer.rs:30`) built with
`add_property` (`:48`).
- `EdgesBuilder::new(...)` (`src/writer/edge_writer.rs:56`) → `add_edge(Edge)` (`:71`) →
`dump()` (`:76`). `Edge::new(src, dst)` (`:27`) keys endpoints by `_gar_id`.
**Readers** (read side, `src/reader/`) stream a graph's chunks back to Arrow:
- `VertexReader::new(info, base_dir, vertex_count)` (`src/reader/vertex_reader.rs:19`);
`read_property_group(group)` (`:36`), `read_merged()` (`:64`) joins all groups row-aligned,
`read_merged_groups(groups)` (`:70`), `read_by_property(name)` (`:77`).
- `EdgeReader::new(...)` (`src/reader/edge_reader.rs:23`); `read_adj_list(...)` (`:50`),
`read_offset(...)` (`:78`), `read_property_group(...)` (`:98`),
`iter_vertex_chunks(...)` (`:124`).
Row-aligned joining of property groups is exposed standalone as
`merge_property_groups(&[Vec<RecordBatch>]) -> RecordBatch` (`src/merge.rs:21`).
## f4 — ChunkStore over object_store
`ChunkStore` (`src/storage.rs:246`) is the storage seam: one API over any `object_store`
backend. Constructors: `new(store, prefix)` (`:254`), `local(root)` (`:372`),
`from_url(url)` (`:284`), `s3_compatible(...)` (`:387`). Async chunk ops:
`read_chunk(rel, &FileType)` (`:442`), `write_chunk(rel, &[RecordBatch], &FileType)` (`:452`),
`get_bytes` (`:463`), `put_bytes` (`:469`). Free-standing codecs
`read_chunk_bytes`/`write_chunk_bytes` (`:166`,`:199`) do the in-memory encode/decode.
**Retry** is `object_store`'s native backoff, fronted by the typed `RetryOptions`
(`src/storage.rs:51`: `max_retries`, `init_backoff`, `max_backoff`, `retry_timeout`;
`Default` matches the library's 10 retries / 100 ms→15 s / 3-min budget; `RetryOptions::none()`
(`:78`) opts out; `From<RetryOptions> for RetryConfig` at `:97`). `from_url` /
`s3_compatible` use the default; `from_url_with_retry` (`:297`) and
`s3_compatible_with_retry` (`:407`) override it — `from_url_with_retry` dispatches on
`ObjectStoreScheme` and applies `with_retry` per-scheme (it can't thread `RetryConfig`
through `parse_url_opts`).
**Multipart** upload is fronted by `MultipartOptions` (`src/storage.rs:136`: `threshold`
default 16 MiB, `part_size` default 8 MiB). Attached builder-style via
`with_multipart(opts)` (`:267`), read back by `multipart_options()` (`:273`). A write whose
encoded length exceeds `threshold` is sliced into `part_size` parts and uploaded with
`put_multipart` (sequential, bounded memory, `abort()` on error); at or under it a single
`put` runs unchanged.
## f5 — validation
`validate::validate_graph_dir(graph_yml)` (`src/validate.rs:127`) walks a written graph and
returns `Vec<ValidationIssue>` (`src/validate.rs:41`; empty ⇒ conformant). Each issue carries
a `Severity::{Error,Warning}` (`src/validate.rs:32`), a message, and the offending path. It is
read-only and never touches the network. `validate_graph_dir_with(.., ValidateOptions)`
(`:132`) relaxes checks via `ValidateOptions` (`:91`: `extensionless_chunks`, `require_chunks`;
`::corpus()` preset for the upstream extensionless corpus). Info-struct entry points:
`GraphInfo::validate(base)` (`src/validate.rs:571`) and `EdgeInfo::validate_dir(base)` /
`VertexInfo::validate_dir(base)` (`:578`,`:585`); the split checkers are
`validate_graph_info` (`:152`), `validate_vertex_info` (`:197`), `validate_edge_info` (`:247`).
Rules checked: **info-file schema** — names/types/triples non-empty, `version == gar/v1`,
`chunk_size`/`src_chunk_size`/`dst_chunk_size > 0`, ≥ 1 property group (vertex) / adjacency
list (edge), valid adj-list type, each property in exactly one group, ≤ 1 primary property.
**Chunk layout** — every declared group has its `vertex/<label>/<group>/` dir with a gapless
`chunk0..chunkN` run; each edge adjacency list has `<adj>/adj_list/partK/chunkN`, ordered
lists have an `<adj>/offset/` dir (unordered must not), edge property groups partition the
same way. Partitions may be **sparse** (a vertex chunk with no incident edges has no `partK`);
numbering *within* a partition must be gapless. Stray/extra files are warnings, not errors.