<!-- nornir:class: design -->
# `skade::connect` — one trait for opening a lake's catalog
> **Design (chosen path only).** skade-flight, knut-bifrost and korp all need
> "open that lake's Iceberg catalog". One trait in the skade LIBRARY answers it,
> so no consumer grows its own Nessie or S3 client.
## f0 — the whole idea
```rust
#[async_trait]
pub trait CatalogConnector: Send + Sync + Debug {
fn kind(&self) -> &'static str; // "skade-local" | "nessie-v2" | "iceberg-rest" | "polaris" | "skade-grpc" | …
fn describe(&self) -> String; // endpoint / ref / warehouse — never a secret
fn read_only(&self) -> bool;
async fn open(&self) -> skade::Result<Arc<dyn iceberg::Catalog>>;
}
```
A connector is a plain config struct; `open` yields an `iceberg::Catalog`, which
is all a reader needs — skade's read path (`read_all`, `read_delta`,
`read_changelog`, …) works on any loaded `iceberg::table::Table`.
| connector | kind | feature | read-only |
|---|---|---|---|
| `SkadeLocal { dir }` | `skade-local` | always | no |
| `SkadeObjectStore { db_path, warehouse_uri, store }` | `skade-object-store` | always (backend features `rustfs`/`s3`/`aws-s3`) | no |
| `NessieV2 { uri, reference, token, tls, s3 }` | `nessie-v2` | `nessie` | **always** |
| `IcebergRest { uri, warehouse, prefix_or_ref, token, credential, scope, oauth2_server_uri, headers, vended_credentials, tls, s3, read_only, sigv4 }` | `iceberg-rest` | `rest-catalog` | configurable |
| `AIStor { s3, warehouse, read_only }` | `aistor` | `catalog-aistor` | configurable |
| `Polaris { uri, catalog, credential, scope, token, tls, vended_credentials, s3, read_only }` | `polaris` | `catalog-polaris` | configurable |
| `UnityCatalog { uri, catalog, token, tls, vended_credentials, s3, read_only }` | `unity` | `catalog-unity` | by default (`new`) |
| `Gravitino { uri, warehouse, simple_user, credential, scope, token, tls, vended_credentials, s3, read_only }` | `gravitino` | `catalog-gravitino` | configurable |
| `SkadeGrpc { uri, tls, s3, read_only }` (`s3: None` → local fs) | `skade-grpc` | `grpc-catalog` | configurable |
`catalogs` turns on the four vendor presets. **One feature per connector**, so a
consumer (knut's `bifrost-falkordb-<vendor>` build rows) carries only the
catalog it talks to.
## NessieV2 — an existing Nessie lake over API v2
For a Nessie that must not be reconfigured and whose Iceberg-REST side is not set
up (Spark on Nessie's native API, table locations `s3a://…`):
| need | Nessie API v2 |
|---|---|
| resolve the reference | `GET {uri}/trees/{ref}` → `reference.hash` |
| namespaces, tables | `GET {uri}/trees/{ref}@{hash}/entries` (paged by `page-token`) |
| metadata location | `POST {uri}/trees/{ref}@{hash}/contents` `{"requestedKeys":[{"elements":[…]}]}` |
then the metadata JSON through the storage factory, table built `readonly(true)`.
- **One commit per catalog walk.** A top-level `list_namespaces` — the first call
of every DataFusion provider build — re-resolves the reference; later lists and
loads read `ref@hash`. Opening resolves once, so a bad URI/ref/token fails at open.
- **Keys in a POST body**, never a path segment: no escaping for dots or slashes.
- **Namespaces derived** from entries (`NAMESPACE` content + every table's parent).
- **Every write** → `ErrorKind::FeatureUnsupported`, message names read-only.
- **HTTP** is the ureq already in the tree (rustls), on `spawn_blocking`; bearer
token when set. No new dependency.
## IcebergRest — a lake behind an Iceberg REST catalog
Nessie's `/iceberg` endpoint once it is set up, Polaris, or a remote
`skade-katalog serve`: the vendored `iceberg-catalog-rest-arrow59` client (the one
skade's bench drives Nessie and Polaris with). Its reqwest client is built from
`IcebergRest.tls` (`use_preconfigured_tls` with the one rustls config) and handed
in with `RestCatalogBuilder::with_client`, so it survives the `/v1/config`
refresh. The branch rides the REST `prefix`; opening makes no network call;
`read_only: true` wraps the catalog so every write is `FeatureUnsupported`.
### OAuth2, headers, vended credentials
`credential` (`client_id:client_secret`) is exchanged for a token at
`oauth2_server_uri` (default `<uri>/v1/oauth/tokens`) with `scope`; `headers` go
on every catalog request. No secret is ever printed by `describe`.
`vended_credentials: true` sends `X-Iceberg-Access-Delegation:
vended-credentials` and reads each table's data with the `s3.*` keys the catalog
returns for it (`s3.access-key-id`, `s3.secret-access-key`, `s3.session-token`).
The vendored REST client picks the **longest matching** storage-credential prefix
over the config's keys; aegir signs the STS session token as
`x-amz-security-token`. `IcebergRest.s3` stays the base: endpoint, bucket, TLS,
and the fallback keys when the catalog vends none.
## Vendor presets — config over `IcebergRest`
Each preset is a plain struct whose `rest()` returns the `IcebergRest` it is;
`rest_preset!` implements `CatalogConnector` by delegating. No new dependency.
| preset | catalog URI | warehouse | auth | vended |
|---|---|---|---|---|
| `AIStor` | `<s3 endpoint>/_iceberg` | table bucket | SigV4 `s3tables`, the S3 keys | no |
| `Polaris` | `<root>/api/catalog` | Polaris catalog | OAuth2 client creds, scope `PRINCIPAL_ROLE:ALL` | on |
| `UnityCatalog` | `<root>/api/2.1/unity-catalog/iceberg` | Unity catalog | bearer token | on; read-only by default |
| `Gravitino` | `<root>/iceberg` | `warehouse` | none, `simple_user` (HTTP Basic), OAuth2 or bearer | configurable |
## SkadeGrpc — a remote skade-katalog over gRPC
`skade-katalog serve --grpc`, over skade's own `skade.katalog.v1` wire (tonic,
rustls). The client is `skade_katalog_grpc::GrpcCatalog` (crate feature `tls`;
`connect_lazy` makes no network call). Metadata travels on the wire; data is read
through the connector's own storage — a shared `file://` warehouse or S3/MinIO
over aegir. TLS is `ClientTls`, the same type as every other leg.
## Shared S3 wiring
One helper builds the iceberg storage from an `S3Config`: skade's **aegir** store
(`ObjectStoreConfig::Aegir`, SigV4 over ureq). `S3Config::strip_to_key` maps
`s3://`, `s3a://` and `s3n://` to the same key — Spark's Hadoop schemes read like
`s3://`. Credentials are the connector's unless `vended_credentials` is on (see
above); one bucket per connector.
## TLS — one type, every leg, mode by configuration
`skade::tls::ClientTls { ca_pem, client_cert_pem, client_key_pem }` (PEM file
paths, serde-defaulted) and ONE builder, `ClientTls::rustls_config` (pure rustls,
ring provider). Every leg takes it: `S3Config.tls`, `NessieV2.tls` and
`IcebergRest.tls` (and through it every preset) and `SkadeGrpc.tls`.
| endpoint | `tls` | mode |
|---|---|---|
| `http://` | any | plain |
| `https://` | `None` | TLS, webpki roots |
| `https://` | `ca_pem` | TLS, that CA |
| `https://` | `ca_pem` + cert + key | mutual TLS |
Production today — Nessie plain or TLS without a client certificate, MinIO
mutual TLS — is configuration, not code; either leg switches mode by settings.
A certificate without its key (or the reverse) is refused by name.
- **Why aegir for S3:** rust-s3 0.35 builds its own rustls 0.21 config with
`with_no_client_auth()` and takes none from outside, so it cannot present a
client certificate. aegir (skade's pure-Rust client) gained
`Client::with_tls_config(Arc<rustls::ClientConfig>)`. `RustS3Store` and
`AwsS3Store` refuse an `S3Config` with `tls`, by name.
- `S3Config` without `tls` (old JSON) still deserializes.
## Proven by
- `skade/tests/tls.rs` — the builder in all three https modes + refusals; a REAL
S3 round-trip over **mutual TLS** through `AegirS3Store` against an in-process
rustls server that requires a client certificate (CA-only and roots-only
clients refused); `NessieV2` over **TLS with a custom CA** (no client cert, no
auth) against an in-process rustls server; old `S3Config` JSON.
- `skade/tests/connect.rs` — a `Vec<Arc<dyn CatalogConnector>>` (local, object
store, Nessie mock) opened and walked through one code path; NessieV2 against an
in-process mock of Nessie 0.105's v2 shapes reading a real skade table whose
metadata and data sit under `s3a://` in a `MemoryStore` (pinning, paging, views
not tables, `TableNotFound`, write refusals, bearer token on every request).
- `object_store.rs` `s3_key_tests` — the three schemes map to one key.
- `connect/rest.rs` unit tests — REST props and redaction, the http client built
plain / webpki / custom CA and a half identity refused by name, read-only
refusals; `tests/connect.rs` walks an `IcebergRest` over a plain-http mock of
the Iceberg REST catalog through the same trait path as the others.
- `connect/{aistor,polaris,unity,gravitino}.rs` unit tests — each preset's URI,
warehouse, auth props and read-only default; `rest.rs` — OAuth2, headers and
vended-credentials props, and that `describe` redacts them; `object_store.rs`
`vended` tests — the per-table keys and session token reach aegir.
- vendored `iceberg-catalog-rest-arrow59` —
`the_longest_matching_storage_credential_wins_over_config`.
- `bench/aegir` `a_session_token_is_sent_and_signed`.
- `grpc/tests/client_roundtrip.rs` and `connect/grpc.rs` — `GrpcCatalog` and
`SkadeGrpc` against an in-process `skade-katalog` gRPC server.