Gunnar

nordisk/skade

.nornir/serve-object-store-for-idiots.md

# "My table is in MinIO and Doris can't see it" — the idiot guide

You wrote a big table with skade. The Parquet is sitting in MinIO. Doris says
there are no tables. Here is what is going on and what to type.

## The mental model: two separate things

A skade warehouse is **two** things, and people mix them up:

1. **The catalog** — one small file, `something.redb`, on a local disk. It is a
   phone book: "the table `main.personer` currently lives at *this* address".
2. **The data** — Parquet files, Avro manifests and JSON metadata. These can
   live anywhere: a local directory, or a bucket in MinIO.

`skade-katalog serve` is a little server that reads the phone book out loud over
HTTP, in the language every query engine speaks (Iceberg-REST). Doris calls it,
gets the address, and then goes and reads the Parquet from MinIO by itself.

So three things have to be true:

* serve must open **the actual phone book** you wrote into;
* serve must be able to **reach the addresses** in it (if they say `s3://…`,
  serve needs S3 credentials, because it has to read the metadata JSON);
* Doris must be able to reach the same bucket (it has its own credentials).

## Failure 1: "namespaces is empty"

```
$ curl localhost:8181/v1/namespaces
{"namespaces":[]}
```

This almost always means **serve opened the wrong file**. redb does not complain
about a missing database — it quietly creates a new, empty one. An empty phone
book has no names in it.

Check the file before you start serve:

```sh
ls -l /path/to/your.redb        # should exist, and be megabytes, not zero
```

The classic way to get this wrong is the writer's argument order:

```rust
// RIGHT — the local redb file comes FIRST
Warehouse::open_with_store("/var/lib/skade/wh.redb", "s3://red/skade-warehouse", store)

// WRONG — silently writes the redb to ./s3:/red/skade-warehouse
Warehouse::open_with_store("s3://red/skade-warehouse", "/var/lib/skade/wh.redb", store)
```

If you did the wrong one, your redb is in a weird directory called `s3:` under
wherever you ran the program. Look for it there.

## Failure 2: "the tables are listed but I can't open one"

```
$ curl localhost:8181/v1/namespaces/main/tables/personer
{"error":{"message":"Failed to read file s3://red/…/00001-….metadata.json:
                     No such file or directory"}}
```

The phone book is fine. serve just can't *reach* the address: it went looking on
its own hard disk for something that lives in MinIO. Give it the bucket:

```sh
export MINIO_ACCESS_KEY=…
export MINIO_SECRET_KEY=…

skade-katalog serve \
  --db            /var/lib/skade/wh.redb \
  --warehouse-uri s3://red/skade-warehouse \
  --s3-bucket     red \
  --s3-endpoint   https://172.16.12.9:9000 \
  --rest          0.0.0.0:8181
```

Two catches:

* the binary must have been **built with the feature**:
  `cargo build --release --features object-store` (a plain build tells you so).
* `--warehouse-uri` must be *exactly* the URI the writer used. Not a similar
  one. It is how the addresses were written down.

Newer builds will not even start in that state — if you give a `s3://…`
`--warehouse-uri` and forget `--s3-bucket`, serve says so and exits instead of
coming up and listing tables it cannot open. That is on purpose: a server that
half-works is much harder to debug than one that refuses.

## Failure 3: "403 InvalidAccessKeyId"

Your keys aren't in the environment, or they're the wrong keys. serve never
takes credentials on the command line on purpose — anyone running `ps` would see
them. Export them and restart.

If instead you see

```
Failed to parse json string, source: expected value at line 1 column 1
```

that is the **same problem on an old build**. S3 answered "no you may not", in
XML, and skade handed that XML to the JSON parser as if it were your table's
metadata. Current builds say `s3 get <key>: HTTP 403: …AccessDenied…` instead.
Upgrade before you spend an afternoon on it.

## Am I sure it works? Ask the machine

Rather than curl three URLs and squint, run the shipped check. It writes a tiny
table into your bucket, serves it, reads it back over REST and tidies up:

```sh
export MINIO_ACCESS_KEY=… MINIO_SECRET_KEY=…
export SKADE_LIVE_S3_ENDPOINT=https://172.16.12.9:9000 SKADE_LIVE_S3_BUCKET=red
cargo run --release --features object-store --example live_object_store
```

The last line is `live_object_store: PASS` or it is not.

## Then, in Doris

```sql
CREATE CATALOG skade PROPERTIES (
  'type'                 = 'iceberg',
  'iceberg.catalog.type' = 'rest',
  'uri'                  = 'http://<host-running-serve>:8181',
  'warehouse'            = 's3://red/skade-warehouse',
  's3.endpoint'          = 'https://172.16.12.9:9000',
  's3.access_key'        = '…',
  's3.secret_key'        = '…',
  'use_path_style'       = 'true'
);

SWITCH skade;
SHOW DATABASES;          -- expect: main
SELECT count(*) FROM main.personer;
```

Doris needs the S3 keys **as well**, because it reads the Parquet itself. serve
only hands out addresses.

## Two things people expect that aren't true

* **`iceberg.catalog.type = hadoop` will not work on a skade warehouse.** The
  hadoop catalog finds tables by looking for `v3.metadata.json` plus a
  `version-hint.text` file. skade writes `00001-<uuid>.metadata.json` and keeps
  the pointer in the redb catalog — which is the whole point of having a
  catalog. Doris will see the namespace directory and no tables. Use `rest`.
* **Only one process at a time may own the redb.** That is deliberate: serve is
  the single writer, and everything else talks to serve. Stop the writer before
  starting serve.