<!-- nornir:class: idiot -->
# skade for absolute beginners
No prior data-lake knowledge assumed. By the end you'll know *what* skade is,
*why* it exists, and you'll have run your first query. When you want the real
API, go to [`tutorial.md`](tutorial.md); for how it works inside, [`design.md`](design.md).
## The one-sentence version
**skade turns a single folder on your disk into a real, transactional,
queryable analytics database — no server to run, no JVM, no Postgres.** You hand
it columns of data; you get SQL back.
## The problem it solves (in plain words)
Imagine you have a lot of tabular data — millions of rows, growing over time —
and you want to:
1. **append** to it safely (no half-written messes if the program crashes),
2. **query** it with SQL,
3. **keep its history** so you can ask "what did this table look like last
Tuesday?",
4. and do all of that **without standing up a database server**.
The industry's answer to "big tables in files, with history and transactions" is
a format called **Apache Iceberg**. Iceberg is great, but the usual way to run it
drags in a *catalog server* (often a JVM service backed by Postgres) whose only
job is to remember "which file is the current version of each table." That's a
lot of moving parts for something that could be a single file.
**skade is that catalog — and a writer/reader — as an embedded library.** The
"which file is current" bookkeeping lives in one small transactional file
(`catalog.redb`) right next to your data. Your whole lakehouse is one directory.
## An analogy
Think of a **library**:
- The **books** are your data files (Parquet — compressed columns of rows).
- The **card catalog** is what tells you *which* book is the current edition of
each title. In a normal Iceberg setup the card catalog is a whole separate
building with staff (a server). **skade keeps the card catalog as one tidy
index card box on the same shelf as the books** — instant to consult, and it
never disagrees with itself because every update is atomic.
- **Editions = snapshots.** Every time you add data, Iceberg writes a *new*
edition and updates the catalog to point at it. The old editions stay on the
shelf, so you can always go back and read last week's edition (*time travel*).
## What you actually get
- **One directory = one lakehouse.** Copy it, back it up, `rsync` it — done.
- **ACID.** A write either fully happens or doesn't; a crash never leaves a
table half-updated.
- **SQL.** `SELECT … FROM my_table JOIN other …` over your data.
- **Time travel.** Read any past snapshot.
- **Fast.** Reading "the current version of table X" is a ~100-nanosecond
in-memory lookup, not a network round-trip.
- **No server, no JVM.** It's a Rust crate you link into your program.
## Your first five minutes
Add it to a Rust project:
```toml
[dependencies]
skade = { version = "0.4", features = ["sql"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
anyhow = "1"
```
Open a folder, make a table, put three rows in, count them with SQL:
```rust,no_run
use std::sync::Arc;
use skade::iceberg::spec::{NestedField, PrimitiveType, Schema, Type};
use skade::arrow_array::{RecordBatch, Int64Array};
use skade::arrow_schema::{Field, Schema as ArrowSchema, DataType};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// 1. The folder "lake" becomes your whole lakehouse.
let wh = skade::open("./lake").await?;
// 2. Say what columns the table has (the "schema"). Each column needs an id.
let schema = Schema::builder()
.with_schema_id(0)
.with_fields(vec![
NestedField::required(1, "id", Type::Primitive(PrimitiveType::Long)).into(),
])
.build()?;
let mut t = wh.create_table("greetings", &schema).await?;
// 3. Hand it some data (Arrow = the in-memory columnar format).
let arrow = Arc::new(ArrowSchema::new(vec![Field::new("id", DataType::Int64, false)]));
let batch = RecordBatch::try_new(arrow, vec![Arc::new(Int64Array::from(vec![1, 2, 3]))])?;
t.append(&[batch]).await?; // this is one durable commit / snapshot
// 4. Ask a question.
let answer = wh.sql("SELECT count(*) AS n FROM greetings").await?;
println!("{answer:?}"); // n = 3
Ok(())
}
```
Run it. Look in `./lake`: there's `catalog.redb` (the index-card box) and a
`greetings/` folder with `.parquet` data and `.metadata.json` editions. That's a
production-shaped Iceberg lakehouse you can hand to Spark, DuckDB, or anything
else that reads Iceberg.
## Tiny glossary
| word | plain meaning |
|---|---|
| **Iceberg** | an open standard for "big tables stored as files, with transactions + history" |
| **catalog** | the bookkeeping that maps each table name → its *current* metadata file |
| **snapshot** | one version of a table; every write makes a new one; old ones stay (time travel) |
| **metadata file** | a small JSON file describing a table version (which data files belong to it) |
| **Parquet** | the compressed columnar file format the actual rows live in |
| **Arrow / RecordBatch** | the in-memory columnar format you build to hand data to skade |
| **redb** | the tiny embedded transactional key-value store skade keeps the catalog in |
| **ACID** | writes are all-or-nothing and survive crashes |
| **commit** | one atomic write that advances a table to a new snapshot |
| **namespace** | a folder/grouping for tables (skade's default is `main`; `logs.events` puts `events` in namespace `logs`) |
## Two layers (so the docs make sense)
skade is two crates, and you'll see both names:
- **`skade`** — the friendly part you just used (`open`, `create_table`,
`append`, `sql`). Start here.
- **`skade-katalog`** — the bare catalog engine underneath (implements the
Iceberg `Catalog` trait over redb). You only touch it directly for advanced
things like custom storage backends or multi-table atomic releases.
## When skade is the right tool — and when it isn't
**Great fit:** one program (or one machine) owning a dataset; embedding a
lakehouse in an app or CLI; fast local analytics; CI pipelines that publish
result tables; anywhere you'd otherwise resent running a catalog server.
**Not a fit:** many separate processes/hosts writing the *same* catalog at once.
redb is a single-file lock — one writer process at a time. (It's a library, not a
shared server. For that, you'd want a REST catalog like Nessie/Polaris — which
skade is benchmarked against, and beats on single-node latency.)
## Where to go next
- Hands-on, all the real API + bulk ingest + time travel → [`tutorial.md`](tutorial.md)
- How the speed actually works (the cache layers, the Ragnar search tree) →
[`design.md`](design.md), then [`plan.md`](plan.md)
- What's planned / not built yet → [`TODO.md`](TODO.md)