rio-build

A Kubernetes-native distributed build backend for Nix.

rio-build speaks Nix's own wire protocol (ssh-ng://), so from Nix's perspective it is a remote store and builder — no client patches, no wrapper scripts. From Kubernetes' perspective it's an operator that schedules build DAGs across builder pods with chunk-deduplicated storage and cache-locality-aware placement.

nix build --store ssh-ng://rio .#your-package

Why?

Nix derivations form a pure, deterministic DAG — theoretically ideal for distributed execution. In practice, the ecosystem's distributed build story is thin: Hydra is monolithic, nix.buildMachines is round-robin with no global view, and binary caches solve distribution of results but not distribution of work.

rio-build fills the gap:

• Protocol-native — implements the Nix worker protocol directly; zero custom client tooling
• DAG-aware scheduling — critical-path priority, transfer-cost-weighted locality, SLA-driven per-derivation sizing
• Chunked CAS — FastCDC + BLAKE3 cross-build deduplication, inline fast-path for small NARs
• FUSE builder stores — lazy on-demand fetch from CAS, local SSD caching, per-build overlay isolation
• CA-ready — store schema and scheduler support content-addressed derivations from day one
• Observable — structured JSON logging, Prometheus metrics, OTLP tracing

What it is not

rio-build is a build execution backend, not a CI system. Out of scope: Nix evaluation (bring nix-eval-jobs), VCS webhooks, PR status checks, notifications, Hydra-style jobsets, Darwin workers, recursive Nix. See the design book for where each of these lives instead.

Architecture

      nix build --store ssh-ng://rio ...
                      │
                      ▼  ssh-ng (Nix worker protocol)
           ┌─────────────────────┐
           │     rio-gateway     │  SSH server (russh), protocol handler,
           │                     │  translates wire ops → gRPC
           └─────┬──────────┬────┘
           gRPC  │          │  gRPC
                 ▼          ▼
    ┌──────────────────┐ ┌────────────────────────┐
    │  rio-scheduler   │ │       rio-store        │
    │                  │ │                        │
    │  global DAG      │ │  chunked CAS (FastCDC) │
    │  critical-path   │ │  PostgreSQL metadata   │
    │  locality scoring│ │  S3 blobs              │
    │  PostgreSQL      │ │  binary-cache HTTP     │
    └────────┬─────────┘ └───────────┬────────────┘
             │ PullAssignment /      │
             │ ReportOutcome (gRPC)  │ gRPC + HTTP
             ▼                       ▼
   ┌──────────────────────────────────────────────┐
   │            rio-builder (K8s pods)            │
   │  FUSE /nix/store  •  per-build overlayfs     │
   │  synthetic SQLite db  •  nix-daemon sandbox  │
   │  cgroup v2 per-build resource tracking       │
   └──────────────────────────────────────────────┘

   rio-controller (K8s operator): Pool CRD, one-shot Job
   reconciliation, demand-driven autoscaling, drain-aware termination.

Status

Pre-1.0, under active development. Phase-gated — see the design book for the roadmap. Currently through Phase 4a (multi-tenancy: JWT auth, per-tenant quotas, per-tenant signing keys, query-level isolation; CA cutoff-compare; CLI tooling).

Multi-tenancy warning: Multi-tenant deployments with untrusted tenants are unsafe before Phase 5 (incomplete query-level isolation). Deploy single-tenant or trusted-tenant until then.

Crates

Crate	Role
rio-nix	Nix wire protocol, ATerm/NAR parsers, store path types (leaf — no rio-* deps)
rio-proto	gRPC service definitions (tonic)
rio-common	Shared config, observability, limits, Arc newtypes
rio-auth	JWT/HMAC token sign-verify, gRPC tenant interceptor
rio-gateway	SSH server + Nix worker protocol frontend
rio-scheduler	DAG scheduler, critical-path priority, SLA-driven per-drv sizing
rio-store	Chunked CAS, narinfo signing, binary-cache HTTP server
rio-builder	Build executor, FUSE store, overlayfs isolation, cgroup metering
rio-controller	Kubernetes operator (Pool CRD, autoscaler)
rio-crds	Kubernetes CRD types (kube-derive), shared between controller and CLI
rio-lease	Kubernetes Lease leader election (extracted from rio-scheduler)
rio-migrations	SQL migrations + embedded MIGRATOR + per-migration commentary
rio-cli	Admin CLI (trigger GC, tenant mgmt, backfill, dry-run introspection)
rio-test-support	Ephemeral PostgreSQL bootstrap, mock gRPC, wire helpers

Development

The dev environment is a Nix flake. direnv activates it automatically via .envrc.

# Enter the dev shell (nightly Rust by default — cargo-fuzz works)
nix develop

# Build / test / lint
cargo build
cargo nextest run
cargo clippy --all-targets -- --deny warnings

# Format (rustfmt + nixfmt + taplo)
treefmt

# Full CI (per-member clippy/doc/nextest + 2min fuzz + NixOS VM tests; needs KVM)
nix-fast-build --flake .#checks.x86_64-linux

The default dev shell uses nightly Rust so cargo fuzz works directly. CI checks use stable via rust-toolchain.toml — nightly-only code is rejected by nix flake check. Use nix develop .#stable for CI-parity local dev.

PostgreSQL integration tests bootstrap their own ephemeral server via rio-test-support — no manual setup needed.

Fuzzing

Fuzz targets live in per-crate fuzz/<crate> workspaces (separate Cargo.lock each):

cd fuzz/rio-nix && cargo fuzz run wire_primitives
# or via nix (one 2min check per fuzz target):
nix build .#checks.x86_64-linux.fuzz-wire_primitives

VM Tests

Per-scenario NixOS VM tests validate end-to-end behavior against a real nix client:

nix build .#checks.x86_64-linux.vm-protocol-warm-standalone   # gateway + builder + store, no k3s
nix build .#checks.x86_64-linux.vm-cli-k3s                    # k3s + controller + CRDs + CLI

Design Book

Full design docs are in docs/ (typst; build with nix build .#docs for HTML or .#docs-pdf). Start with:

• Introduction — goals, non-goals, landscape
• Architecture — component diagram, data flows
• Crate Structure — dependency graph, module layout
• Observability — metric naming, tracing conventions

License

See LICENSE.