Wadjet supports distributed query execution across multiple worker nodes, coordinated via NATS JetStream and backed by shared S3-compatible object storage.
graph TD
CL["Clients"]
CO["Coordinator<br/><sub>HTTP + gRPC API, Query Planner,<br/>Embedded NATS, Result Merger</sub>"]
W1["Worker 1<br/><sub>Executor, LRU Cache</sub>"]
W2["Worker 2<br/><sub>Executor, LRU Cache</sub>"]
W3["Worker 3<br/><sub>Executor, LRU Cache</sub>"]
S3["S3 / MinIO<br/>(shared)"]
CL --> CO
CO -- "NATS JetStream" --> W1
CO -- "NATS JetStream" --> W2
CO -- "NATS JetStream" --> W3
W1 --> S3
W2 --> S3
W3 --> S3
The coordinator is the single entry point for queries:
- Receives SQL via HTTP or gRPC API
- Parses and plans the query
- Routes the query: under
--local-fastpath-bytesof post-pruning scan bytes (64 MiB default) it executes in-process on the coordinator; otherwise it dispatches a multi-stage DAG - For the DAG: emits stages whose outputs materialize to object storage, choosing broadcast, probe-split or hash-shuffle per join
- For federated queries, expands scan stages per-cluster via
ExpandFederatedScans - Waits for result notifications from workers (request/reply with ACK)
- Merges partial results: re-aggregation, sort/limit, DISTINCT dedup
The coordinator also embeds a NATS server (no external NATS required).
Workers are stateless compute nodes that:
- Connect to the coordinator's NATS server
- Subscribe to the
tasksJetStream consumer, filtered by cluster ID - Check if query is still active before executing (prevents stale task waste)
- Execute pipeline tasks: full SQL on the worker, with optional probe-split file filter
- Read data from S3 (checking LRU cache first)
- Publish result notifications with request/reply ACK (results under 512 KB are inlined)
- Send heartbeats every 10 seconds (with cluster ID and active task IDs for liveness tracking)
NATS provides the messaging backbone:
tasksstream — Durable stream for task distribution. Tasks are pulled by workers via a JetStream consumer with at-least-once delivery.- Result subjects — Per-query subjects for result notifications. The coordinator subscribes to these to collect task completions.
- Heartbeat subjects — Workers publish periodic heartbeats; the coordinator monitors them and reaps dead workers.
./wadjetd serve \
--mode coordinator \
--http-addr :8080 \
--grpc-addr :9090 \
--nats-port 4222 \
--cluster-id central \
--endpoint minio.internal:9000 \
--access-key $S3_ACCESS_KEY \
--secret-key $S3_SECRET_KEY \
--bucket wadjet \
--config wadjet.yamlOn each worker node:
./wadjetd serve \
--mode worker \
--nats-url nats://coordinator.internal:4222 \
--cluster-id central \
--memory-budget 268435456 \
--spill-dir /var/wadjet/spill \
--result-store 134217728 \
--endpoint minio.internal:9000 \
--access-key $S3_ACCESS_KEY \
--secret-key $S3_SECRET_KEY \
--bucket wadjetWorkers automatically register with the coordinator and begin pulling tasks. The --cluster-id determines which tasks a worker pulls — it only processes tasks targeted at its cluster. See Performance Tuning for guidance on --memory-budget, --spill-dir, and --result-store sizing.
The repository ships no Dockerfile and publishes no container image. Build and push your own from
./cmd/wadjetd, withwadjetdas the image entrypoint, then replacebuild: .and theimage:values below with your registry path. See LICENSING.md for the binary split.
version: "3.8"
services:
minio:
image: minio/minio
command: server /data --console-address ":9001"
ports:
- "9000:9000"
- "9001:9001"
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
volumes:
- minio-data:/data
coordinator:
build: .
command: >
serve --mode coordinator
--http-addr :8080 --grpc-addr :9090 --nats-port 4222
--cluster-id central
--endpoint minio:9000
--access-key minioadmin --secret-key minioadmin
--bucket wadjet
ports:
- "8080:8080"
- "9090:9090"
- "4222:4222"
depends_on:
- minio
worker-1:
build: .
command: >
serve --mode worker
--nats-url nats://coordinator:4222
--cluster-id central
--memory-budget 268435456
--result-store 134217728
--spill-dir /tmp/wadjet-spill
--endpoint minio:9000
--access-key minioadmin --secret-key minioadmin
--bucket wadjet
depends_on:
- coordinator
worker-2:
build: .
command: >
serve --mode worker
--nats-url nats://coordinator:4222
--cluster-id central
--memory-budget 268435456
--result-store 134217728
--spill-dir /tmp/wadjet-spill
--endpoint minio:9000
--access-key minioadmin --secret-key minioadmin
--bucket wadjet
depends_on:
- coordinator
volumes:
minio-data:docker-compose up -d
# Scale workers dynamically
docker-compose up -d --scale worker-1=5# coordinator-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: wadjet-coordinator
spec:
replicas: 1 # Single coordinator
selector:
matchLabels:
app: wadjet-coordinator
template:
metadata:
labels:
app: wadjet-coordinator
spec:
containers:
- name: coordinator
image: ghcr.io/derekmwright/wadjet:latest
args:
- serve
- --mode=coordinator
- --http-addr=:8080
- --grpc-addr=:9090
- --nats-port=4222
- --cluster-id=central
- --endpoint=$(S3_ENDPOINT)
- --access-key=$(S3_ACCESS_KEY)
- --secret-key=$(S3_SECRET_KEY)
- --bucket=wadjet
ports:
- containerPort: 8080
name: http
- containerPort: 9090
name: grpc
- containerPort: 4222
name: nats
env:
- name: S3_ENDPOINT
valueFrom:
configMapKeyRef:
name: wadjet-config
key: s3-endpoint
- name: S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: wadjet-s3
key: access-key
- name: S3_SECRET_KEY
valueFrom:
secretKeyRef:
name: wadjet-s3
key: secret-key
---
# worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: wadjet-worker
spec:
replicas: 3 # Scale as needed
selector:
matchLabels:
app: wadjet-worker
template:
metadata:
labels:
app: wadjet-worker
spec:
containers:
- name: worker
image: ghcr.io/derekmwright/wadjet:latest
args:
- serve
- --mode=worker
- --nats-url=nats://wadjet-coordinator:4222
- --cluster-id=central
- --memory-budget=268435456
- --result-store=134217728
- --spill-dir=/tmp/wadjet-spill
- --endpoint=$(S3_ENDPOINT)
- --access-key=$(S3_ACCESS_KEY)
- --secret-key=$(S3_SECRET_KEY)
- --bucket=wadjet
resources:
requests:
cpu: "2"
memory: 2Gi
limits:
cpu: "4"
memory: 4Gi
env:
- name: S3_ENDPOINT
valueFrom:
configMapKeyRef:
name: wadjet-config
key: s3-endpoint
- name: S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: wadjet-s3
key: access-key
- name: S3_SECRET_KEY
valueFrom:
secretKeyRef:
name: wadjet-s3
key: secret-key
---
# coordinator-service.yaml
apiVersion: v1
kind: Service
metadata:
name: wadjet-coordinator
spec:
selector:
app: wadjet-coordinator
ports:
- name: http
port: 8080
- name: grpc
port: 9090
- name: nats
port: 4222graph TD
C["1. Client sends SQL"]
P["2. Parse → Logical Plan → Physical Plan"]
F["3. ExpandFederatedScans<br/>(split scans per cluster)"]
S1A["Stage 1a: Scan (central)<br/><sub>Task 1a: date=2026-03-14 → Central Worker 1<br/>Task 1b: date=2026-03-15 → Central Worker 2</sub>"]
S1B["Stage 1b: Scan (site-east)<br/><sub>Task 1c: date=2026-03-15 → Site-East Worker 1</sub>"]
S2["Stage 2: Aggregate<br/><sub>Task 2a: chunk 1 → Central Worker 1<br/>Task 2b: chunk 2 → Central Worker 2</sub>"]
S3["Stage 3: Final Merge<br/><sub>Task 3a: merge + sort + limit → Central Worker 1</sub>"]
R["6. Coordinator returns result to client"]
C --> P --> F
F --> S1A
F --> S1B
S1A --> S2
S1B --> S2
S2 --> S3 --> R
| Type | Description | Parallelism |
|---|---|---|
scan |
Read Parquet files, apply filters | Per-partition or per-file |
aggregate |
Group-by with aggregate functions | Per-chunk of scan results |
join |
Hash join between two datasets | Per-partition of probe side |
sort |
Sort intermediate results | Single task (pipeline breaker) |
window |
Window functions (ROW_NUMBER, RANK, etc.) | Per-partition |
shuffle |
Hash-partition rows into N .wshf exchange files |
Per input file slice |
All task types respect per-task memory budgets. Every pipeline breaker — hash join, hash aggregate, sort and window — spills to disk when its memory budget is exceeded.
Each worker defaults to processing 4 tasks concurrently. For CPU-heavy workloads (complex aggregates), reduce this. For I/O-heavy workloads (large scans from S3), increase it.
Workers can be configured with a per-task memory budget (--memory-budget). When a pipeline breaker (HashJoin, HashAggregate, Sort, Window) exceeds the budget, it spills intermediate state to disk instead of growing memory unboundedly, so a worker's peak memory tracks its budget and caches rather than the size of its input. What a worker actually needs is therefore a sizing question, not a fixed number — see Performance Tuning for the profiles, and the benchmarks index for the process footprints that have been measured (an idle worker is ~33 MiB RSS; the banked SF100 runs use 32 GB workers).
Set --spill-dir to a fast local disk (SSD/NVMe preferred) for best spill performance. Spill files are cleaned up automatically after each task completes.
Two workers on one host may share one --spill-dir. Each worker owns a private subdirectory under it, and a task's scratch is placed under that by query and by task, so two workers running the same task ID — which happens whenever they serve the same query — never write into or delete each other's files.
Enable --result-store to cache intermediate stage results in memory, avoiding S3 round-trips between stages that execute on the same worker. This is the single biggest optimization for multi-stage query latency.
Results below 512 KB are always passed inline via NATS messages regardless of result store configuration.
Workers cache recently-read Parquet file data in an LRU cache (--cache-bytes; the default of 0 auto-sizes it to 10% of the Go memory limit, ~7.5% of detected memory — the Go limit is itself only 75% of what's detected). This benefits:
- Repeated queries against the same time range
- Join probes that reference the same build-side data
- Interactive exploration (user refining queries)
| Workload | CPU | Memory | Workers | Recommended Flags |
|---|---|---|---|---|
| Small (< 10 GB/day) | 2 cores | 1 GB | 1–2 | --memory-budget 67108864 --result-store 16777216 |
| Medium (10–100 GB/day) | 4 cores | 2–4 GB | 3–5 | --memory-budget 268435456 --result-store 134217728 |
| Large (100+ GB/day) | 8 cores | 8+ GB | 5–10+ | --memory-budget 1073741824 --result-store 1073741824 |
For detailed tuning guidance, see Performance Tuning.
- Worker crash: The coordinator detects missed heartbeats and reaps the worker. Tasks assigned to the dead worker time out and are redelivered by JetStream to surviving workers.
- Coordinator crash: All in-flight queries fail. Restart the coordinator; workers reconnect automatically. No data is lost (all state is on S3).
- S3 unavailability: Queries and ingestion fail with storage errors. Resume when S3 recovers. Catalog metadata in NATS KV is unaffected.
- NATS partition: Workers lose connection and stop receiving tasks. They reconnect automatically when the partition heals.
Federation allows Wadjet to query data spread across multiple clusters — for example, a central data center and multiple regional sites.
graph TD
CO["Coordinator (central)<br/><sub>Embedded NATS</sub>"]
W1["Worker 1<br/>(central)"]
W2["Worker 2<br/>(central)"]
RN["Remote NATS (site-east)<br/><sub>Leaf Node</sub>"]
W3["Worker 3<br/>(site-east)"]
CO --> W1
CO --> W2
CO -- "Leaf Node Connection" --> RN
RN --> W3
- Each cluster has a unique
--cluster-id(e.g.,central,site-east) - Workers subscribe to cluster-scoped NATS subjects:
wadjet.tasks.<cluster-id>.> - The catalog stores per-cluster partition manifests
- When a table exists on multiple clusters, the coordinator's
ExpandFederatedScanssplits scan stages per-cluster - Each cluster's workers scan their local data; results flow back for centralized aggregation
Central coordinator:
./wadjetd serve --mode coordinator \
--cluster-id central \
--nats-port 4222 \
--endpoint minio-central:9000 \
--access-key $S3_KEY --secret-key $S3_SECRET --bucket wadjetCentral workers:
./wadjetd serve --mode worker \
--cluster-id central \
--nats-url nats://coordinator:4222 \
--memory-budget 268435456 --result-store 134217728 \
--endpoint minio-central:9000 \
--access-key $S3_KEY --secret-key $S3_SECRET --bucket wadjetRemote site workers (with leaf node connection):
./wadjetd serve --mode worker \
--cluster-id site-east \
--nats-url nats://coordinator:4222 \
--leaf-remote nats://coordinator:4222 \
--memory-budget 67108864 --result-store 16777216 \
--endpoint minio-east:9000 \
--access-key $S3_KEY --secret-key $S3_SECRET --bucket wadjetTasks are published to cluster-scoped subjects:
wadjet.tasks.<cluster-id>.<type>.<query-id>.<stage-id>
Workers subscribe to their cluster's wildcard filter:
wadjet.tasks.<cluster-id>.>
This ensures workers only pull tasks for data in their local object store, avoiding cross-cluster data transfers for scan operations.
| Aspect | Standalone | Distributed | Federated |
|---|---|---|---|
| Processes | 1 | 1 coordinator + N workers | 1 coordinator + N workers across clusters |
| NATS | Embedded (in-process) | Embedded in coordinator | Coordinator + leaf nodes |
| Query parallelism | Single-process pipeline | Multi-node task distribution | Multi-cluster task routing |
| Best for | Development, small datasets | Production, large datasets | Geo-distributed data |
| Scaling | Vertical only | Horizontal (add workers) | Horizontal + geographic |