Skip to content

Latest commit

 

History

History
469 lines (392 loc) · 15.8 KB

distributed.md

File metadata and controls

469 lines (392 loc) · 15.8 KB

Distributed Deployment

Wadjet supports distributed query execution across multiple worker nodes, coordinated via NATS JetStream and backed by shared S3-compatible object storage.

Architecture

graph TD
    CL["Clients"]
    CO["Coordinator<br/><sub>HTTP + gRPC API, Query Planner,<br/>Embedded NATS, Result Merger</sub>"]
    W1["Worker 1<br/><sub>Executor, LRU Cache</sub>"]
    W2["Worker 2<br/><sub>Executor, LRU Cache</sub>"]
    W3["Worker 3<br/><sub>Executor, LRU Cache</sub>"]
    S3["S3 / MinIO<br/>(shared)"]

    CL --> CO
    CO -- "NATS JetStream" --> W1
    CO -- "NATS JetStream" --> W2
    CO -- "NATS JetStream" --> W3
    W1 --> S3
    W2 --> S3
    W3 --> S3
Loading

Components

Coordinator

The coordinator is the single entry point for queries:

  1. Receives SQL via HTTP or gRPC API
  2. Parses and plans the query
  3. Routes the query: under --local-fastpath-bytes of post-pruning scan bytes (64 MiB default) it executes in-process on the coordinator; otherwise it dispatches a multi-stage DAG
  4. For the DAG: emits stages whose outputs materialize to object storage, choosing broadcast, probe-split or hash-shuffle per join
  5. For federated queries, expands scan stages per-cluster via ExpandFederatedScans
  6. Waits for result notifications from workers (request/reply with ACK)
  7. Merges partial results: re-aggregation, sort/limit, DISTINCT dedup

The coordinator also embeds a NATS server (no external NATS required).

Workers

Workers are stateless compute nodes that:

  1. Connect to the coordinator's NATS server
  2. Subscribe to the tasks JetStream consumer, filtered by cluster ID
  3. Check if query is still active before executing (prevents stale task waste)
  4. Execute pipeline tasks: full SQL on the worker, with optional probe-split file filter
  5. Read data from S3 (checking LRU cache first)
  6. Publish result notifications with request/reply ACK (results under 512 KB are inlined)
  7. Send heartbeats every 10 seconds (with cluster ID and active task IDs for liveness tracking)

NATS JetStream

NATS provides the messaging backbone:

  • tasks stream — Durable stream for task distribution. Tasks are pulled by workers via a JetStream consumer with at-least-once delivery.
  • Result subjects — Per-query subjects for result notifications. The coordinator subscribes to these to collect task completions.
  • Heartbeat subjects — Workers publish periodic heartbeats; the coordinator monitors them and reaps dead workers.

Deployment

Start the Coordinator

./wadjetd serve \
  --mode coordinator \
  --http-addr :8080 \
  --grpc-addr :9090 \
  --nats-port 4222 \
  --cluster-id central \
  --endpoint minio.internal:9000 \
  --access-key $S3_ACCESS_KEY \
  --secret-key $S3_SECRET_KEY \
  --bucket wadjet \
  --config wadjet.yaml

Start Workers

On each worker node:

./wadjetd serve \
  --mode worker \
  --nats-url nats://coordinator.internal:4222 \
  --cluster-id central \
  --memory-budget 268435456 \
  --spill-dir /var/wadjet/spill \
  --result-store 134217728 \
  --endpoint minio.internal:9000 \
  --access-key $S3_ACCESS_KEY \
  --secret-key $S3_SECRET_KEY \
  --bucket wadjet

Workers automatically register with the coordinator and begin pulling tasks. The --cluster-id determines which tasks a worker pulls — it only processes tasks targeted at its cluster. See Performance Tuning for guidance on --memory-budget, --spill-dir, and --result-store sizing.

Docker Compose Example

The repository ships no Dockerfile and publishes no container image. Build and push your own from ./cmd/wadjetd, with wadjetd as the image entrypoint, then replace build: . and the image: values below with your registry path. See LICENSING.md for the binary split.

version: "3.8"

services:
  minio:
    image: minio/minio
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - minio-data:/data

  coordinator:
    build: .
    command: >
      serve --mode coordinator
      --http-addr :8080 --grpc-addr :9090 --nats-port 4222
      --cluster-id central
      --endpoint minio:9000
      --access-key minioadmin --secret-key minioadmin
      --bucket wadjet
    ports:
      - "8080:8080"
      - "9090:9090"
      - "4222:4222"
    depends_on:
      - minio

  worker-1:
    build: .
    command: >
      serve --mode worker
      --nats-url nats://coordinator:4222
      --cluster-id central
      --memory-budget 268435456
      --result-store 134217728
      --spill-dir /tmp/wadjet-spill
      --endpoint minio:9000
      --access-key minioadmin --secret-key minioadmin
      --bucket wadjet
    depends_on:
      - coordinator

  worker-2:
    build: .
    command: >
      serve --mode worker
      --nats-url nats://coordinator:4222
      --cluster-id central
      --memory-budget 268435456
      --result-store 134217728
      --spill-dir /tmp/wadjet-spill
      --endpoint minio:9000
      --access-key minioadmin --secret-key minioadmin
      --bucket wadjet
    depends_on:
      - coordinator

volumes:
  minio-data:
docker-compose up -d
# Scale workers dynamically
docker-compose up -d --scale worker-1=5

Kubernetes Deployment

# coordinator-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wadjet-coordinator
spec:
  replicas: 1  # Single coordinator
  selector:
    matchLabels:
      app: wadjet-coordinator
  template:
    metadata:
      labels:
        app: wadjet-coordinator
    spec:
      containers:
        - name: coordinator
          image: ghcr.io/derekmwright/wadjet:latest
          args:
            - serve
            - --mode=coordinator
            - --http-addr=:8080
            - --grpc-addr=:9090
            - --nats-port=4222
            - --cluster-id=central
            - --endpoint=$(S3_ENDPOINT)
            - --access-key=$(S3_ACCESS_KEY)
            - --secret-key=$(S3_SECRET_KEY)
            - --bucket=wadjet
          ports:
            - containerPort: 8080
              name: http
            - containerPort: 9090
              name: grpc
            - containerPort: 4222
              name: nats
          env:
            - name: S3_ENDPOINT
              valueFrom:
                configMapKeyRef:
                  name: wadjet-config
                  key: s3-endpoint
            - name: S3_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: wadjet-s3
                  key: access-key
            - name: S3_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: wadjet-s3
                  key: secret-key
---
# worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wadjet-worker
spec:
  replicas: 3  # Scale as needed
  selector:
    matchLabels:
      app: wadjet-worker
  template:
    metadata:
      labels:
        app: wadjet-worker
    spec:
      containers:
        - name: worker
          image: ghcr.io/derekmwright/wadjet:latest
          args:
            - serve
            - --mode=worker
            - --nats-url=nats://wadjet-coordinator:4222
            - --cluster-id=central
            - --memory-budget=268435456
            - --result-store=134217728
            - --spill-dir=/tmp/wadjet-spill
            - --endpoint=$(S3_ENDPOINT)
            - --access-key=$(S3_ACCESS_KEY)
            - --secret-key=$(S3_SECRET_KEY)
            - --bucket=wadjet
          resources:
            requests:
              cpu: "2"
              memory: 2Gi
            limits:
              cpu: "4"
              memory: 4Gi
          env:
            - name: S3_ENDPOINT
              valueFrom:
                configMapKeyRef:
                  name: wadjet-config
                  key: s3-endpoint
            - name: S3_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: wadjet-s3
                  key: access-key
            - name: S3_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: wadjet-s3
                  key: secret-key
---
# coordinator-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: wadjet-coordinator
spec:
  selector:
    app: wadjet-coordinator
  ports:
    - name: http
      port: 8080
    - name: grpc
      port: 9090
    - name: nats
      port: 4222

Distributed Query Execution Flow

graph TD
    C["1. Client sends SQL"]
    P["2. Parse → Logical Plan → Physical Plan"]
    F["3. ExpandFederatedScans<br/>(split scans per cluster)"]

    S1A["Stage 1a: Scan (central)<br/><sub>Task 1a: date=2026-03-14 → Central Worker 1<br/>Task 1b: date=2026-03-15 → Central Worker 2</sub>"]
    S1B["Stage 1b: Scan (site-east)<br/><sub>Task 1c: date=2026-03-15 → Site-East Worker 1</sub>"]
    S2["Stage 2: Aggregate<br/><sub>Task 2a: chunk 1 → Central Worker 1<br/>Task 2b: chunk 2 → Central Worker 2</sub>"]
    S3["Stage 3: Final Merge<br/><sub>Task 3a: merge + sort + limit → Central Worker 1</sub>"]
    R["6. Coordinator returns result to client"]

    C --> P --> F
    F --> S1A
    F --> S1B
    S1A --> S2
    S1B --> S2
    S2 --> S3 --> R
Loading

Task Types

Type Description Parallelism
scan Read Parquet files, apply filters Per-partition or per-file
aggregate Group-by with aggregate functions Per-chunk of scan results
join Hash join between two datasets Per-partition of probe side
sort Sort intermediate results Single task (pipeline breaker)
window Window functions (ROW_NUMBER, RANK, etc.) Per-partition
shuffle Hash-partition rows into N .wshf exchange files Per input file slice

All task types respect per-task memory budgets. Every pipeline breaker — hash join, hash aggregate, sort and window — spills to disk when its memory budget is exceeded.

Worker Tuning

Concurrency

Each worker defaults to processing 4 tasks concurrently. For CPU-heavy workloads (complex aggregates), reduce this. For I/O-heavy workloads (large scans from S3), increase it.

Memory Budget and Spill-to-Disk

Workers can be configured with a per-task memory budget (--memory-budget). When a pipeline breaker (HashJoin, HashAggregate, Sort, Window) exceeds the budget, it spills intermediate state to disk instead of growing memory unboundedly, so a worker's peak memory tracks its budget and caches rather than the size of its input. What a worker actually needs is therefore a sizing question, not a fixed number — see Performance Tuning for the profiles, and the benchmarks index for the process footprints that have been measured (an idle worker is ~33 MiB RSS; the banked SF100 runs use 32 GB workers).

Set --spill-dir to a fast local disk (SSD/NVMe preferred) for best spill performance. Spill files are cleaned up automatically after each task completes.

Two workers on one host may share one --spill-dir. Each worker owns a private subdirectory under it, and a task's scratch is placed under that by query and by task, so two workers running the same task ID — which happens whenever they serve the same query — never write into or delete each other's files.

In-Memory Result Store

Enable --result-store to cache intermediate stage results in memory, avoiding S3 round-trips between stages that execute on the same worker. This is the single biggest optimization for multi-stage query latency.

Results below 512 KB are always passed inline via NATS messages regardless of result store configuration.

LRU Cache

Workers cache recently-read Parquet file data in an LRU cache (--cache-bytes; the default of 0 auto-sizes it to 10% of the Go memory limit, ~7.5% of detected memory — the Go limit is itself only 75% of what's detected). This benefits:

  • Repeated queries against the same time range
  • Join probes that reference the same build-side data
  • Interactive exploration (user refining queries)

Resource Sizing

Workload CPU Memory Workers Recommended Flags
Small (< 10 GB/day) 2 cores 1 GB 1–2 --memory-budget 67108864 --result-store 16777216
Medium (10–100 GB/day) 4 cores 2–4 GB 3–5 --memory-budget 268435456 --result-store 134217728
Large (100+ GB/day) 8 cores 8+ GB 5–10+ --memory-budget 1073741824 --result-store 1073741824

For detailed tuning guidance, see Performance Tuning.

Fault Tolerance

  • Worker crash: The coordinator detects missed heartbeats and reaps the worker. Tasks assigned to the dead worker time out and are redelivered by JetStream to surviving workers.
  • Coordinator crash: All in-flight queries fail. Restart the coordinator; workers reconnect automatically. No data is lost (all state is on S3).
  • S3 unavailability: Queries and ingestion fail with storage errors. Resume when S3 recovers. Catalog metadata in NATS KV is unaffected.
  • NATS partition: Workers lose connection and stop receiving tasks. They reconnect automatically when the partition heals.

Federation

Federation allows Wadjet to query data spread across multiple clusters — for example, a central data center and multiple regional sites.

Architecture

graph TD
    CO["Coordinator (central)<br/><sub>Embedded NATS</sub>"]
    W1["Worker 1<br/>(central)"]
    W2["Worker 2<br/>(central)"]
    RN["Remote NATS (site-east)<br/><sub>Leaf Node</sub>"]
    W3["Worker 3<br/>(site-east)"]

    CO --> W1
    CO --> W2
    CO -- "Leaf Node Connection" --> RN
    RN --> W3
Loading

How It Works

  1. Each cluster has a unique --cluster-id (e.g., central, site-east)
  2. Workers subscribe to cluster-scoped NATS subjects: wadjet.tasks.<cluster-id>.>
  3. The catalog stores per-cluster partition manifests
  4. When a table exists on multiple clusters, the coordinator's ExpandFederatedScans splits scan stages per-cluster
  5. Each cluster's workers scan their local data; results flow back for centralized aggregation

Setup

Central coordinator:

./wadjetd serve --mode coordinator \
  --cluster-id central \
  --nats-port 4222 \
  --endpoint minio-central:9000 \
  --access-key $S3_KEY --secret-key $S3_SECRET --bucket wadjet

Central workers:

./wadjetd serve --mode worker \
  --cluster-id central \
  --nats-url nats://coordinator:4222 \
  --memory-budget 268435456 --result-store 134217728 \
  --endpoint minio-central:9000 \
  --access-key $S3_KEY --secret-key $S3_SECRET --bucket wadjet

Remote site workers (with leaf node connection):

./wadjetd serve --mode worker \
  --cluster-id site-east \
  --nats-url nats://coordinator:4222 \
  --leaf-remote nats://coordinator:4222 \
  --memory-budget 67108864 --result-store 16777216 \
  --endpoint minio-east:9000 \
  --access-key $S3_KEY --secret-key $S3_SECRET --bucket wadjet

NATS Subject Routing

Tasks are published to cluster-scoped subjects:

wadjet.tasks.<cluster-id>.<type>.<query-id>.<stage-id>

Workers subscribe to their cluster's wildcard filter:

wadjet.tasks.<cluster-id>.>

This ensures workers only pull tasks for data in their local object store, avoiding cross-cluster data transfers for scan operations.

Standalone vs. Distributed

Aspect Standalone Distributed Federated
Processes 1 1 coordinator + N workers 1 coordinator + N workers across clusters
NATS Embedded (in-process) Embedded in coordinator Coordinator + leaf nodes
Query parallelism Single-process pipeline Multi-node task distribution Multi-cluster task routing
Best for Development, small datasets Production, large datasets Geo-distributed data
Scaling Vertical only Horizontal (add workers) Horizontal + geographic