Skip to content

Route across an Engine fleet with concurrent, fenced dispatch - #1

Merged
Abhayrkhot merged 3 commits into
mainfrom
p8-fleet-routing
Nov 19, 2026
Merged

Abhayrkhot merged 3 commits into
mainfrom
p8-fleet-routing

Conversation

@Abhayrkhot

@Abhayrkhot Abhayrkhot commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Closes the gaps between the README and the durable runtime. Design notes: docs/p8-multi-engine-routing.md.

What was missing

  • Each durable Router talked to the first Engine that registered and dispatched one request at a time. Heartbeat capacity was unused and there was no failover.
  • The network Engine drained its queue and ran to idle, so new arrivals waited for the whole batch.
  • Redis held nothing; inference.traces was never written; Cancel was never called.
  • The Router exited after --requests N, and the gateway kept unbounded in-memory state with no timeouts.

What changed

  • Engine: iteration-level admission across RPCs, TTFT streaming, fenced cancellation that frees KV, bounded memory, multi-Router registration.
  • Router: 128 dispatch workers over a 16-connection Postgres pool. Selection is load-aware plus incarnation-checked Redis prefix affinity. Failed attempts retry on other Engines, and an owner-fenced FAILED_TERMINAL replaces silent drops. Adds cancel on revoke/deadline, SIGTERM drain, and trace spans.
  • Gateway: broker-acked 202, long-poll, expiry, bounded caches, /readyz, Prometheus /metrics.
  • Found while measuring:
    • one synchronous Kafka commit per offset;
    • an unindexed outbox scan per completion;
    • the gateway replaying the whole results topic on start;
    • a load generator that saturated its own CPU.

Results (shared Linux host, see docs/benchmarks.md)

Before After
Router burst, zero compute 68 req/s ~1,450 req/s
HTTP path at 600 QPS, 2 Engines — (ceiling ~50 QPS) 558 QPS, 58 ms p50, 98 ms p99

Testing

  • 68/68 tests pass on Ubuntu 24.04 with live Kafka, Postgres, and Redis: 47 hermetic + 21 live.
  • 19 of the tests are new, including multi-Engine SIGKILL failover with exactly-once results, Redis affinity surviving FLUSHALL, and in-process gRPC Engine tests.
  • ASan: 47/47 hermetic tests pass on Linux.
  • TSan on Linux (GCC 13 + Ubuntu's uninstrumented gRPC 1.51) reports races only inside gRPC internals. The unchanged gRPC tests on main report the same ones. macOS TSan in CI is the reference.

CI changes

  • macOS runner: the hermetic jobs moved from macos-14 to macos-15. GitHub now rejects the old runner ("macos-14-arm64-us-default is not supported"), so those jobs were cancelled before they ran. This also affects main.
  • ASan moved to Ubuntu 24.04. On macOS, Homebrew now installs protobuf 36.2, and ASan builds crash in its sanitizer-only TcParser::VerifyHasBitConsistency when generated messages are destroyed. The untouched smoke test crashes the same way. The macOS TSan job runs the same check without crashing. On Ubuntu, all 47 hermetic tests pass under ASan with LeakSanitizer.

The durable path dispatched one request at a time to the first Engine
that registered, batched per RPC, kept nothing in Redis, and exited after
a fixed request count. This makes it a running service without moving
the correctness boundary.

Engine
- One worker runs ContinuousEngine iterations back to back; Dispatch RPCs
  join the live batch at the next iteration boundary.
- Stream PREFILL_DONE at the first decoded token, then COMPLETED/FAILED.
- Fenced Cancel and broken client streams free the sequence and its KV.
- Drain results, forget finished requests, cap remembered fences.
- Register with several Routers and re-register after a lost lease.

Router
- 128 dispatch workers share a 16-connection Postgres pool, leased only
  for fenced writes, never while streaming.
- Pick the least-loaded Engine, or an incarnation-checked Redis prefix
  hint within a load slack; select and reserve atomically.
- Retry rejected/failed/lost attempts on other Engines with backoff;
  owner-fenced FAILED_TERMINAL (result + outbox) when the budget runs out.
- Cancel attempts on partition revocation and deadline expiry.
- Run until SIGTERM, drain, and print stats; JSON-escape result envelopes.
- Emit router.ingest/dispatch/request spans to inference.traces.

Fixes found while measuring
- Coalesce Kafka offset commits and send them asynchronously every
  100 ms instead of one synchronous commit per offset.
- Index outbox by source position (IsSourcePublished was a growing
  sequential scan) and batch ingest commits and published checks.
- Start the gateway results consumer at the high watermark instead of
  replaying the whole topic on every start; gate /readyz on it.
- The load generator saturated its own CPU near 100 req/s; spread it
  across processes, add warm-up, and report client saturation.

Gateway
- 202 only after the broker acknowledges; 503 otherwise.
- Long-poll results, expire lost requests, bound caches, ignore other
  gateways results, Prometheus /metrics, JSON /v1/stats.

Router burst throughput rose from 68 to ~1,450 req/s; the full HTTP path
sustains 600 QPS at 98 ms p99 on the shared Linux host (docs/benchmarks.md).
GitHub no longer accepts the macos-14 runner configuration
("macos-14-arm64-us-default is not supported"), so the hermetic
matrix was cancelled before it got a runner.
On macos-15, every test that destroys a generated message crashes in
protobuf's sanitizer-only TcParser::VerifyHasBitConsistency check:
code built with ASan runs against Homebrew's uninstrumented
libprotobuf 36.2. It reproduces with the untouched smoke test, and the
TSan job runs the same check without crashing. Ubuntu 24.04 packages
pass all 47 hermetic tests under ASan with LeakSanitizer enabled.
@Abhayrkhot
Abhayrkhot merged commit 4d0cab6 into main Nov 19, 2026
4 checks passed
@Abhayrkhot
Abhayrkhot deleted the p8-fleet-routing branch November 19, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant