Repository navigation
Route across an Engine fleet with concurrent, fenced dispatch - #1
Merged
Merged
Conversation
The durable path dispatched one request at a time to the first Engine that registered, batched per RPC, kept nothing in Redis, and exited after a fixed request count. This makes it a running service without moving the correctness boundary. Engine - One worker runs ContinuousEngine iterations back to back; Dispatch RPCs join the live batch at the next iteration boundary. - Stream PREFILL_DONE at the first decoded token, then COMPLETED/FAILED. - Fenced Cancel and broken client streams free the sequence and its KV. - Drain results, forget finished requests, cap remembered fences. - Register with several Routers and re-register after a lost lease. Router - 128 dispatch workers share a 16-connection Postgres pool, leased only for fenced writes, never while streaming. - Pick the least-loaded Engine, or an incarnation-checked Redis prefix hint within a load slack; select and reserve atomically. - Retry rejected/failed/lost attempts on other Engines with backoff; owner-fenced FAILED_TERMINAL (result + outbox) when the budget runs out. - Cancel attempts on partition revocation and deadline expiry. - Run until SIGTERM, drain, and print stats; JSON-escape result envelopes. - Emit router.ingest/dispatch/request spans to inference.traces. Fixes found while measuring - Coalesce Kafka offset commits and send them asynchronously every 100 ms instead of one synchronous commit per offset. - Index outbox by source position (IsSourcePublished was a growing sequential scan) and batch ingest commits and published checks. - Start the gateway results consumer at the high watermark instead of replaying the whole topic on every start; gate /readyz on it. - The load generator saturated its own CPU near 100 req/s; spread it across processes, add warm-up, and report client saturation. Gateway - 202 only after the broker acknowledges; 503 otherwise. - Long-poll results, expire lost requests, bound caches, ignore other gateways results, Prometheus /metrics, JSON /v1/stats. Router burst throughput rose from 68 to ~1,450 req/s; the full HTTP path sustains 600 QPS at 98 ms p99 on the shared Linux host (docs/benchmarks.md).
GitHub no longer accepts the macos-14 runner configuration
("macos-14-arm64-us-default is not supported"), so the hermetic
matrix was cancelled before it got a runner.
On macos-15, every test that destroys a generated message crashes in protobuf's sanitizer-only TcParser::VerifyHasBitConsistency check: code built with ASan runs against Homebrew's uninstrumented libprotobuf 36.2. It reproduces with the untouched smoke test, and the TSan job runs the same check without crashing. Ubuntu 24.04 packages pass all 47 hermetic tests under ASan with LeakSanitizer enabled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the gaps between the README and the durable runtime. Design notes:
docs/p8-multi-engine-routing.md.What was missing
inference.traceswas never written;Cancelwas never called.--requests N, and the gateway kept unbounded in-memory state with no timeouts.What changed
FAILED_TERMINALreplaces silent drops. Adds cancel on revoke/deadline, SIGTERM drain, and trace spans./readyz, Prometheus/metrics.Results (shared Linux host, see
docs/benchmarks.md)Testing
FLUSHALL, and in-process gRPC Engine tests.mainreport the same ones. macOS TSan in CI is the reference.CI changes
macos-14tomacos-15. GitHub now rejects the old runner ("macos-14-arm64-us-default is not supported"), so those jobs were cancelled before they ran. This also affectsmain.TcParser::VerifyHasBitConsistencywhen generated messages are destroyed. The untouched smoke test crashes the same way. The macOS TSan job runs the same check without crashing. On Ubuntu, all 47 hermetic tests pass under ASan with LeakSanitizer.