Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,24 @@ are bucket upper bounds; distributed percentiles are exact, client-observed, and
warm-up on a host shared with other workloads. The distributed results are single-Router,
single-gateway measurements, not a horizontal scaling curve.

## Compute efficiency

Throughput alone can rise by burning more compute, so the Engine also accounts modeled accelerator
time per delivered token. Each experiment below changes one scheduling decision. Details, caveats,
and repeat runs are in [`docs/p9-compute-efficiency.md`](docs/p9-compute-efficiency.md).

| Decision | Setup | Effect on compute |
|---|---|---|
| Continuous vs request-level batching | Same Engine, mixed output lengths, memory-bound cost model | Sustains 400 vs ~188 QPS; 44% less decode compute per token at 300 QPS (deterministic) |
| Redis prefix affinity | 4 Engines, 64 shared 512-token prefixes | Prefill computed −84%, fleet hardware time −29% to −33% |
| Cancellation of abandoned attempts | 10% of requests miss a 60 ms deadline | Hardware time −34% to −42% for the same 2,703 completions |
| Least-loaded vs round-robin | 4 Engines, one 3× slower | Hardware time −10% to −19%, p99 −22% (quiet run) |

Two findings cut the other way. Continuous admission charges each new prefill into the running
batch, which raises p99 on prefill-heavy workloads (chunked prefill is not implemented). Sending all
traffic to one Engine uses about 60% less compute than any spreading policy, so spreading is a
latency choice, not a saving. "Compute" is the cost model's modeled time, not GPU kernels.

## Reliability model

- **Delivery:** Kafka ingestion is at least once.
Expand Down Expand Up @@ -166,7 +184,10 @@ PYTHONPATH=python .venv/bin/python python/loadgen.py --url http://127.0.0.1:8000
```

Benchmarks: `scripts/run_p7_distributed_bench.sh` (full HTTP path) and `scripts/run_router_burst.sh`
(Router path only).
(Router path only). Efficiency experiments: `scripts/exp_batching_policy.sh` (deterministic) and
`scripts/exp_fleet_efficiency.py` (live services). Engines print their compute counters as a JSON
line on SIGTERM; `--policy static`, `--cancellation off`, and the `--*-us` cost-model flags exist
for these experiments. The Router takes `--routing least-loaded|round-robin|first`.

## Further reading

Expand All @@ -176,4 +197,5 @@ Benchmarks: `scripts/run_p7_distributed_bench.sh` (full HTTP path) and `scripts/
- [`docs/p6-multi-router-rebalancing.md`](docs/p6-multi-router-rebalancing.md): rebalance and crash recovery
- [`docs/p7-control-plane-observability.md`](docs/p7-control-plane-observability.md): gateway, load generation, and metrics
- [`docs/p8-multi-engine-routing.md`](docs/p8-multi-engine-routing.md): multi-Engine dispatch, retries, cancellation, Redis hints, tracing
- [`docs/p9-compute-efficiency.md`](docs/p9-compute-efficiency.md): compute accounting and one-variable efficiency experiments
- [`docs/benchmarks.md`](docs/benchmarks.md): reproducible measurements and claim boundaries
38 changes: 38 additions & 0 deletions architecture.html
Original file line number Diff line number Diff line change
Expand Up @@ -894,6 +894,29 @@ <h4>Evidence</h4>
</div>
</div>
</details>

<details class="work-item">
<summary>
<span class="phase-number">9</span>
<span class="phase-title">Compute efficiency</span>
<span class="phase-goal">Show that scheduling decisions spend less modeled accelerator time per delivered token, not just raise throughput.</span>
<span class="status">Implemented</span>
</summary>
<div class="work-body">
<div>
<h4>Understand</h4>
<p>The Engine accounts batch occupancy, useful versus wasted tokens, and prefill computed versus reused. Each experiment changes one decision: admission policy, routing policy, prefix hints, or cancellation.</p>
</div>
<div>
<h4>Read</h4>
<p><code>continuous_engine.cpp</code>, <code>engine_service.cpp</code>, <code>scripts/exp_batching_policy.sh</code>, <code>scripts/exp_fleet_efficiency.py</code>, and <code>docs/p9-compute-efficiency.md</code>.</p>
</div>
<div>
<h4>Evidence</h4>
<p>A deterministic CTest freezes the batching result; fleet runs repeat three times with compute deltas stable across host load. Findings that cut the other way (prefill interference, consolidation using less compute than spreading) are reported too.</p>
</div>
</div>
</details>
</div>
</section>

Expand Down Expand Up @@ -944,6 +967,21 @@ <h2 id="results-title">Measured results</h2>
<td>6,000 open-loop requests at 600 offered QPS, 2 Engines</td>
<td>558 achieved QPS, 58 ms p50, 98 ms p99, 0 errors</td>
</tr>
<tr>
<td>Continuous vs request-level batching (deterministic)</td>
<td>Mixed output lengths, memory-bound cost model</td>
<td>Sustains 400 vs ~188 QPS; 44% less decode compute per token at 300 QPS</td>
</tr>
<tr>
<td>Redis prefix affinity (Linux host)</td>
<td>4 Engines, 64 shared 512-token prefixes</td>
<td>Prefill computed −84%, fleet hardware time −29% to −33%</td>
</tr>
<tr>
<td>Cancellation of abandoned attempts (Linux host)</td>
<td>10% of requests miss a 60 ms deadline</td>
<td>Hardware time −34% to −42% for the same completions</td>
</tr>
</tbody>
</table>
</div>
Expand Down
8 changes: 8 additions & 0 deletions cpp/bench/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,10 @@
add_executable(infersched_local_bench scheduler_bench.cpp)
target_link_libraries(infersched_local_bench PRIVATE infersched_core)

# Guards the documented continuous-vs-static batching result (deterministic).
add_test(
NAME Experiments.ContinuousBatchingOutperformsStaticOnMixedWorkload
COMMAND ${CMAKE_SOURCE_DIR}/scripts/check_batching_claim.sh
$<TARGET_FILE:infersched_local_bench>)
set_tests_properties(Experiments.ContinuousBatchingOutperformsStaticOnMixedWorkload
PROPERTIES TIMEOUT 60 LABELS hermetic)
72 changes: 72 additions & 0 deletions cpp/bench/scheduler_bench.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ struct Arguments {
std::uint64_t seed{7};
bool telemetry{true};
std::string cache{"none"};
std::string policy{"continuous"}; // open_loop: continuous | static
std::string cost_model{"p7"}; // open_loop: p7 | memory-bound
std::string workload{"short"}; // open_loop: short | mixed
};

infersched::engine::EngineConfig BenchmarkConfig();
Expand Down Expand Up @@ -79,6 +82,12 @@ Arguments ParseArguments(const int argc, char** argv) {
arguments.telemetry = value == "enabled";
} else if (argument == "--cache") {
arguments.cache = value;
} else if (argument == "--policy") {
arguments.policy = value;
} else if (argument == "--cost-model") {
arguments.cost_model = value;
} else if (argument == "--workload") {
arguments.workload = value;
} else {
std::cerr << "unknown argument: " << argument << '\n';
std::exit(EXIT_FAILURE);
Expand All @@ -97,6 +106,21 @@ void RunOpenLoop(const Arguments& arguments) {
config.cost_model.prefill_per_token = std::chrono::microseconds(10);
config.cost_model.decode_fixed = std::chrono::microseconds(50);
config.cost_model.decode_per_sequence = std::chrono::microseconds(10);
if (arguments.cost_model == "memory-bound") {
// Decode dominated by a per-step fixed cost (reading weights once per
// step), as on bandwidth-bound accelerators: batching is nearly free.
config.cost_model.decode_fixed = std::chrono::microseconds(500);
config.cost_model.decode_per_sequence = std::chrono::microseconds(10);
} else if (arguments.cost_model != "p7") {
std::cerr << "--cost-model must be p7 or memory-bound\n";
std::exit(EXIT_FAILURE);
}
if (arguments.policy == "static") {
config.admission = infersched::engine::AdmissionPolicy::kStatic;
} else if (arguments.policy != "continuous") {
std::cerr << "--policy must be continuous or static\n";
std::exit(EXIT_FAILURE);
}
config.cost_model.jitter_basis_points = 200;
config.kv_blocks = 16'384;
config.prefix_cache_entries = 1024;
Expand All @@ -114,6 +138,18 @@ void RunOpenLoop(const Arguments& arguments) {
std::chrono::duration<double>(interarrival(random)));
}
auto request = MakeRequest(index);
if (arguments.workload == "mixed") {
// High output-length variance, the case request-level batching handles
// worst: 80% short replies (1-16 tokens), 20% long ones (64-512 tokens).
// Deterministic from the index so both policies see identical requests.
const std::uint64_t mix = (static_cast<std::uint64_t>(index) * 2654435761ULL) >> 7;
request.max_output_tokens =
mix % 5 == 0 ? 64 + static_cast<std::size_t>(mix % 449)
: 1 + static_cast<std::size_t>(mix % 16);
} else if (arguments.workload != "short") {
std::cerr << "--workload must be short or mixed\n";
std::exit(EXIT_FAILURE);
}
if ((arguments.cache == "result" || arguments.cache == "both") &&
index % 5 == 4) {
const std::size_t origin = index - 4;
Expand All @@ -140,6 +176,10 @@ void RunOpenLoop(const Arguments& arguments) {
infersched::metrics::Histogram ttft;
infersched::metrics::Histogram end_to_end;
infersched::metrics::Histogram low_priority;
std::vector<double> e2e_exact_ms;
std::vector<double> ttft_exact_ms;
e2e_exact_ms.reserve(engine.results().size());
ttft_exact_ms.reserve(engine.results().size());
for (const auto& result : engine.results()) {
const auto submitted = arrivals.at(result.request_id);
const auto ttft_us = std::chrono::duration_cast<std::chrono::microseconds>(
Expand All @@ -148,6 +188,8 @@ void RunOpenLoop(const Arguments& arguments) {
const auto e2e_us = std::chrono::duration_cast<std::chrono::microseconds>(
result.completed_at - submitted)
.count();
e2e_exact_ms.push_back(static_cast<double>(e2e_us) / 1000.0);
ttft_exact_ms.push_back(static_cast<double>(ttft_us) / 1000.0);
if (arguments.telemetry) {
ttft.Record(
static_cast<std::uint64_t>(std::max<std::int64_t>(ttft_us, 0)));
Expand All @@ -169,6 +211,21 @@ void RunOpenLoop(const Arguments& arguments) {
const auto drain_lag = std::chrono::duration_cast<std::chrono::microseconds>(
engine.clock().Now() - arrival);
const bool complete = engine.results().size() == arguments.requests;
const auto exact = [](std::vector<double>& values, const double quantile) {
if (values.empty()) {
return 0.0;
}
std::ranges::sort(values);
const auto index = static_cast<std::size_t>(
static_cast<double>(values.size() - 1) * quantile);
return values[index];
};
const auto& stats = engine.stats();
const double decode_us_per_useful_token =
stats.useful_output_tokens == 0
? 0.0
: std::chrono::duration<double, std::micro>(stats.simulated_decode_time).count() /
static_cast<double>(stats.useful_output_tokens);
engine.ClearCaches();
const bool zero_leak = engine.kv_cache().allocated_blocks() == 0;
const double benchmark_wall_ms =
Expand Down Expand Up @@ -204,6 +261,21 @@ void RunOpenLoop(const Arguments& arguments) {
: static_cast<double>(
engine.stats().result_cache_request_hits) /
static_cast<double>(engine.stats().result_cache_requests))
<< ",\"policy\":\"" << arguments.policy << "\""
<< ",\"cost_model\":\"" << arguments.cost_model << "\""
<< ",\"workload\":\"" << arguments.workload << "\""
<< ",\"decode_iterations\":" << stats.iterations
<< ",\"mean_batch_occupancy\":" << stats.mean_batch_occupancy()
<< ",\"useful_output_tokens\":" << stats.useful_output_tokens
<< ",\"wasted_decode_tokens\":" << stats.wasted_decode_tokens
<< ",\"preemptions\":" << stats.preemptions
<< ",\"modeled_busy_ms\":"
<< std::chrono::duration<double, std::milli>(stats.simulated_busy_time).count()
<< ",\"decode_us_per_useful_token\":" << decode_us_per_useful_token
<< ",\"useful_tokens_per_busy_s\":" << stats.useful_tokens_per_busy_second()
<< ",\"ttft_p99_exact_ms\":" << exact(ttft_exact_ms, 0.99)
<< ",\"e2e_p50_exact_ms\":" << exact(e2e_exact_ms, 0.50)
<< ",\"e2e_p99_exact_ms\":" << exact(e2e_exact_ms, 0.99)
<< ",\"complete\":" << std::boolalpha << complete
<< ",\"zero_kv_leak\":" << zero_leak << "}\n";
if (!complete || !zero_leak) {
Expand Down
2 changes: 1 addition & 1 deletion cpp/include/infersched/coordination/redis_hints.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ class RedisHints {
RedisHints(const RedisHints&) = delete;
RedisHints& operator=(const RedisHints&) = delete;

// Parses "host:port". An empty string yields nullopt (hints disabled).
// Parses "host:port". "", "none", or "off" yield nullopt (hints disabled).
[[nodiscard]] static std::optional<RedisHintsOptions> ParseAddress(
std::string_view address);

Expand Down
18 changes: 15 additions & 3 deletions cpp/include/infersched/distributed/engine_selector.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,14 @@

namespace infersched::distributed {

enum class RoutingPolicy {
kLeastLoaded, // Load estimate + optional prefix affinity (default).
kRoundRobin, // Rotate through live Engines, ignoring load.
kFirst, // Always the lowest engine_id: the pre-P8 single-Engine behaviour.
};

[[nodiscard]] std::optional<RoutingPolicy> ParseRoutingPolicy(std::string_view name);

struct SelectionInput {
const std::vector<EngineRecord>& engines;
// Attempts this Router has in flight per engine_id (exact for this Router).
Expand All @@ -27,6 +35,9 @@ struct SelectionInput {
// the request id — so ties spread across Engines instead of always landing
// on the lexicographically first one.
std::string_view tie_breaker{};
RoutingPolicy policy{RoutingPolicy::kLeastLoaded};
// Monotonic per-Router counter used by kRoundRobin.
std::size_t round_robin_ticket{};
};

struct Selection {
Expand All @@ -40,9 +51,10 @@ struct Selection {
[[nodiscard]] std::size_t EstimateLoad(const EngineRecord& engine,
std::size_t local_inflight);

// Deterministic choice: valid affinity hint within slack, otherwise the least
// loaded non-saturated Engine (ties broken by engine_id). nullopt when every
// live Engine is excluded or there are none.
// kLeastLoaded: valid affinity hint within slack, otherwise the least loaded
// non-saturated Engine (ties by hash). kRoundRobin / kFirst ignore load and
// hints but still skip excluded Engines. nullopt when every live Engine is
// excluded or there are none.
[[nodiscard]] std::optional<Selection> SelectEngine(const SelectionInput& input);

} // namespace infersched::distributed
16 changes: 16 additions & 0 deletions cpp/include/infersched/distributed/engine_service.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
#include <deque>
#include <memory>
#include <mutex>
#include <optional>
#include <string>
#include <thread>
#include <unordered_map>
Expand All @@ -32,6 +33,10 @@ struct EngineServiceOptions {
// Bound on remembered (request_id -> attempt) fences used to reject stale
// or duplicate dispatches. Postgres remains the authoritative fence.
std::size_t fence_memory{65536};
// When false, Cancel RPCs and abandoned streams no longer stop the
// sequence: it runs to completion and its result is discarded. Exists only
// to measure the compute that cancellation saves.
bool cancellation{true};
};

// Network wrapper around one ContinuousEngine. A single worker thread owns the
Expand Down Expand Up @@ -59,6 +64,9 @@ class EngineService final : public v1::EngineControl::Service {
[[nodiscard]] v1::EngineCapacity Capacity() const;

struct Stats {
// Output tokens computed for attempts whose result nobody accepted
// (stream abandoned or attempt superseded) because cancellation was off.
std::uint64_t abandoned_output_tokens{};
std::uint64_t accepted{};
std::uint64_t rejected{};
std::uint64_t completed{};
Expand All @@ -67,6 +75,10 @@ class EngineService final : public v1::EngineControl::Service {
std::size_t peak_running{};
};
[[nodiscard]] Stats stats() const;
// Compute-efficiency snapshot as one JSON object: engine counters (modeled
// busy time, batch occupancy, useful vs wasted tokens, prefill reuse) plus
// wall-clock busy fraction over the window from first accept to last finish.
[[nodiscard]] std::string UtilizationJson(const std::string& engine_id) const;

private:
struct Job {
Expand All @@ -76,6 +88,7 @@ class EngineService final : public v1::EngineControl::Service {
std::deque<v1::DispatchEvent> events;
bool done{};
std::atomic<bool> superseded{};
std::atomic<bool> abandoned{}; // Client stream gone before the result.
};

[[nodiscard]] std::string ValidateAndReserve(
Expand Down Expand Up @@ -106,6 +119,9 @@ class EngineService final : public v1::EngineControl::Service {
bool draining_{};
bool stop_{};
Stats stats_;
engine::ContinuousEngineStats engine_stats_; // Worker's latest copy.
std::optional<std::chrono::steady_clock::time_point> first_accept_;
std::optional<std::chrono::steady_clock::time_point> last_finish_;

// Published by the worker for Capacity()/heartbeats.
std::atomic<std::size_t> free_kv_blocks_;
Expand Down
29 changes: 29 additions & 0 deletions cpp/include/infersched/engine/continuous_engine.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,34 @@ struct ContinuousEngineStats {
std::size_t result_cache_request_hits{};
std::size_t peak_pending_requests{};
std::chrono::nanoseconds simulated_busy_time{};

// Compute accounting (all in tokens of modeled work).
std::chrono::nanoseconds simulated_prefill_time{};
std::chrono::nanoseconds simulated_decode_time{};
// Sum over decode iterations of the batch size: decode tokens computed.
std::size_t decode_token_slots{};
// Output tokens of requests that completed successfully.
std::size_t useful_output_tokens{};
// Decode tokens thrown away: preempted sequences restart from scratch, and
// cancelled sequences produce nothing anyone accepts.
std::size_t wasted_decode_tokens{};
std::size_t prefill_tokens_computed{}; // Uncached prompt tokens prefilled.
std::size_t prefill_tokens_reused{}; // Served from the prefix cache.
std::size_t wasted_prefill_tokens{}; // Prefill of preempted/cancelled work.
std::size_t cancellations{};
std::size_t completed_requests{};

// Mean sequences per decode step: how full the batch was while decoding.
[[nodiscard]] double mean_batch_occupancy() const noexcept {
return iterations == 0 ? 0.0
: static_cast<double>(decode_token_slots) /
static_cast<double>(iterations);
}
// Useful output tokens per second of modeled compute.
[[nodiscard]] double useful_tokens_per_busy_second() const noexcept {
const double busy = std::chrono::duration<double>(simulated_busy_time).count();
return busy == 0.0 ? 0.0 : static_cast<double>(useful_output_tokens) / busy;
}
};

class ContinuousEngine {
Expand Down Expand Up @@ -77,6 +105,7 @@ class ContinuousEngine {
std::size_t generated_tokens{};
bool prefix_cache_hit{};
FakeClock::time_point first_token_at{};
std::size_t prefilled_tokens{}; // Uncached prompt tokens this admission computed.
};

[[nodiscard]] bool AdmitReady();
Expand Down
7 changes: 7 additions & 0 deletions cpp/include/infersched/engine/deterministic_engine.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,14 @@

namespace infersched::engine {

// How ContinuousEngine admits waiting requests.
// kContinuous: at every iteration boundary, into any free batch slot.
// kStatic: only once the running batch has fully drained (request-level
// batching). Same data structures, so it isolates the policy.
enum class AdmissionPolicy { kContinuous, kStatic };

struct EngineConfig {
AdmissionPolicy admission{AdmissionPolicy::kContinuous};
std::size_t kv_blocks{256};
std::size_t kv_block_size_tokens{16};
std::size_t max_batch_sequences{8};
Expand Down
Loading
Loading