support WASIp3 instance reuse in wasmtime serve (#11807)

* support WASIp3 instance reuse in `wasmtime serve`

This is a draft implementation of WASIp3 instance reuse.  Previously, we used
one store and one instance per request for both p2 and p3 handlers, discarding
the store and instance immediately after the request was handled.  Now we
attempt to reuse p3 instances and their stores instead, both serially and
concurrently.

Advantages of reuse:

- Higher throughput: We can amortize the time spent creating and disposing of instances and stores over a number of requests.
- Lower memory and address space usage: Concurrent instance reuse allows us to handle more requests with fewer instances, especially when the handler is I/O bound.
- Developers will presumably need to make changes to their applications anyway when migrating from p2 to p3, so this is an ideal time to change the default from no reuse to (some) reuse, assuming we want to do it at all.

Disadvantages of reuse:

- Reduced isolation: When the same instance is used to handle multiple requests, the blast radius of bugs in the guest is larger; a trap can affect unrelated requests, and a security flaw could leak state across requests.
- Host implementation complexity: There's more code to manage and more configuration knobs to tune.
- Guest implementation complexity: You can't just stash state in a global variable and call it a day.  Similarly, shortcuts like using a leaking GC require additional thought.

Given the tradeoffs, we'll definitely need to provide components a way to opt
out of reuse if desired.  See
WebAssembly/wasi-http#190 for related discussion.

Implementation details:

I've moved `ProxyHandler` from `wasmtime_cli::commands::serve` to
`wasmtime_wasi_http::handler` (so it can be reused by custom embedders),
abstracted away the `serve`-specific parts, and added support for instance reuse
via a new `ProxyHandler::push` function.  The `push` function takes a work item
representing an incoming request and dispatches it to a dynamically sized pool
of worker tasks, each with its own store and `wasi:http/[email protected]` instance.
Worker tasks accept work items as capacity allows until they either reach a
maximum total reuse count or hit an idle timeout, at which point they exit.

Performance:

I've run a few benchmarks using `wasmtime-serve-rps.sh` and
[hello-wasip3-http](https://github.com/dicej/hello-wasip3-http/blob/main/src/lib.rs)
(a simple "hello, world" request handler).  Highlights:

- 45% more requests per second with instance reuse enabled vs. disabled
- 64% fewer concurrent instances required for an I/O-bound handler
    - Here I added a 40ms delay to the "hello, world" handler to simulate I/O

TODOs:

- Support host-enforced request timeouts.  Note that we can't simply use the epoch-based trap-on-timeout approach we've traditionally used since a given instance may be responsible for a number of concurrent requests.  This will require adding a public API to the `wasmtime` crate for cleanly cancelling a guest task without affecting other ones.
- Add CLI options for the configuration knobs (e.g. max request count per instance, idle timeout, etc.)
- Use guest signals like `backpressure`, explicit `wasi:cli/exit/exit`, WebAssembly/wasi-http#190, etc. to drive reuse decisions

Signed-off-by: Joel Dice <[email protected]>

* additional instance reuse support

This addresses a couple of TODO items from my previous instance reuse PR:

- Support request timeouts by gracefully shutting down the worker as soon as any request hits a timeout, letting any other pending tasks finish or time out before actually dropping the instance.
- Convert the previously hard-coded config options to CLI options.
- Enable profiling when instance reuse is enabled.

Signed-off-by: Joel Dice <[email protected]>

* start worker tasks more aggressively

This improves `wasmtime serve` performance when instance reuse is enabled by:

- spawning a worker task any time there are no idle workers available
- considering a worker to be unavailable until it has finished initializing itself

Signed-off-by: Joel Dice <[email protected]>

* revert unecessary wasmtime-serve-rps.sh changes

Signed-off-by: Joel Dice <[email protected]>

* enable WASIp2 instance reuse

Signed-off-by: Joel Dice <[email protected]>

* add clarifying comments about when and why we start worker tasks

Signed-off-by: Joel Dice <[email protected]>

* remove duplicated code in `ProxyHandler::spawn`

By bypassing the task queue when there are no already-available workers, I've
narrowed the performance gap between the "fast path" code and the normal path to
about 2%, which is close enough that we can remove the duplicated code.

Signed-off-by: Joel Dice <[email protected]>

* ensure timeouts enforced if event loop stalls

This fixes an issue such that timeouts weren't being reliably enforced when the
guest either busy loops for an extended time or makes a sync call to a host
function that takes exclusive access to the `Store` and blocks for a while.  In
either of those cases, the `StoreContextMut::run_concurrent` event loop will
stall, being unable to make progress while the task fiber monopolizes the
`Store`.  In this case, we must enforce timeouts _outside_ the event loop.  See
the new code comments for details on how that is done.

This also tweaks `wasmtime serve` stderr/stdout output a bit to address
`cli_tests::cli_serve_*` test regressions.

Signed-off-by: Joel Dice <[email protected]>

* add reuse and timeout tests to `cli_tests`

This adds several new integration tests which exercise various aspects of
`wasmtime serve`'s instance reuse feature, including timeouts for p2-style
(synchronous) sleeps and p3-style (asynchronous) sleeps.

Signed-off-by: Joel Dice <[email protected]>

* fix foreach_api breakage

Signed-off-by: Joel Dice <[email protected]>

* Minor cleanups to `handler.rs`

* Don't use `FutureExt`, use `async { .. }` instead.
* Don't use `T`, instead use `S::StoreData`.

* Shift an `unreachable!()` to be closer to the "source"

* add more comments to `handler.rs`

These comments draw attention to a few of the more subtle aspects of the code
and the invariants it is upholding.

Signed-off-by: Joel Dice <[email protected]>

* add docs regarding the `req_id` parameter

Signed-off-by: Joel Dice <[email protected]>

* add comments regarding atomic load/store orderings

This also changes one of the `SeqCst` orderings to `Relaxed`.

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>
Co-authored-by: Alex Crichton <[email protected]>

Author: dicej

Cargo.toml

@@ -87,6 +87,7 @@ tokio = { workspace = true, optional = true, features = [ "signal", "macros" ] }
 hyper = { workspace = true, optional = true }
 http = { workspace = true, optional = true }
 http-body-util = { workspace = true, optional = true }
+futures = { workspace = true, optional = true }
 
 [target.'cfg(unix)'.dependencies]
 rustix = { workspace = true, features = ["mm", "process"] }
@@ -504,6 +505,7 @@ component-model-async = [
   "component-model",
   "wasmtime-wasi?/p3",
   "wasmtime-wasi-http?/p3",
+  "dep:futures",
 ]
 
 # This feature, when enabled, will statically compile out all logging statements
@@ -549,6 +551,7 @@ debug = ["wasmtime-cli-flags/debug", "wasmtime/debug"]
 # for more information on each subcommand.
 serve = [
   "wasi-http",
+  "wasmtime-wasi-http/component-model-async",
   "component-model",
   "dep:http-body-util",
   "dep:http",

crates/test-programs/src/bin/p2_cli_serve_hello_world.rs

@@ -0,0 +1,27 @@
+use test_programs::proxy;
+use test_programs::wasi::http::types::{
+    Fields, IncomingRequest, OutgoingBody, OutgoingResponse, ResponseOutparam,
+};
+
+struct T;
+
+proxy::export!(T);
+
+impl proxy::exports::wasi::http::incoming_handler::Guest for T {
+    fn handle(_: IncomingRequest, outparam: ResponseOutparam) {
+        let fields = Fields::new();
+        let resp = OutgoingResponse::new(fields);
+        let body = resp.body().expect("outgoing response");
+
+        ResponseOutparam::set(outparam, Ok(resp));
+
+        let out = body.write().expect("outgoing stream");
+        out.blocking_write_and_flush(b"Hello, WASI!")
+            .expect("writing response");
+
+        drop(out);
+        OutgoingBody::finish(body, None).expect("outgoing-body.finish");
+    }
+}
+
+fn main() {}

crates/test-programs/src/bin/p3_cli_serve_sleep.rs

@@ -0,0 +1,18 @@
+use test_programs::p3::proxy;
+use test_programs::p3::wasi::clocks::monotonic_clock;
+use test_programs::p3::wasi::http::types::{ErrorCode, Request, Response};
+
+struct T;
+
+proxy::export!(T);
+
+impl proxy::exports::wasi::http::handler::Guest for T {
+    async fn handle(_request: Request) -> Result<Response, ErrorCode> {
+        monotonic_clock::wait_for(u64::MAX).await;
+        unreachable!()
+    }
+}
+
+fn main() {
+    unreachable!()
+}

crates/wasi-http/Cargo.toml

@@ -18,6 +18,7 @@ all-features = true
 default = ["default-send-request"]
 default-send-request = ["dep:tokio-rustls", "dep:rustls", "dep:webpki-roots"]
 p3 = ["wasmtime-wasi/p3", "dep:tokio-util"]
+component-model-async = ["futures/alloc", "wasmtime/component-model-async"]
 
 [dependencies]
 anyhow = { workspace = true }