test: add BDD E2E tests with cucumber-rs

[fredespi]

Summary

Adds a behavioral end-to-end test layer using cucumber-rs — standard Gherkin .feature files backed by Rust step functions. Tests exercise the full user journey (install → examine → configure → serve → detect → chat) to catch cross-component integration failures that unit tests and smoke scripts miss.

Why cucumber-rs over Robot Framework / plain Rust tests

This PR started with Robot Framework, then switched to cucumber-rs after team discussion. The reasoning:

• BDD earns its keep regardless of audience. The value isn't non-engineers reading scenarios — it's tests that describe user-facing behavior independently of implementation. "Short model names are expanded to their full name" tells you what's broken without reading code. assert_cmd/insta couples to output format and breaks on cosmetic changes.
• Already proven. The 6 bugs at component seams were found by thinking in user journeys, not by reading code. That's what BDD forces you to do.
• One toolchain. Rust step functions compile with the project. No Python dependency, no separate CI setup. Addresses the concern from EAI-7164 about consolidating on Rust.
• Not two stacks. The xtask acceptance path handles install/upgrade lifecycle (imperative, infrastructure-level). Cucumber handles user journey scenarios (declarative, behavioral). Different concerns, one toolchain.
• Real Gherkin. cucumber-rs uses standard Gherkin with Scenario Outline, Background, Rule, and @tags — not Robot Framework's cosmetic prefix stripping. The .feature file is the executable test, not a rewrite of a separate spec.
• Reporting included. HTML report (comparable to Robot Framework's), JUnit XML for GitHub Actions, and Cucumber JSON — all generated automatically.

What's included

• 20 BDD scenarios across 4 features: examine, runtime setup, model serving, chat/detection
• Gherkin .feature files — the spec is the test
• Rust step functions — organized by feature area (chat_steps.rs, examine_steps.rs, serving_steps.rs, runtime_steps.rs)
• Mock OpenAI server — axum-based, OS-assigned ports (no port conflicts)
• HTML report — summary, statistics by tag/feature, expandable scenario details with step-by-step results
• Environment isolation — each scenario gets isolated config/data/cache directories
• CI job on hosted runners (every PR)

Expected-failure pattern

Scenarios that exercise known bugs are tagged @expected-failure-EAI-XXXX. On CI without GPU hardware, these scenarios fail and produce clear failure messages. On GPU hardware, they exercise the actual bug. When a bug is fixed, the test passes — signaling the tag can be removed.

Tag	Scenarios	Bug
@expected-failure-EAI-7218	model_serving 5	PyTorch engine dependency resolution
@expected-failure-EAI-7219	model_serving 1, 2, 6	Short model names not expanded
@expected-failure-EAI-7220	model_serving 3, 4; chat 2, 4	Service registry / default port not in detection
@expected-failure-EAI-7221	chat 6	TUI hardcodes model name instead of querying endpoint
@expected-failure-EAI-7223	chat 5	vLLM rejects tool-calling requests

Each Jira ticket has a comment linking back to its test scenario(s).

Running tests

# All scenarios:
bash tests/e2e-cucumber/run.sh

# Via cargo directly:
ROCM_CLI_BINARY=./target/release/rocm cargo test -p e2e-cucumber --test e2e

# Filter by name:
cargo test -p e2e-cucumber --test e2e -- -n "short model"

# Skip known bugs:
cargo test -p e2e-cucumber --test e2e -- -t "not @expected-failure-EAI-7219"

Test plan

• All 20 scenarios compile and run
• Mock-tier scenarios pass locally
• Expected-failure scenarios fail with clear messages
• HTML report generated with summary, statistics, and step details
• JUnit XML generated for GitHub Actions
• CI job passes on this PR

[rominf]

Nice to see a real behavioral layer going in — the feature files read very cleanly. A few things before this is ready, one of which is functional and important.

🔴 Blocking — the suite can't currently fail CI

tests/e2e.rs uses the builder form E2eWorld::cucumber()....run(...).await; and discards the result. In cucumber 0.23 that path records failures into the writers but does not set a non-zero exit code — only run_and_exit() (or World::run()) does. So cargo test -p e2e-cucumber --test e2e exits 0 even when scenarios fail, and the e2e job goes green regardless of results. That undercuts the PR's premise ("as bugs are fixed, more tests pass") — right now the job wouldn't report either way.

Fix: capture the returned writer and std::process::exit(1) on failed_steps() + parsing_errors() + failed_hooks() > 0 (after the HTML report is generated so the artifact still uploads), or switch to run_and_exit().

Related, same theme:

• @expected-failure-* tags are inert in CI. They only take effect when a human passes -t "not @expected-failure-*"; neither the e2e nor e2e-gpu job passes any filter, so tagged scenarios run like any other. Once the exit-code issue above is fixed, every known-bug scenario would turn the job red — there's no xfail/skip mechanism wired up. Needs a real design (dedicated non-blocking job for the tagged set, or genuine skip-on-tag handling).
• A couple of scenarios don't exercise rocm. e.g. "Chat requests use the model name reported by the endpoint" — the assertion checks what the test's own helper sent to the mock (it reads the model from /models and echoes it back); the rocm binary is never invoked, so it can't fail for a product reason. The mock-based services list scenarios assert the registry contains a mock that's never registered with the CLI, so they fail today for the wrong reason and won't flip to passing when the underlying bug is fixed. Worth routing these through the actual binary or dropping them.

🟠 Fit the project

• run.sh → an xtask subcommand. We already standardize dev/CI tasks in xtask (signing, manifest, tpn, powershell-lint…). A bash wrapper is off-convention and bash-only in a repo that also builds/tests on Windows, and the build step is now duplicated between run.sh and the CI job. A cargo xtask e2e [args] used by both would be cross-platform and single-source.
• src/report.rs (≈400 lines of hand-built HTML) — do we need it? The crate already emits JUnit XML (rendered natively in the GitHub Actions checks UI) and Cucumber JSON (consumable by standard HTML reporters). The custom generator duplicates that, and it currently miscounts undefined/ambiguous steps as passed and prints a placeholder -xx-xx date. I'd lean toward dropping it in favor of the two standard outputs; if we want an HTML artifact in-tree, a compile-checked template (maud/askama) beats format! soup.

🟡 Should-fix

• Per-scenario isolation isn't isolated (e2e.rs): the temp root is keyed only on PID, so all concurrent scenarios share one config/data/cache tree, and the first scenario's Drop remove_dir_alls it out from under the others (risking a fall-back to the real ~/.rocm). Add a per-World unique suffix.
• reqwest::blocking inside async steps (serving_steps.rs) panics under the Tokio runtime ("cannot start a runtime from within a runtime"). GPU-tier only, but it hard-panics instead of waiting — use an async poll loop.
• The new crate gets no clippy (excluded from both the clippy job and the pre-commit hook). Add cargo clippy -p e2e-cucumber --all-targets -- -D warnings to the e2e job.
• .feature files aren't in the heavy path filter, so a feature-only change skips the e2e job. Add **/*.feature.

🔵 Nit

• @expected-failure-EAI-XXXX puts internal Jira IDs in files contributors read/type. Non-blocking and I know it's under discussion — flagging only so it's tracked; GitHub issue numbers would keep the same test↔bug link.

[fredespi]

Thanks for the thorough review — all points addressed. Rebased on main and pushed as one commit (6741054). Summary of what changed:

🔴 Blocking

• Suite can now fail CI. The runner captures the summarized writer and process::exit(1) on failed_steps + parsing_errors + hook_errors (after the HTML report is generated so the artifact still uploads). Verified: the blocking selection exits 0 when green and 1 when a scenario fails.
• @expected-failure tags are now real. Cucumber tag filters are exact-match (no globbing), so each known-bug scenario carries a bare @expected-failure (for filtering) plus its @expected-failure-EAI-NNNN (for traceability). CI now runs three selections:
  • e2e (blocking): not @gpu and not @expected-failure
  • e2e-known-bugs (non-blocking): @expected-failure and not @gpu
  • e2e-gpu (non-blocking): @gpu
• Scenarios now exercise rocm. Mock-based scenarios plant a managed-service record (plain JSON matching the on-disk schema, no crate import) into the isolated services dir, so rocm services list and the local chat provider actually discover the mock. The scenarios that were mislabeled EAI-7220 are retagged/untagged — EAI-7220 is a TUI-only wrong-port bug unrelated to services list, and its real surface can't be driven by the non-interactive CLI (noted below). The mock-only "chat model name" scenario that only tested the helper was dropped.

🟠 Fit the project

• run.sh → cargo xtask e2e. New xtask subcommand builds the release binary + runs the suite (forwarding cucumber args), used by both CI and local dev. Cross-platform, single-source; run.sh removed.
• report.rs kept, bugs fixed. Rather than drop it, fixed the two concrete issues: undefined/ambiguous steps now count as failures (not passes), and the -xx-xx placeholder is replaced with a real UTC timestamp. Happy to revisit dropping it in favor of the JUnit/JSON outputs as a follow-up if you'd prefer.

🟡 Should-fix

• Per-scenario isolation: temp root now has a per-World unique suffix (was PID-only, so scenarios shared one tree and Drop raced).
• reqwest::blocking → async poll loop in the serving steps (dropped the now-unused blocking feature).
• Clippy on the crate: added cargo clippy -p e2e-cucumber --all-targets -- -D warnings to the e2e job; fixed the warnings it surfaced.
• .feature in the heavy path filter so feature-only changes trigger the job.

🔵 Nit

• Kept the EAI-XXXX IDs for now since it's under discussion — happy to switch to GitHub issue numbers if we land on that.

Also, while rebasing

• Engine set: main now supports only lemonade/vllm (Limit serving engines to Lemonade and vLLM #79), so I updated the engines-list assertion, dropped the PyTorch scenario (EAI-7218 is Won't Fix and the engine is gone), and compare expansion across the supported engines.
• Restored the mock/gpu split that the cucumber rewrite had lost: hardware-dependent scenarios (install sdk, real serve, GPU detect) are tagged @gpu so the mock-tier job stays green on ubuntu-latest.

The full workspace test suite and the mock-tier e2e selection pass on Linux.