feat(remote): add rocm remote — serve a model on a remote GPU host over SSH#76
feat(remote): add rocm remote — serve a model on a remote GPU host over SSH#76volen-silo wants to merge 3 commits into
rocm remote — serve a model on a remote GPU host over SSH#76Conversation
volen-silo
changed the title
rocm remote — serve a model on a remote GPU host over SSH
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| fn temp_paths(tag: &str) -> (std::path::PathBuf, AppPaths) { | ||
| let root = | ||
| std::env::temp_dir().join(format!("rocm-remote-test-{tag}-{}", std::process::id())); | ||
| let _ = std::fs::remove_dir_all(&root); |
| // Unknown handle errors. | ||
| assert!(resolve_session(&paths, "nope").is_err()); | ||
|
|
||
| let _ = std::fs::remove_dir_all(root); |
| assert!(err.contains("gpu-box-11435")); | ||
| assert!(err.contains("gpu-box-11436")); | ||
|
|
||
| let _ = std::fs::remove_dir_all(root); |
| .context("remote session path must have a parent directory")?; | ||
| fs::create_dir_all(parent) | ||
| .with_context(|| format!("failed to create {}", parent.display()))?; | ||
| fs::write( |
| /// skipped rather than failing the whole listing. | ||
| pub fn load_remote_sessions(paths: &AppPaths) -> Result<Vec<RemoteSessionRecord>> { | ||
| let dir = paths.remote_sessions_dir(); | ||
| if !dir.is_dir() { |
| return Ok(Vec::new()); | ||
| } | ||
| let mut records = Vec::new(); | ||
| for entry in fs::read_dir(&dir).with_context(|| format!("failed to read {}", dir.display()))? { |
CodeQL
|
| Alert | Sink | "user-provided value" is… | Verdict |
|---|---|---|---|
| 604–606 | remove_dir_all in remote/mod.rs tests |
the process's own TMPDIR (std::env::temp_dir()) |
test-only cleanup of our own temp dir |
| 608–609 | read_dir / is_dir on remote_sessions_dir() |
the app's own ROCM_CLI_DATA_DIR config env var |
reading our own data dir |
| 607 | fs::write in RemoteSessionRecord::write |
session_id, derived from the host CLI arg |
session_id_for() sanitizes host to [A-Za-z0-9-] (no /, no .), so it can't escape the sessions dir |
This is the same pattern the existing managed-service code uses (load_managed_services, ManagedServiceRecord::write), which is why the repo already has ~87 open alerts of this exact class. Nothing here widens the trust boundary: the "inputs" are the tool's own config env var, its own TMPDIR, and a sanitized CLI argument.
Recommendation: dismiss as "won't fix / false positive", consistent with the existing alerts of this class. Happy to add a defensive [A-Za-z0-9._-]-only sanitizer at the remote_session_path boundary if we'd prefer belt-and-suspenders — though it would only address alert 607 and diverge from how the rest of the codebase treats the data dir.
volen-silo
force-pushed
the
rocm-remote
branch
2 times, most recently
from
7897377 to
da6c506
Compare
Bring up a managed model server on a remote GPU host with one command: connect over SSH, ensure the CLI + ROCm are present (auto-push the CLI if missing, detect-and-fail if ROCm is absent), start a managed `rocm serve` on the remote, open a loopback port-forward, and print an OpenAI-compatible base URL. The command holds the tunnel open until interrupted; the remote server is managed and persists. The new remote module reuses the remote's existing serve/services machinery rather than reimplementing engine/runtime/device selection. Transport is a trait (SshTransport shells out to system ssh/scp) so a Tailscale-native transport can slot in later. Adds `rocm services list --json` as the machine-readable listing the remote orchestration reads back. Also registers `remote` in the natural-language planner's structured-command allowlist so `rocm remote ...` routes to clap instead of the planner. Phase 2 (session registry, detached tunnel keeper, remote list/status/stop with reconciliation) is not included. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Remote serving is now async and always-persistent. `serve` starts the managed server on the remote (which loads the model in the background), opens a detached SSH tunnel that survives the command, records the session locally, and returns immediately with a Pending status and the OpenAI base URL — no more foreground hold. New verbs reconcile two independent lifecycles: `status [<id>] [--watch]` shows a table of sessions with remote-server health (polled over SSH via `services list --json`) and local tunnel liveness (PID check) as separate columns; `attach` rebuilds a dropped tunnel without redeploying; `stop` closes the tunnel and stops the remote server. `list` is reserved for configured hosts. The Pending->Healthy progression comes from the remote's own managed-service lifecycle, so there is no local reconciler daemon — status polls on demand. The tunnel keeper is a standalone detached ssh process using ControlMaster=no so killing its PID deterministically closes the forward. Bootstrap now reports progress when pushing the CLI to a remote that lacks it. State lives in a new RemoteSessionRecord (one JSON file per session under remote-sessions/), mirroring ManagedServiceRecord. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
How to test
|
|
We would likely use TCP for remote connectivity since it's already wired in. |
3 participants
Summary
rocm remote serve <host> <model>brings up an OpenAI-compatible model server on a remote AMD GPU box and makes it callable from your machine with one command: connect over SSH, ensure the CLI + ROCm are present (pushing the CLI if missing), start a managedrocm serveon the remote, open a loopback tunnel, and print a ready-to-use base URL. Sessions are persistent and observable —rocm remote status/attach/stopmanage them afterward.Built in two commits (one per phase).
Commit 1 —
rocm remote serverocm remotecommand group andapps/rocm/src/remote/module (transport,bootstrap,mod).Transporttrait with anSshTransportthat shells out to the systemssh/scp— reuses the user's keys /~/.ssh/config, adds no new dependency, and stays synchronous. A Tailscale-native transport can slot in behind the same trait later.rocm serve/rocm services— no reimplementation of engine/runtime/device selection; the server runs where the GPU is.rocm --version+ ROCm and: auto-pushes the local binary if the CLI is missing (the whole runtime is one self-contained binary —rocm daemonis the supervisor, there is no separaterocmdto install), and detects-and-fails if ROCm is absent (never installs the multi-GB SDK).rocm services list --json— the machine-readable listing the remote orchestration reads back.Commit 2 — persistent, observable sessions
serveis now async and always-persistent: it starts the managed server (which loads the model in the background), opens a detached tunnel that survives the command exiting, records the session locally, and returns immediately with aPendingstatus and the base URL.rocm remote status [<id>] [--watch]reconciles two independent lifecycles into a table — remote-server health (polled over SSH viaservices list --json) and local tunnel liveness (PID check) — shown as separate columns.attachrebuilds a dropped tunnel without redeploying;stoptears down both.listis intentionally reserved for listing configured hosts (distinct from active sessions).Pending → Healthyprogression comes from the remote's own managed-service lifecycle;statuspolls on demand.sshprocess usingControlMaster=no, so killing its PID deterministically closes the forward.RemoteSessionRecord(one JSON file per session underremote-sessions/), mirroringManagedServiceRecord.Design decisions
ssh/scp, not a Rust SSH crate — synchronous, no new dependency, and the tunnel is a child process that fits the existing detach + PID-tracking model.Tunnelcolumn becomesvia mesh).Test plan
cargo test -p rocm -p rocm-core— green, incl. ~19 new tests: ssh/scp argv construction (incl.ControlMaster=noteardown option), session-id + host resolver (incl. ambiguous-host), status classification + table rendering via an injected status stub (no network), bootstrap probe branches, and the natural-language-planner allowlist routing forremote.cargo clippy --all-targets -- -D warningsclean;cargo fmt --checkclean.rocm: serve →status(Remote Healthy / Tunnel Up) → reachable through the tunnel → kill the tunnel PID → endpoint refused (confirms theControlMaster=noteardown) →statusshows degraded (Tunnel Down / Remote Healthy) →attachrestores it →stoptears down both server and tunnel.Not included / follow-ups
Pending → Healthy) — deferred; only verifiable against live hardware.rocm remote listfor configured hosts, and a Tailscale transport for NAT traversal are future increments.