Add @tenki compute backend (Tenki Sandbox microVMs)

[AlvaroDeleglise]

PR Type

• Bug fix
• New feature
• Core Runtime change
• Docs / tooling
• Refactoring

Summary

Add @tenki, a new compute backend that runs each flow step inside a disposable Tenki Sandbox Linux microVM — the Tenki analogue of @kubernetes / @batch. It reuses Metaflow's existing remote-execution contract (the bash -c entrypoint + METAFLOW_* env vars) and only swaps the "where it runs" layer (a K8s Job → sb.exec). Entirely additive and opt-in: datastore, metadata, cards, @retry, @resources, @catch, resume, and @pypi/@conda are unchanged.

Issue

Closes #3310 (opened first, per the Core Runtime process).

What's in it

New plugin metaflow/plugins/tenki/:

• tenki_decorator.py — TenkiDecorator(StepDecorator): lifecycle hooks, @resources merge via compute_resource_attributes (the @batch idiom), datastore validation (requires a remote datastore; rejects local / @parallel).
• tenki.py — runner: builds the same bash entrypoint as @kubernetes (get_package_commands + bootstrap_commands + mflog + save_logs), creates a tagged sandbox, runs sb.exec on a background thread while tailing logs from the datastore, interprets the result / SDK exceptions into Metaflow's retry semantics, and tears the sandbox down.
• tenki_cli.py — click trampoline tenki step + flow-scoped tenki list / tenki kill cleanup commands.
• tenki_client.py — thin, lazily-imported wrapper over the tenki SDK (a soft dependency, like the k8s/cloud SDKs — not in setup.py), with a >= 0.5.4 version guard.

Wiring / config:

• metaflow/plugins/__init__.py — register tenki in TRAMPOLINE_CLIS_DESC + STEP_DECORATORS_DESC (2 lines).
• metaflow/metaflow_config.py — a TENKI_* config block (TENKI_API_KEY, TENKI_BASE_URL, TENKI_WORKSPACE_ID, TENKI_CONTAINER_IMAGE, TENKI_CPU, TENKI_MEMORY, TENKI_SANDBOX_INIT_SCRIPT).
• metaflow/plugins/pypi/conda_decorator.py — one line (see Core Runtime rationale below).

Datastores: s3 (incl. S3-compatible endpoints), azure, gs. The backend forwards only the active datastore's cloud credentials into the microVM (a microVM has no ambient cloud identity); for GS it materializes the service-account JSON inside the sandbox. Secret-backend prefixes, METAFLOW_S3_SERVER_SIDE_ENCRYPTION, and OTEL config are forwarded for parity with @kubernetes.

Reproduction

A minimal flow that runs each step on a Tenki microVM (datastore = S3; validated with MinIO reached over a temporary tunnel so the sandbox can read/write it):

# tenki_flow.py
from metaflow import FlowSpec, step, tenki

class TenkiFlow(FlowSpec):
    @tenki(cpu=1, memory=1024)
    @step
    def start(self):
        self.x = 21
        self.next(self.end)

    @tenki(cpu=1, memory=1024)
    @step
    def end(self):
        print("RESULT", self.x * 2)

if __name__ == "__main__":
    TenkiFlow()

pip install tenki
export TENKI_API_KEY=tk_...
export METAFLOW_DEFAULT_DATASTORE=s3
export METAFLOW_DATASTORE_SYSROOT_S3=s3://your-bucket/mf
# self-hosted / non-AWS S3 also needs: export METAFLOW_S3_ENDPOINT_URL=https://...
python tenki_flow.py run

Evidence shows up in the parent console (per-step logs streamed from the datastore), in the artifacts (via the Client API), and in python tenki_flow.py tenki list. On the stock Tenki image each step ran in a fresh microVM, the code package downloaded and metaflow imported from it, and the artifact persisted across steps (self.x = 21 → RESULT 42), ending in Done!.

Core Runtime rationale

The only change to a shared core file is one line in metaflow/plugins/pypi/conda_decorator.py: it keeps a hardcoded allowlist of remote backends for which it leaves the trampoline interpreter alone (interpreter = None). A backend not on that list has its local tenki step trampoline swapped into the resolved @pypi/@conda environment, which then can't import the datastore/metadata dependencies — so @pypi + @tenki would fail. Adding "tenki" (alongside batch / kubernetes / nvidia) gives it the same treatment; no behavior change for other backends. A unit test drives the real CondaStepDecorator and asserts interpreter is None for a @tenki step (fails without the line, passes with it).

Everything else is additive: the new metaflow/plugins/tenki/ package, two registration lines, and the TENKI_* config block.

Why the design is sound:

• Reuses the @kubernetes remote-execution contract verbatim (same bash -c "…" entrypoint via the same shlex.split transform), so remote behavior matches an established backend; the Tenki-specific surface is small (create / exec / teardown + credential forwarding).
• Synchronous exec → background thread + datastore log tailing, preserving Metaflow's live log streaming despite sb.exec blocking.
• Success is the SDK's own result.ok (exit_code == 0 and not signal), not a partial exit-code check, so a signalled task is never read as success.
• Retries owned by the runtime (@retry); one fresh sandbox per attempt.

Failure modes considered

1. Concurrency / retries — one fresh sandbox per attempt (unique name so a retry never collides with a still-terminating prior VM); concurrent foreach (3 parallel microVMs) validated live.
2. Credential leakage — only the active datastore's credentials are forwarded into the VM (s3 creds never reach an azure/gs run); unit-tested.
3. Cleanup scoping (shared Tenki accounts) — tenki list/kill are always scoped to the current flow (sandboxes are tagged metaflow-flow:<flow> at launch), adopting the @kubernetes/@batch parse_cli_options semantics (--my-runs, default to the latest run of the current flow). An unscoped tenki kill can never reach another flow's or user's sandboxes — verified live against Tenki.
4. Orphan / hard-crash teardown — in-process teardown (finally + atexit, retried on the atexit pass, warns if it still fails), tenki kill (tries every teardown method, reports an accurate terminated/failed summary), and a server-side max_duration cap. A hard SIGKILL of the orchestrator relies on tenki kill / the cap.
5. Launch vs. run failures / retryability — launch_job does synchronous network I/O (auth, create), so a transient API/network failure at launch stays retryable under @retry: launch exceptions are classified via the SDK's own retryable flag (UNAVAILABLE / rate-limit → retry; auth / permission / quota / bad-image → no retry), with unknown errors defaulting to non-retryable so @retry never loops on a misconfig. A task killed by a signal (exit 0 but signalled) is a failure; signal/reason/errno are surfaced; both timeout types map to a retryable timeout.
6. Base-image variance — Tenki can't pull public registries and its stock image ships python3 but no python; a startup shim provisions python/pip (no-op on images that already have them) and fails with a clear error if the image has no python3 at all.
7. Command-string escaping — the entrypoint is built and shlex.split the exact way @kubernetes does, so the \"-escaped inner quotes resolve through sb.exec(*argv).

Tests

• Unit tests added — test/unit/test_tenki.py, 48 tests, all green; no regressions to the existing @kubernetes / resource-merge tests. black-clean.
• CI passes.
• Reproduction / live validation provided (below). The tenki SDK is a soft dependency, so CI unit tests run against a stubbed SDK; real-SDK integration can't run in CI without a Tenki account, so that validation is manual.

Unit coverage includes: resource merge; datastore / @parallel validation; sandbox-name and tag sanitization; the _command bash contract + runtime-shim ordering + no-python3 guard; runner behavior against a stubbed SDK (result.ok success, non-zero / OOM(137) / segfault(139) / unknown / signalled results, signal/reason/errno diagnostics, both timeout types, session-lost, permission-denied non-retryable, stderr surfacing); _cleanup retry + warn-on-final-pass; flow-scoped list/kill + parse_cli_options semantics + teardown fallback/reporting; launch-failure classification (permanent vs transient); per-datastore credential isolation + GCP JSON materialization; secrets/S3-SSE/OTEL forwarding; SDK minimum-version enforcement; and the conda_decorator regression above.

Live (real Tenki microVMs; datastore = MinIO over a tunnel): happy path (linear + foreach + join) and plain @tenki steps on the stock image; @retry (fresh sandbox per attempt); concurrency (--max-workers 3); @timeout; --metadata=service (real Postgres-backed service); @catch; resume; @pypi (ran a real package inside the microVM); flow-scoped cleanup; and the tag / kill-scoping isolation.

Not validated live: Azure/GS (forwarding is implemented + unit-tested; needs a real account) and a custom TENKI_CONTAINER_IMAGE (Tenki serves images only from its own registry, so it needs a Metaflow-ready image pushed there).

Non-goals

• @parallel / gang scheduling, GPU, and persistent volumes/snapshots — deferred.
• Tenki native storage as the Metaflow datastore — it is session-bound and can't meet the datastore contract; a shared blob store (S3/Azure/GS) is still needed.
• Changing how compute-backend SDK dependencies are declared — the SDK is kept a soft dependency (matching kubernetes / azure-*); glad to expose an extras_require={"tenki": ["tenki>=0.5.4"]} extra instead if you prefer.
• Core vs. extension packaging — proposed as core (modeled 1:1 on @kubernetes), but structured to move to a metaflow_extensions plugin if you'd rather keep vendor backends out of core.

AI Tool Usage

• No AI tools were used in this contribution
• AI tools were used (describe below)

Tool: Claude Code. Used for: implementation scaffolding, unit tests, and the live-validation harness (MinIO/tunnel setup, e2e probes). Accountability: all code was reviewed and is understood by me; the design decisions, failure modes, and tests reflect my own judgment.

[greptile-apps]

Greptile Summary

This PR adds Tenki Sandbox microVMs as an opt-in Metaflow compute backend. The main changes are:

• A new @tenki step decorator and remote task runner.
• Sandbox lifecycle, retry handling, log streaming, and cleanup commands.
• Scoped flow, run, and user tags for sandbox management.
• Remote datastore credential and configuration forwarding.
• Plugin registration, environment integration, and unit coverage.

Confidence Score: 5/5

This looks safe to merge.

• The updated cleanup commands keep their default scope within the current user's flow and run.
• Sandbox teardown now tries each supported method before reporting failure.
• Launch errors can still be classified when client construction fails.
• No blocking issues remain in the reviewed changes.

Important Files Changed

Filename	Overview
metaflow/plugins/tenki/tenki.py	Adds sandbox launch, remote execution, log streaming, result handling, retry classification, tagging, and cleanup.
metaflow/plugins/tenki/tenki_cli.py	Adds the task trampoline and flow-scoped sandbox list and kill commands.
metaflow/plugins/tenki/tenki_client.py	Adds a lazily loaded Tenki SDK wrapper with version checks and shared client configuration.
metaflow/plugins/tenki/tenki_decorator.py	Adds step lifecycle integration, resource handling, datastore validation, and runtime metadata.
metaflow/plugins/init.py	Registers the Tenki decorator and CLI with the plugin system.
metaflow/plugins/pypi/conda_decorator.py	Preserves the remote trampoline interpreter for Tenki steps using Conda or PyPI environments.

_{Reviews (7): Last reviewed commit: "Classify launch errors even when the Ten..." | Re-trigger Greptile}

[Shriprasad-P]

Review

PR: Add @tenki compute backend (Tenki Sandbox microVMs)

Touched: metaflow/metaflow_config.py, metaflow/plugins/__init__.py, metaflow/plugins/pypi/conda_decorator.py, metaflow/plugins/tenki/__init__.py, metaflow/plugins/tenki/tenki.py, metaflow/plugins/tenki/tenki_cli.py

• Security-sensitive change — please double-check edge cases and defaults.
• CI/tooling change — confirm the pipeline still passes on this branch.
• Diff is fairly large (+2256/-0); a short summary of risk areas from the author would help reviewers.

Commenting as a drive-by reviewer after reading the diff. Happy to look again if maintainers want a deeper pass on a specific file.