Skip to content

Latest commit

 

History

History
232 lines (184 loc) · 11.6 KB

engine-contract.md

File metadata and controls

232 lines (184 loc) · 11.6 KB

Owner Mode engine API

The adapter selects an installed profile by its opaque owner_mode name and sets owner_mode_scale in (0, 1]. The inference engine applies that selection only after authenticating the request. Profile files are installed separately and scoped to a model revision and serving format.

This is the raw engine contract. An Always On FlashAlign gateway selects and signs every routed request, including all tools and subagents, before it reaches this boundary. A caller that sends requests directly to the raw engine is outside that gateway's policy. See direct access.

Optional profile discovery

The setup wizard can read GET <model-base-url>/owner-mode/profiles. An engine that implements this optional extension returns:

{
  "data": [
    {
      "name": "example-profile",
      "model": {"id": "publisher/model", "revision": "pinned-revision", "format": "FP8"},
      "loaded": true
    }
  ]
}

Only loaded profiles are offered. Explicit model.aliases can identify the same served model under additional IDs. The /models response may include revision and format fields so the wizard can reject mismatches, and input, contextWindow and maxTokens to configure the model descriptor.

This is a discovery extension, not an upload API or an existing-engine support claim. A 404/405 falls back to local profiles or an operator-entered profile name. The wizard independently checks standard and selected modes with two one-token Chat Completions calls, including the required signed owner request. It never assumes a listed or structurally valid profile is active.

Request authentication

The engine and adapter share a protected 32-byte signing key. The header is:

X-Owner-Mode-Auth: v1:<unix_seconds>:<random_nonce>:<hex_hmac_sha256>

Sign timestamp + "\n" + nonce + "\n" + sha256(raw_body) with HMAC-SHA256. The engine enforces a 60-second timestamp window and rejects nonce replay throughout that window. Replay-store failure must fail closed. Keep keys and authentication headers out of logs.

The bundled Python engine integrations persist keyed nonce hashes and expiry times across API restarts. vLLM and SGLang use $XDG_STATE_HOME/flashalign/replay/nonces.sqlite3, or ~/.local/state/flashalign/replay/nonces.sqlite3 when that variable is unset. FLASHALIGN_REPLAY_STORE can select a stable private path. The llama.cpp launcher keeps the store in its managed state directory. Keep this state while reusing the deployment key. Its directory must be owned by the serving user with mode 0700, and its database must have mode 0600.

TensorFold uses a private replay directory selected with --owner-replay-dir; direct launches default to a sidecar beside the key file. The wizard chooses a stable per-key directory outside versioned runtime folders. The store holds an exclusive writer lock and durably saves accepted nonce digests before authorizing a request. Preserve it across restarts and upgrades. A new signing key is required when deliberately resetting this state.

The stores contain nonce hashes, expiration times and integrity metadata. A full, unavailable or replaced store denies Owner Mode. Ordinary requests continue in standard mode. Model prompt and KV-cache policy is configured separately.

Missing or invalid authentication uses standard mode. Valid authentication selects the installed profile; an authenticated unknown profile is an error. Return X-Owner-Mode-Effective with the actual selection, including off.

Errors

The adapter recognizes these exact HTTP 400 messages for invalid_owner_mode:

Case Message
Profiles unavailable this server was started without owner-mode profiles: owner_mode must be "off"
Unknown profile owner_mode names no profile this server loaded
Invalid type owner_mode must be "off" or an owner-mode profile name
Invalid scale owner_mode_scale must be a number in (0, 2]

The engine error string for scale is matched verbatim. This adapter accepts only (0, 1]. In Verified contacts mode, only the profiles-unavailable response causes one retry in standard mode. Always On returns the failure without a stock retry. Other errors are returned to the caller. Recheck this contract when upgrading an engine.

Isolation and profile loading

Isolate caches by every effective profile, scale and model-revision dimension, or bypass profile-mode cache reads and writes. Standard requests must not reuse profile-mode cache entries or transforms. The engine must validate the profile format and reject unsupported extensions before advertising support.

An in-memory profile index is a sufficient cache discriminator only while the loaded model and profile set stay immutable. Reloading either must discard those entries, or use new keys that include the changed model/profile content. Keys must also distinguish the effective strength. A profile name alone is insufficient.

An effective-profile header reports selection; it does not itself establish numerical correctness or cache isolation. Engine implementations must test those properties independently. Model profiles never grant tool permissions.

Backend manifests declare maximum_rank, the largest projection-basis rank their loader and runtime accept. Setup checks all operators, including shared operators in grouped profiles and the global rank of writer-group profiles, before preparing an engine. The declaration does not extend a backend's model, site or transform support and is separate from its recorded GPU qualification.

An adapter that supports partial profiles must leave omitted sites unchanged. Accepting a grouped profile must not imply support for every operator family or rank: unsupported requirements, targets and ranks must fail before activation.

Native matrix profiles

The DeepSeek starter uses format: "flashalign.native_wo_b.v1" and transform: "native_matrix_override_v1". It is separate from an output projection. Its manifest declares the model revision, tensor encodings, dimensions, payload checksum, and stock and edited tensor-partition checksums. The runtime validates these before activation.

This operator keeps stock and profile matrices in immutable bindings. An authenticated owner row uses the profile binding at the declared WO-B sites; an off row uses the stock binding. Selection happens before tensor-parallel reduction and residual mixing. Mixed batches must route rows independently, including during speculative verification and graph replay.

The current native profile accepts strength 1.0 only. A loader must reject other strengths or unsupported requirements rather than interpolate, convert the artifact to a projection, or silently serve a different operator. Its cache identity includes the profile content digest. See the native vLLM backend and TensorFold setup for the exact checkpoint and runtime limits. A TensorFold 1.2.0 bundle declares its compatible EXL3 serving base separately from the profile's original target model. It selects the same edited FP8 tensors without converting the profile.

Optional weighted output projection

A profile may declare transform: "weighted_output_projection_v1" and requires: ["per_direction_strength"]. Each site must have scale: 1 and these tensors in profile.safetensors:

Tensor key Type and shape Validation
L{layer}.{site}.Q FP32 [rank, hidden_size] Finite, orthonormal rows
L{layer}.{site}.strengths FP32 [rank] Finite, strictly positive values

For a row batch y of the named matrix's unmodified outputs, apply the operator before residual addition. With request strength alpha = owner_mode_scale:

y_new = y - alpha * ((y @ Q.T) * strengths) @ Q

Each direction keeps its own strength. A loader must reject a non-unit site scale, missing or invalid tensors, or unsupported transform and requires values. Engines must explicitly support this weighted operator; accepting the file schema does not establish runtime support. Do not average the strengths or silently use an ordinary unweighted projection. Standard-mode rows bypass all transform arithmetic, and mixed batches modify only the authenticated rows.

Managed external resources

An engine kit that starts resources outside its process tree declares a shutdown helper in backend.json:

{"external_cleanup": {"schema": "flashalign.external-cleanup.v1", "script": "cleanup.py", "timeout_seconds": 35}}

Setup pins the staged helper's checksum. After reaping managed processes, the caller runs that helper with --state-dir and --deadline-monotonic, within the deployment's total shutdown budget. The helper must verify resource ownership and return JSON with the same schema, status: "stopped" and external_resources_stopped: true. An unknown resource state, timeout or failed check leaves shutdown incomplete and preserves its recovery records. A deployment that declares an external runtime without this contract must be prepared again before managed shutdown can confirm success.

The native Docker helper checks the daemon identity, container ID, image and ownership labels recorded by the launcher. An unresolved creation intent does not establish that no container exists. Other engines can supply their own helper using the same contract; host-only engines need no helper.

Optional runtime status

GET /v1/flashalign/status returns schema: "flashalign.runtime-status.v1". Use a new instance_id when the serving process restarts. generation identifies the active profile generation or alias epoch. Counters cover all requests in that engine instance, including stock requests; they are not per-profile benchmarks. A client must discard deltas across instance, model or generation changes and when a counter decreases.

Every implementation of this status schema includes these counter fields. Use JSON null when a measurement is unavailable; never substitute zero:

Field Meaning
output_tokens_total Cumulative generated output tokens reported by the engine's metrics collector, including reasoning tokens when counted by that collector
ttft_seconds_sum Cumulative seconds to first token for requests observed by the engine's first-token histogram
ttft_seconds_count Number of observations contributing to ttft_seconds_sum

Clients treat a missing counter field as unavailable.

Counters must be finite and nonnegative; token and observation counts must be integers. Return the TTFT sum and count together, or both as null. A real zero is distinct from a missing metric. Output tokens per second is the token counter delta divided by elapsed time. Mean TTFT is the sum delta divided by the observation-count delta. A zero count delta has no mean.

The vLLM adapters read existing in-process vllm:generation_tokens_total and vllm:time_to_first_token_seconds_{sum,count} samples. SGLang uses the matching sglang: names, reading generation tokens from the tensor-parallel leader when it reports one, otherwise from the API process, and first-token observations from the API process. Label series within a collector are summed; replicated workers are not summed. Collection is optional: if the metrics package or samples are absent, status reports null. No metrics package is imported or installed by the status reader. The llama.cpp adapter currently reports these counters as unavailable.

Status may also include graph_mode, expected_sites, installed_sites, exercised_sites, active_requests, and gpu_memory_bytes. The memory field is allocated device memory as reported by the adapter, not an admission budget. These fields describe engine state and do not attest numerical correctness.