obs(api): Redis storage layer is missing key metrics across lock, transition, scan, and data-quality paths #3607

Background

The Redis storage layer (packages/api/internal/sandbox/storage/redis) has
grown a solid foundation of expiration-index and publisher metrics over recent
PRs (#3604, #3606). However, several high-traffic paths — distributed locking,
state transitions, background scans, and PubSub delivery — still emit nothing
to OTel. The gaps below are ordered by operational impact.

Missing metrics

P0 — silent failures, no alert possible

① Lock wait latency + timeout counter (locker.go · Obtain)

Every sandbox state change (StartRemoving) must acquire a distributed lock.
Obtain can block until lockTimeout (1 min) before returning ctx.Err(),
but no metric records how long callers waited or how often the timeout was hit.
Lock contention is the most direct cause of sandbox API tail latency and is
currently invisible.

Proposed:

• api.redis_storage.lock.wait_duration — histogram (ms), from Obtain entry to lock acquired
• api.redis_storage.lock.timeout_total — counter, incremented on ctx.Err() / deadline exceeded

---

② Transition callback failure counter (state_change.go · createCallback)

The callback that finalises every state transition has three failure paths
(lock acquire failure, SET result key failure, DEL transition key failure),
each logging only Warn. A failing callback leaves waiters blocked until the
30 s result-key TTL expires. There is no metric to page on.

Proposed:

• api.redis_storage.transition.callback_failures_total — counter with
  reason=lock_failed|set_result_failed|del_transition_key_failed

---

③ forEachSandboxBatch skipped-team counter (scan.go · forEachSandboxBatch)

Both healExpirationIndex and AllRunningItems call forEachSandboxBatch.
Per-team scan errors are silently continued with a Warn log. A team that
fails every pass never gets its sandboxes healed and never appears in
AllRunningItems, yet there is no counter to alert on.

Proposed:

• api.redis_storage.scan.teams_skipped_total — counter, incremented on each skipped team

---

P1 — critical path latency invisible

④ State transition end-to-end duration (state_change.go · StartRemoving)

The full transition — lock acquire → Lua write → callback invoked — is the
core SLI for sandbox kill/pause/resume. No histogram captures this latency.
A rising P99 is the earliest signal of Redis or lock pressure.

Proposed:

• api.redis_storage.transition.duration — histogram (ms), from StartRemoving entry to callback invocation

---

⑤ Joined transition counter (state_change.go · handleExistingTransition)

joined.Mark is already called when a concurrent caller joins an in-flight
transition, but the call is not wired to an OTel counter. Join rate is a proxy
for hot-sandbox concurrency; a spike indicates the same sandbox is being
operated on by many callers simultaneously.

Proposed:

• api.redis_storage.transition.joined_total — counter

---

⑥ WaitForTransition fallback poll counter (state_change.go · waitForTransition)

When a PubSub notification is missed, waitForTransition falls back to a 1 s
ticker. Each fallback tick means a waiter is blocked for an extra second. This
counter, correlated with publisher.dropped, gives the full picture of PubSub
reliability degradation.

Proposed:

• api.redis_storage.transition.wait_fallback_polls_total — counter

---

P2 — data quality and race visibility

⑦ ErrExecutionMismatch counter (state_change.go · StartRemoving)

ErrExecutionMismatch is returned when a pinned removal finds the sandbox has
been resumed under a new execution ID. Currently error-logged only. A high rate
indicates client retry storms or rapid pause→resume cycling.

Proposed:

• api.redis_storage.transition.execution_mismatch_total — counter

---

⑧ Corrupt sandbox record counter (scan.go · fetchSandboxBatch)

JSON unmarshal failures during SSCAN+MGET are Warn-logged and skipped. A
batch of corrupt records would flood logs and silently shrink heal/reconcile
coverage. A counter makes the signal alertable.

Proposed:

• api.redis_storage.scan.corrupt_records_total — counter

---

Summary table

#	Metric	Type	File · Function	Priority
①	lock.wait_duration	histogram	locker.go · Obtain	P0
①	lock.timeout_total	counter	locker.go · Obtain	P0
②	transition.callback_failures_total	counter	state_change.go · createCallback	P0
③	scan.teams_skipped_total	counter	scan.go · forEachSandboxBatch	P0
④	transition.duration	histogram	state_change.go · StartRemoving	P1
⑤	transition.joined_total	counter	state_change.go · handleExistingTransition	P1
⑥	transition.wait_fallback_polls_total	counter	state_change.go · waitForTransition	P1
⑦	transition.execution_mismatch_total	counter	state_change.go · StartRemoving	P2
⑧	scan.corrupt_records_total	counter	scan.go · fetchSandboxBatch	P2

All metric names follow the existing api.redis_storage.* namespace established
in #3604 and #3606.

Related

• obs(api): record team sandbox index SET size as a histogram #3604 — per-team index SET size histogram
• obs(api): no metrics on global expiration ZSET size or evictor sweep latency #3605 / obs(api): add expiration index size and evictor sweep duration metrics #3606 — expiration index size + sweep duration
• perf(api): cjson.decode in startTransitionScript Lua script blocks Redis event loop under concurrent evictions #3602 / perf(api): avoid cjson.decode in startTransitionScript hot path #3603 — cjson.decode hot path

/cc @jakubno @dobrac @ValentaTomas