bug: chat response latency reported by users

[JeffOtano]

Context

Multiple users from the r/tonalgym launch thread reported the chat feeling slow/laggy. One user specifically called it out:

The only issue I'm having is the chat seems to be lagging.

What to investigate

• Gemini API response times (are we hitting rate limits or cold starts on gemini-3-flash-preview?)
• Context window size — the training snapshot injected via contextHandler could be large for users with extensive history, slowing time-to-first-token
• Tool call chains — the coach has 31 tools and maxSteps: 25. Complex questions may trigger multiple sequential tool calls before producing a visible response
• Streaming behavior — verify the frontend is streaming tokens as they arrive rather than waiting for the full response

Notes

• This may partly be perception (AI coach responses involve tool calls that take time before text starts flowing) vs an actual performance issue.
• Worth checking if latency correlates with users who have long Tonal histories (larger training snapshots).
• Related: feat: multi-provider support via OpenAI-compatible API #85 (multi-provider support) — some users are requesting Claude/other providers, partly motivated by perceived Gemini latency.

[JeffOtano]

Investigation Results

Latency sources (in pipeline execution order)

1. Scheduler queue delay

chat.ts:206,328 - Both createThreadWithMessage and sendMessageToThread use ctx.scheduler.runAfter(0, internal.chat.processMessage). This is an enqueue, not immediate execution. Under load, processMessage may wait 1-5+ seconds before starting.

2. Pre-inference serial overhead (all before first token streams)

Provider key resolution:
resolveUserProviderConfig - 1 DB query + potential AES decrypt (byok.ts:32-85)

Daily budget check:
1 DB query (aiUsage.getDailyTokenUsage)

Training snapshot (buildTrainingSnapshot):
9 parallel DB queries (context.ts:44-82):

• getCurrentStrengthScores
• getMuscleReadiness
• getRecentCompletedWorkouts (limit 20)
• getActiveBlock
• getRecentFeedback
• getActiveGoals
• getActiveInjuries
• getRecentExternalActivities (limit 20)
• getAllMovements (full movement catalog - every message!)

Plus a sequential missed-session detection step (context.ts:318-369) that fetches the week plan, then N parallel fetches for each workout plan ID.

Agent context retrieval:

• Embedding API call to gemini-embedding-001 to vectorize the user's message
• Vector similarity search across all threads (searchOtherThreads: true)
• Text search (textSearch: true)
• Up to 10 results with context windows

None of this work is streamed. Users see the ThinkingIndicator dots for the entire pre-inference duration.

3. First-token latency

Base model response time (gemini-3-flash-preview), typically 1-3s. Outside codebase control.

4. Tool call chains

maxSteps: 25 (coach.ts:188). Each tool call is a full LLM -> tool execution -> LLM round-trip. Complex flows (weekly programming) invoke 3-5+ tools serially. Some tools make live Tonal API calls on cache miss, adding external HTTP latency. Every tool call also writes a recordToolCall mutation via withToolTracking (helpers.ts:13-41).

5. Stream write throttle

throttleMs: 100 (resilience.ts:136) - words arrive at the client at max 10 Hz.

6. Retry delay

3000ms hard wait on transient errors (429, 500-503) in streamWithRetry (resilience.ts:143-180). See #140 for the triple-call issue.

UI handling

• ThinkingIndicator.tsx has phase text at 3s, 8s, and 15s - implicit acknowledgment of expected latency
• ChatInput.tsx:43 uses optimisticallySendMessage for instant user message display
• ChatThread.tsx:43 uses useUIMessages({stream: true}) for reactive streaming

Optimization opportunities

1. Cache getAllMovements - the full catalog doesn't change between messages (synced daily via cron)
2. Parallelize embedding with snapshot build - currently serial
3. Skip vector search for simple follow-ups - not every message needs cross-thread context
4. Reduce recentMessages: 30 - fewer messages in context = faster inference
5. Pre-warm provider config - resolve BYOK key during the mutation, before scheduling the action

[JeffOtano]

Verified against production aiRun telemetry on 2026-05-12 UTC.

Phoenix Cloud cannot be queried from this workspace because PHOENIX_API_KEY is not set in the worktree or root .env.local, so this check used persisted Convex telemetry instead.

30-day production rows by provider/model:

• google.generative-ai/gemini-3-flash-preview: 990 rows, TTFT p50 13.5s, TTFT p90 45.0s, total TTFT p50 16.1s, steps p50 2 / p90 5, 29 errors, 6 fallbacks.
• google.generative-ai/gemini-2.5-flash: 122 rows, TTFT p50 5.5s, TTFT p90 23.7s, total TTFT p50 8.1s, steps p50 1 / p90 3, 2 errors, 11 fallbacks.

Current main already pins Gemini chat/programming to stable gemini-2.5-flash, which addresses the preview-model part of this issue. I would keep this open for now because p90 TTFT is still high and likely tied to tool-call chains / routing overhead; #189 is the next scoped fix to reduce unnecessary tool calls before considering #87 resolved.

[JeffOtano]

Live follow-up on 2026-07-29 after the recent routing/tool-performance deployments: production aiRun currently has 0 rows created after the #590 deployment boundary (2f093285415de5e93bc73f697069e0c7c3247138).

That means the new code is deployed but there is not yet a valid after-sample for TTFT, step count, or tool-chain latency. Keeping this issue open. The next evidence gate is a non-empty production sample segmented by model, with p50/p90 total TTFT and tool steps compared against the prior baseline; deployment success alone is not latency proof.