Validation: Performance Benchmarks

[ucguy4u]

Performance Verification

Benchmark
Cold start, Warm start, Model loading time, First transcript latency, Summary generation time, Embedding speed, Speaker detection latency, Memory usage, CPU usage, GPU/NPU utilization, Battery consumption, Storage growth

Engineering Practices to Adopt

• Every feature must include telemetry (stored locally if offline).
• Every background task must be resumable.
• Every download must be recoverable.
• Every expensive operation must expose progress.
• Every AI task must be cancellable.
• Every model must support integrity verification.
• Every database schema change must include migrations.
• Every release must include performance benchmarks.
• Every release must include battery impact measurements.
• Every release must include regression tests for recording, downloads, transcription, and search.

[ucguy4u]

Critical Agent Gate

Problem: Airo has release-level performance expectations, but the repository does not yet provide a single deterministic benchmark workflow or artifact checklist covering startup, model warmup, transcript latency, resource usage, and battery impact.
User / actor: Release and DevEx engineers, QA automation, and maintainers shipping Android/mobile releases.
Framework or application layer: Mixed, primarily framework/release infrastructure with application-level validation targets.
Owning agent: Release and DevEx Agent.
Reviewing agents: QA Automation Agent, Meeting Intelligence Agent, Brain / AI Agent.
Impacted modules/files: docs/release/*, Makefile, app/integration_test/*, potential benchmark scripts under scripts/.
Base branch/worktree: confirmed from latest origin/main: yes.
Open questions: Which benchmark items can be host-only vs. require a physical Android device, and which metrics are mandatory release blockers versus informational?
Decision: Ready.

Cross-Agent Contract

Provider agent: Release and DevEx Agent.
Consumer agent: QA Automation Agent and domain owners shipping release-critical AI features.
Interface/API: benchmark runbook, local commands, and release checklist artifacts.
Input shape: target flavor/device, benchmark scenario, output artifact location.
Output shape: reproducible commands plus a benchmark report artifact with measured values and pass/fail thresholds.
State changes: release docs updated, developer commands added, optional benchmark artifacts generated locally.
Errors: missing Android device/emulator opt-in, missing model artifacts, unsupported platform.
Permissions: local filesystem, Android device or approved emulator path.
Privacy/redaction: benchmark outputs must not include prompt contents, raw transcripts, or private user data.
Persistence: benchmark summaries stored in release docs/artifacts, not in app runtime state.
Versioning/migration: docs/process only unless new scripts are introduced.
Tests required: docs completeness, command smoke check, and at least one host-runnable benchmark-related verification path.

Deterministic Use Cases / Automation Flow

1. Maintainer runs a single documented command set before release.
2. The workflow states which checks are host-only and which require physical Android.
3. The maintainer records cold start, warm path, model warmup, and first-response timing in a release artifact.
4. The release checklist blocks sign-off unless benchmark artifacts and battery notes are attached.

[ucguy4u]

Opened follow-up PR: #531

What this PR adds:

• release benchmark runbook in docs/release/PERFORMANCE_BENCHMARKS.md
• benchmark report generator script for artifacts/performance/...
• Make targets for host Gemini Nano warmup validation and Android startup timing capture
• release checklist updates so benchmark/battery evidence is a blocking artifact

What remains before #520 can close:

• run the documented benchmark flow on a release candidate
• attach real Android startup, memory, storage, and battery measurements
• capture the remaining meeting/transcript-related latency values in the generated report