GemmaControl: Private On-Device WhatsApp AI Productivity Agent

GemmaControl is an English-only, private, on-device Android productivity agent for WhatsApp notification workflows. The app captures new WhatsApp notifications after user permission, organises them into a local actionable inbox, lets FunctionGemma propose approved tool calls, saves local follow-ups, priority flags, encrypted local reminders, and supports safe user-confirmed WhatsApp replies.

Built entirely as a native application for Android 16 (API Level 36), GemmaControl keeps WhatsApp capture, storage, voice handling, and model inference local on a Xiaomi Redmi 13 5G handset, utilizing Google's on-device LiteRT-LM SDK and the FunctionGemma MobileActions 270M model.

---

🚀 Key Functional Scope

1. Notification Ingestion & Actionable Inbox

• MessagingStyle Parsing: Extracts chat history lists and sender authors dynamically from bundled notification structures.
• Group Separation: Differentiates group conversation headers from the individual message authors.
• Content Classification: Tags captured notifications as text, media placeholders, missed calls, hidden content, system notifications, or unknown content before local reads and storage.
• Event Deduplication: Uses deterministic parser candidates that are transformed into keyed HMAC dedupeToken values before Room storage.
• Privacy Constraints: Captured alerts are parsed in memory, and stored locally only under explicit storage consent toggles.

2. Local Task Management & Productivity

• Follow-Ups: Save notification events as unresolved local tasks (create_follow_up_from_message), list pending tasks, and mark follow-ups complete.
• Prioritization: Mark captured WhatsApp messages as high or normal priority inside the local inbox.
• Reminders: Schedule encrypted local reminder rows and WorkManager notification delivery (schedule_reminder_for_message).
• Priorities: Flag and pin important messages inside the local Compose inbox.
• Inbox Cleanup: Dismiss and hide noise from the local inbox without clearing notifications inside WhatsApp.

3. Safe Execution Boundaries (Android 16 Target)

• Adaptive Read-Aloud: Read latest WhatsApp messages targets currently active notification rows, while Read stored messages uses the local history index. Reads 1-3 messages directly, summarizes larger sets by chat, and supports continue, chat-specific reads, summarize, and important-only voice phrases.
• Edge-to-Edge Jetpack Compose: Mandatory edge-to-edge UI display with full support for window insets and IME (keyboard) overlays.
• Predictive Back Navigation: Built using predictive-back-compatible AndroidX navigation frameworks and custom BackHandler hooks.
• Verified Click-to-Chat Intents: Securely launches click-to-chat ACTION_VIEW intents utilizing E.164 phone formats.
• Live RemoteInput Reply: Fetches active system notifications by key in real-time, verifying active status and reply parameters before execution.
• UI Confirmation Sheets: Requires a physical user tap on Compose modals before firing direct reply intents. Auto-sending is structurally blocked.

---

🛠️ Architecture & Security Design

• Clean MVVM Architecture: Separates data structures (Room SQLite database), notification handlers, LiteRT engines, and Jetpack Compose views.
• Encryption at Rest: Encrypts sensitive message texts and drafted responses inside Room SQLite utilizing AES-GCM 256-bit encryption keys protected by the Android Keystore.
• Truthful Media Handling: WhatsApp photo/video/audio/document notifications are treated as placeholders only. The app never claims to inspect media contents unless actual media bytes become a future explicit feature.
• Scoped Network Access: android.permission.INTERNET is declared only for explicit FunctionGemma .litertlm model binary downloads. WhatsApp notification data, prompts, tool calls, and replies stay local.
• Local Reminder Notifications: android.permission.POST_NOTIFICATIONS is declared only for user-confirmed local reminder alerts on Android 13+.
• LiteRT-LM Native Tool Boundary: Registers a deliberately small native ToolSet with automatic LiteRT tool callbacks enabled. Tool callbacks only capture typed proposals/results; Kotlin still owns permission checks, safety routing, and all user-confirmed execution.

---

📂 Project Structure

├── docs/                     # Comprehensive technical documentation
│   ├── ARCHITECTURE.md       # On-device data flow, boundaries, and OS limits
│   ├── CAPABILITY_MATRIX.md  # Android permissions, capability sources, and tool mapping
│   ├── GENERIC_NOTIFICATION_SOURCE_ABSTRACTION.md # WhatsApp-first source model for future SMS/Gmail/Phone/Calendar
│   ├── XIAOMI_HYPEROS_RELIABILITY_CHECKLIST.md # Physical reliability checklist for HyperOS capture
│   ├── GEMMA_MODEL_REFERENCE.md # Gemma model/runtime selection reference
│   ├── GEMMA_ASSISTANT_USAGE_PLAN.md # App-specific model usage roadmap
│   ├── OFFLINE_MODEL_STRATEGY_DECISION.md # No-download model architecture decision for #109
│   ├── EMBEDDING_GEMMA_SEMANTIC_MEMORY_PLAN.md # No-download semantic retrieval spike for #113
│   ├── LOCAL_SUMMARIZATION_MODEL_DECISION.md # No-download Gemma 4/Gemma 3n decision for #117
│   ├── FUNCTION_GEMMA_FINE_TUNING_DECISION.md # No-training FunctionGemma routing decision for #116
│   ├── MEDIA_UNDERSTANDING_BOUNDARY.md # No-model media placeholder/user-selected file boundary for #118
│   ├── ASSISTANT_SAFETY_POLICY.md # Deterministic safety and ShieldGemma gate decision for #119
│   ├── ACCESSIBILITY_SERVICE_EVALUATION.md # V1/V2 decision and safety policy for #122
│   ├── DEVICE_INFO.md        # Current connected handset capability snapshot
│   ├── PHONE_TESTING_TOOLKIT.md # Local laptop and phone testing tools, commands, and setup
│   ├── PRODUCT_SCOPE.md      # V1 functional capabilities and non-goals
│   ├── REAL_DEVICE_ASSISTANT_TEST_MATRIX.md # Manual assistant/device validation matrix
│   ├── TOOL_REGISTRY.md      # Parameter type schemas for the 16 registry tools
│   ├── NOTIFICATION_PARSING.md # MessagingStyle parsers and keyed dedupe tokens
│   ├── SECURITY_AND_PRIVACY.md # Key cryptography and search memory trade-offs
│   └── DEVICE_VALIDATION.md  # Physical telemetry parameters and test milestones
├── GemmaControl/             # Native Android 16 Kotlin Application
│   ├── app/                  # Main application module
│   │   ├── build.gradle.kts  # Target and compile configurations (API 36)
│   │   └── src/main/         # Kotlin Compose sources, listener services, and manifest
│   └── build.gradle.kts      # Project root Gradle declaration
├── website/                  # Static TypeScript download website
├── plan.md                   # High-level product roadmap
├── detailed_plan.md          # Actionable phase-by-phase implementation blueprints
└── README.md                 # Project cockpit reference

---

🌐 Download Website

The website/ directory contains a static TypeScript site for distributing GemmaControl without committing app binaries.

cd website
npm run verify
npm run serve

The primary APK button points to the canonical GitHub Releases artifact configured in website/src/release-links.ts. APK/AAB/EXE/MSI files remain ignored and must not be committed.

---

AI Contributor Instructions

This repository is developed against one known physical device profile unless a later issue explicitly expands the matrix:

• Phone: Xiaomi Redmi 13 5G / 2406ERN9CI
• OS: HyperOS OS3.0, Android 16, API 36
• ABI/RAM: arm64-v8a, approximately 6 GB physical RAM
• ADB serial used in current validation docs: 1431df87
• Android app package: com.example.gemmacontrol
• Android project root: GemmaControl/

Before changing behavior, read:

• plan.md for the product roadmap and phase boundaries.
• detailed_plan.md for implementation sequencing and safety constraints.
• docs/ARCHITECTURE.md for runtime boundaries.
• docs/GEMMA_ASSISTANT_USAGE_PLAN.md and docs/GEMMA_MODEL_REFERENCE.md for model roles.
• docs/TOOL_REGISTRY.md for the allowed assistant tool surface.
• docs/PHONE_TESTING_TOOLKIT.md for laptop, ADB, Appium, Maestro, scrcpy, and manual phone testing setup.
• docs/REAL_DEVICE_ASSISTANT_TEST_MATRIX.md for real-device validation scenarios.

Development rules:

• Keep WhatsApp data local. Do not log, commit, or summarize private message contents in issues, PRs, docs, or final reports.
• Do not make direct WhatsApp replies automatic. Reply sending must remain behind a visible user confirmation step.
• Treat FunctionGemma as the local tool-routing model, not as the source of unrestricted app control.
• Kotlin validation owns permissions, safety checks, schema validation, and execution decisions even when LiteRT native callbacks propose tool calls.
• Current media notifications are placeholders only. Do not claim image/audio/video understanding unless a future explicit media feature handles actual user-selected bytes.
• Use the issue -> feat/<issue-number> branch -> PR -> merge -> delete branch -> clean main workflow for implementation work.
• Do not commit APK/AAB/EXE/MSI/model binaries. Large model files belong outside git-tracked source unless a release process explicitly says otherwise.

Minimum verification for code changes:

cd C:\Users\Admin\Desktop\gemma_control\GemmaControl
.\gradlew.bat :app:testDebugUnitTest
.\gradlew.bat :app:assembleDebug

Real-device validation, when the phone is connected:

adb devices -l
adb install -r -d .\app\build\outputs\apk\debug\app-debug.apk
adb shell am start -n com.example.gemmacontrol/.MainActivity
scrcpy --serial 1431df87 --stay-awake --turn-screen-on

Use Appium or Maestro for repeatable UI flows, and use manual testing for real WhatsApp incoming messages, voice prompts, Xiaomi permission prompts, and confirmed replies. Full setup and command examples are documented in docs/PHONE_TESTING_TOOLKIT.md.

---

📱 Hardware Telemetry (Discovered via ADB)

• Manufacturer/Model: Xiaomi 2406ERN9CI (Redmi 13 5G)
• OS Environment: Android 16 (API Level 36)
• Processor (ABI): arm64-v8a
• Total Physical RAM: approximately 6 GB (5,531,208 kB)
• Connection Port: Connected via USB (Serial: 1431df87)
• Verified Package: package:com.whatsapp is confirmed present on the profile.