Push notifications never delivered when app is closed — no Expo push token registered (prime suspect: missing EAS Android FCM V1 credential)

[danljungstrom]

Summary

On the self-hosted/dev setup, push notifications only appear while the app is open; nothing arrives when it's closed. Root cause is not the "app closed" case at all — it's that no Expo push token is ever registered on the server, so the daemon has no one to push to. The notifications seen while open come from a different, foreground-only code path, which masks the broken remote-push path.

Architecture: two independent notification paths

Path	Mechanism	Works when app…
In-app (local)	UI renders a local notification from a live websocket event (activityLocalNotificationBus → sendExpoLocalNotification)	OPEN only — socket disconnects on background (apps/ui/sources/sync/sync.ts:945, apiSocket.disconnect())
Remote push	CLI daemon → Expo → APNs/FCM (dispatchActivityNotification → PushNotificationClient.sendToAllDevicesAsync, apps/cli/src/api/pushNotifications.ts)	Should work CLOSED — currently delivers nothing

App open → local path fires → user sees it. App closed → only remote push can fire → it can't.

Evidence

Server (CT 107, happier.tail6018de.ts.net, serves /v1 + /v2):

• AccountPushToken table = 0 rows, 1 account → daemon fetches an empty recipient list.
• Server logs show only GET /v1/push-tokens (daemon polling); zero POST (app registration) in window.
• Aside (real, intermittent, now mitigated): /v1/push-tokens historically 502 / 15s-timeout repeatedly (daemon log [PUSH] Failed to fetch push tokens: timeout of 15000ms exceeded + Skipping push token fetch during failure cooldown {remainingMs:~27000}). DB was 1.1 GB, SessionMessage = 96.1%. Ran ANALYZE+PRAGMA optimize (backup pre-analyze-2026-06-20T17-01-11Z.bak); queries now ~3 ms.

Client registration path (apps/ui/sources/sync/engine/account/syncAccount.ts:204-332, triggered via InvalidateSync at sync.ts:897-903): registers with all server profiles. It has four silent abort points — any abort before the POST → no token anywhere:

1. loadExpoNotifications() throws → "Push notifications unavailable"
2. permission not granted → return
3. getExpoPushTokenAsync() throws → "Failed to get Expo push token"
4. POST loop (apps/ui/sources/sync/api/session/apiPush.ts → POST /v1/push-tokens)

Zero POSTs + 0 tokens ⇒ aborts before the POST.

Eliminated suspects (verified):

• Wrong/cloud server — app actively syncs to self-hosted (137 sessions, live okhttp sockets).
• OS notification permission — granted (confirmed by owner).
• Native module missing (cf. Dev-channel Android APK ships without in-house native modules (sherpa-native, audio-stream-native) -> Kokoro local TTS/STT unavailable #184) — present: published dev APK (happier-dev-android.apk, build publicdev / dev.happier.app.publicdev) dex contains expo-notifications (7 hits) + FirebaseMessaging (7); sherpa control = 0.
• Firebase config not baked — present: APK resources contain google_app_id / gcm_defaultSenderId and the publicdev app id 1:427065718939:android:fc6fcb80… (project happier-4a4fd).
• EAS wiring — easProjectId 2a550bd7-e4d2-4f59-ab47-dcb778775cee, expo-notifications plugin, googleServicesFile all present and package-matched.

Prime suspect

With module + Firebase + permission + wiring all confirmed good, the only remaining failure point is the runtime getExpoPushTokenAsync() call. Most likely: the EAS project 2a550bd7… has no Android FCM V1 service-account key for dev.happier.app.publicdev. On Android, getExpoPushTokenAsync() calls Expo's backend, which requires FCM credentials; absent → it throws → registration aborts before POST → server stays at 0 tokens.

Verification (either confirms)

• Device: in-app Logs / Notifications → Troubleshooting, search push token → expect Failed to get Expo push token: ….
• EAS: eas credentials -p android (profile publicdev) → "Push Notifications (FCM V1)" assigned, or "None".

Fixes

1. Primary: upload/assign the Android FCM V1 service-account key to the EAS project (no app rebuild needed — server-side credential). Re-verify a token lands in AccountPushToken and a test push reaches a closed app. (Check APNs key for iOS too.)
2. Daemon resilience (apps/cli/src/api/pushNotifications.ts): cache last-known tokens, retry on fetch failure, shorten/skip the 30s cooldown for token fetch so a transient server stall doesn't drop notifications.
3. Server DB retention: prune/retain SessionMessage (96% of DB) to prevent the recurring slow-query/502 cascade.
4. Separate: happier-daemon.service (CT 106) is crashlooping (OOM, ~15 G peak vs MemoryMax) while sessions run under an older root happy-coder daemon — needs its own look.

Acceptance

• AccountPushToken populates after app login.
• Push received with app fully closed.
• [PUSH] Failed to fetch push tokens no longer recurs under normal load.

---

Investigation by Claude Code (read-only repo exploration + live MCP inspection of CT 106/107 + published dev APK).

[danljungstrom]

Device-side confirmation (Android dev build, Troubleshoot screen)

The in-app Push Notification Troubleshoot view on an affected device confirms the prime suspect — the break is the device token read, not account/permission/server:

Row	State
Account setting	Enabled
Permission	Allowed
This device	"Unable to read an Expo push token." ← break
Registered devices	No devices — 0 token(s) on https://happier-dev.tail6018de.ts.net

So registration aborts at getExpoPushTokenAsync() (abort point #3 from the original report), consistent with the EAS project lacking the Android FCM V1 service-account credential for dev.happier.app.publicdev.

Why "Re-register" / "Refresh" appear to do nothing

Reported: pressing Re-register token or Refresh only blips a brief spinner and nothing changes. Traced against origin/dev — both re-run the same throwing getExpoPushTokenAsync(), and the throw is swallowed at three layers so no error ever surfaces and state never changes:

1. apps/ui/sources/components/settings/notifications/PushNotificationTroubleshootingView.tsx:105 — getCurrentExpoPushToken() has an empty catch {} → returns null, no detail.
2. apps/ui/sources/sync/engine/account/syncAccount.ts:244-247 — registerPushTokenIfAvailable catches the throw, does log.log('Failed to get Expo push token: ' + message), then returns (does not re-throw).
3. PushNotificationTroubleshootingView.tsx:228 — the Re-register handler invokes registerPushTokenIfAvailable({ ..., log: { log: () => {} } }) — a no-op logger, so even that diagnostic message is discarded.

reregister's own try/catch (:230-233) only alerts if step 2 throws, which it doesn't. Net: the loading flag toggles (the "brief animation"), state reloads to the same "Unable to read an Expo push token", and the screen looks inert. This is by-construction, not a UI glitch.

Secondary bug surfaced by this

The Troubleshoot screen — whose entire purpose is to diagnose exactly this — discards the underlying getExpoPushTokenAsync error (empty catch at :105; no-op logger passed at :228). It should surface the actual throw message (e.g. via the showErrors path or the row subtitle) so an operator can see why the token read fails instead of a dead-end "Unable to read". Worth tracking separately from the FCM-credential root cause.

To capture the real error message now

The normal sync-triggered registration path (InvalidateSync → registerPushTokenIfAvailable) uses a real logger (unlike this screen's no-op). Open in-app Logs and search Failed to get Expo push token — that line (syncAccount.ts:246) carries the exact throw, which should name the FCM/credential failure and definitively confirm the root cause.

Verified against origin/dev @ 1808b7f57 (2026-06-19). Latest deployed UI build ui-web-v0.2.7-dev.175 (2026-06-20) is post these files' last change.