Harmonizing and improving notifications across platforms

[robbiet480]

Please note, I wrote this while very tired and lacking sleep. It will receive updates over the next few days.

Right now, we live in a similar yet divided world. Home Assistant Companion (hereinafter HAC) for iOS came first and had many years to mature before HAC Android was even a glint in our eyes. Due to that time and somewhat limited thought for the future, iOS made some decisions that don't work well in the Android world. Android is starting to make decisions that will also affect iOS. We should try to cut down on as many differences between the two apps as possible. This will allow us to simplify our documentation but more importantly make it easier for users in households with both Androids and iPhones, in addition to making config sharing easier.

Notifications

The first place to start with is notifications. Right now, both apps support different functionalities. Some functionalities exist because of work developers have done, some will never exist due to OS specific limitations. As much as possible though, we should make it possible for a user to send a notification and 90% of the time it will appear similar across platform. The way that we do this is deriving a schema that both apps can hew to while also providing allowances for platform specific functionality that users want.

As a first step towards that goal, I have come up with this:

{
    "push_token": "",
    "registration_info": {
        "app_id": "io.robbie.HomeAssistant.beta",
        "app_version": "2.0.0 (68)",
        "os_name": "iOS",
        "os_version": "13.1.3"
    },
    "id": "uuid",
    "notification": {
        "message": "The front door is opened",
        "title": "Front Door",
        "badge": 5,
        "image": "https://github.com/home-assistant/home-assistant-assets/blob/master/logo-round-192x192.png?raw=true",
        "tag": "one",
        "click_action": "",
        "actions": [
            {
                "title": "Action 1",
                "id": "action_one"
            }
        ],
        "android": {
            "priority": "high",
            "ttl": "3.5s",
            "sticky": true,
            "visibility": "VISIBILITY_UNSPECIFIED",
            "local_only": true,
            "icon": "ics_launcher",
            "color": "#000000",
            "channel_id": "default",
            "ticker": "Hello World",
            "event_time": "2020-02-14T07:30:21Z",
            "notification_priority": "PRIORITY_DEFAULT",
            "default_sound": true,
            "default_vibrate_timings": true,
            "default_light_settings": true,
            "vibrate_timings": [
                "1s",
                "3.5s"
            ],
            "light_settings": {
                "color": {
                    "red": 1,
                    "green": 1,
                    "blue": 1,
                    "alpha": 0
                },
                "light_on_duration": "3.5s",
                "light_off_duration": "3.5s"
            }
        },
        "ios": {
            "image": {
                "content_type": "image/png",
                "hide_thumbnail": true
            },
            "subtitle": "TwelveTwelve",
            "sound": {
                "name": "default",
                "critical": true,
                "volume": 1
            }
        }
    }
}

As you can see, we have generic fields at the top level (under the notification dictionary) and OS specific fields in their own dictionaries. I also made a spreadsheet that shows common features and their availability across both platforms. We should conform both apps to accept this schema. It will be a pain for users to adapt to one time but will have other benefits to be explained later. The change will be opt in for existing users, at least on iOS. We also have added a UUID to the notification payload to provide lifecycle tracking. The UUID should be generated and stored by Home Assistant Core and used in all follow up events and actions. The actions are now also defined in the notification, whereas previously iOS required defining all possible actions in advance. Actions are now registered with iOS just-in-time before displaying the notification.

Action Events

The next place the apps deviate significantly to the detriment of the user is Notification Action Events. On iOS, this is the event named ios.notification_action_fired. On Android it is mobile_app_notification_action. To start, iOS should adapt the more generic mobile_app_notification_action event name. However, the payloads sent in the events are also very different between platforms and deserve to be unified into one

My proposal is this:

{
    "event_type": "mobile_app_notification_action",
    "data": {
        "action_chosen": "action_one",
        "action_data": {},
        "device_name": "Robbie's iPhone",
        "id": "uuid"
    },
    "origin": "REMOTE",
    "time_fired": "2020-02-02T04:45:05.550251+00:00",
    "context": {
        "id": "abc123",
        "parent_id": null,
        "user_id": "123abc"
    }
}

This provides everything we need to let users determine where the event came from, as well as provides them flexibility to customize by providing the action_data dictionary, a totally user controlled space. They can fill the action_data dict when sending the notification. The id is the same one from the notification example above.

Providing turnkey automation support

Great, so notifications and events are well defined and traceable. That will allow us to implement a new type of automation block which will allow pausing an automation until such time as a user responds to a notification by tapping an action. Users will no longer have to worry about setting up two automations (one to send the notification and one to catch the response event), it will just be handled seamlessly by Home Assistant Core. @balloob came up with this idea and can expand more on it.

Notification Lifecycle Tracking

Now that notifications have unique IDs attached to them, we can monitor them through their entire lifecycle. I propose adding the following new events:

• Notification received by device
• Notification tapped
• Notification dismissed

In addition, I propose that Home Assistant Core start storing these notifications and providing a API to mobile_app implementors to allow viewing notification history.

End to end encrypted notifications

As somewhat of a carrot to get iOS users to opt in to the new notification and event formats, we will begin offering end to end encryption of push notifications on both platforms. On iOS, this is made possible by UNNotificationServiceExtension which will decrypt and map the notification payload before displaying the notification to the user. On Android, we already are in a position where the app is the one processing and displaying the notification, not the OS. mobile_app will be modified to encrypt outgoing notifications using libsodium with the same webhook secret the apps already have. A very minimal forwarder has already been developed and deployed and is available for review here. Here's everything that the forwarder receives:

{
    "encrypted": true,
    "encrypted_data": "j6lwIesVlc+1rYltzw1iFgZZgir3QcZzs+erYXIB6TphTl6bY6xQUYYqhXQ6tGjFAWvMetre/FC89Z85OLExlm3AwX4TzEbbmpLLAoD+zuiG+FugGk756fju58ImTA+Ci5echwgyMNwehKC/RYsbS0V0YieMU34RG/2DT6SDAh++/DmktPBpealTBLnEHRx4/PuNMeqdC7kB5nfkr1gYaBBEXON0R/yLc0sR4qk42RcIyOMd2VrG1apRBSxJizhQVyBrwUWe4z8UdhlXuwLDUhBtieB/xRJSJg1B74ywGvfEtsVJVLlFii8BSsgP/wp/RlcRMLRy8ixa/WPNHLj9YSXLCNYrIvjhTwlxhjAEZncO6fHy0jOuAWbiCrWPgju2q7zWwUxtNO13rGV5pXju++sjiuGn0gKvDBsaw9ZWnVZmR50NeBVha7e4XgUPpS5+Wx78DukpCcuRU/s3N9Q4yCQHkiPtFKszEg6A5FDYb9gGnzD5aSBfKINbNDmh6oBQvN6G2HBsO8E8AdA1cOupMWEFE6b6Y7taHxgNAaX3ik1zYKGe+zDSUSv0SWqIsx8MbbgG+qkSvdNqHswvOw==",
    "push_token": "",
    "registration_info": {
        "app_id": "io.robbie.HomeAssistant.beta",
        "app_version": "2.0.0 (68)",
        "os_version": "13.1.3"
    }
}

and what it sends to the device:

{
	"apns": {
		"payload": {
			"aps": {
				"alert": {
					"title": "Encrypted notification",
					"body": "If you're seeing this, something has gone wrong with encryption"
				},
				"mutable-content": 1
			}
		}
	},
	"data": {
		"encrypted_data": "j6lwIesVlc+1rYltzw1iFgZZgir3QcZzs+erYXIB6TphTl6bY6xQUYYqhXQ6tGjFAWvMetre/FC89Z85OLExlm3AwX4TzEbbmpLLAoD+zuiG+FugGk756fju58ImTA+Ci5echwgyMNwehKC/RYsbS0V0YieMU34RG/2DT6SDAh++/DmktPBpealTBLnEHRx4/PuNMeqdC7kB5nfkr1gYaBBEXON0R/yLc0sR4qk42RcIyOMd2VrG1apRBSxJizhQVyBrwUWe4z8UdhlXuwLDUhBtieB/xRJSJg1B74ywGvfEtsVJVLlFii8BSsgP/wp/RlcRMLRy8ixa/WPNHLj9YSXLCNYrIvjhTwlxhjAEZncO6fHy0jOuAWbiCrWPgju2q7zWwUxtNO13rGV5pXju++sjiuGn0gKvDBsaw9ZWnVZmR50NeBVha7e4XgUPpS5+Wx78DukpCcuRU/s3N9Q4yCQHkiPtFKszEg6A5FDYb9gGnzD5aSBfKINbNDmh6oBQvN6G2HBsO8E8AdA1cOupMWEFE6b6Y7taHxgNAaX3ik1zYKGe+zDSUSv0SWqIsx8MbbgG+qkSvdNqHswvOw==",
		"encrypted": "true",
		"registration_info": "{\"app_id\":\"io.robbie.HomeAssistant.beta\",\"app_version\":\"2.0.0 (68)\",\"os_version\":\"13.1.3\"}"
	},
	"token": ""
}

Obviously our servers are unable to break the encryption and the secret is only ever exchanged directly via app and Home Assistant Core. iOS requires alert and mutable-content to be set to engage our UNNotificationServiceExtension. The user should never actually see that text except if decryption fails for some reason.

The identified downsides to e2e encryption are the following:

• There is no support for Android's ttl or priority fields or APNS headers such as apns-collapse-id or apns-priority. Modifications could be made to notify.mobile_app to allow sending certain whitelisted parameters in the clear.
• All notification translation logic is shipped client side instead of on the server. Therefore, we would either need to commit to only ever adding fields and not removing/modifying existing ones, lest we break older apps, or we version the schema somehow.
• Could make one on one debugging with users harder.

I'm excited to hear the communities thoughts on this proposal. Thanks for reading.

[TomBrien]

I think I'm in favour of pretty much all of this.

The following I think are hugely positive

The actions are now also defined in the notification, whereas previously iOS required defining all possible actions in advance.

Notification Lifecycle Tracking

That will allow us to implement a new type of automation block

End to end encrypted notifications

I'm a little worried about

There is no support for ... APNS headers
But if all that is needed here is to send the APNS headers in pain I think that is fine, we just need to make it clear to users in the docs so they don't use something like 'apns-collapse-id': 'no-one-home-and-frontdoor-open'

I'm keen to hear more on the new blocking event, we have a lot of users who send a notification to all members of say a household and then clear for all when an action (no necessarily in HAC) is taken. I think this could neaten that up a lot

[JBassett]

Notifications

Actions

"actions": [
        {
            "title": "Action 1",
            "id": "action_one"
        }
    ]

When implementing the actions for android I modeled them off the existing html5 standard, maybe stay in line with that? https://developer.mozilla.org/en-US/docs/Web/API/NotificationAction

Light Settings

"light_settings": ""

This isn't just a string, it's a full object: https://firebase.google.com/docs/reference/admin/node/admin.messaging.LightSettings.html

Vibration

"vibrate_timings": ["1s", "3.5s"]

Because we are using the admin sdk to send the notifications we should stick with what it accepts as values: https://firebase.google.com/docs/reference/admin/node/admin.messaging.AndroidNotification.html#optional-vibrate-timings-millis

Action Events

Where does that action_data come from? Assuming it's under an action?

"actions": 
[
  {
    "title": "Action 1",
    "id": "action_one",
    "action_data": { }
  }
]

End to end encrypted notifications

I think this is a great idea overall, however, we need to be careful. FCM has a limit to the payload size. If we aren't careful we might be allowing people to go above the limit causing failed messages. I agree that we might need to identify some meta data about the messages that is send so that we can correctly send the message (ttl and priority).

[dshokouhi]

This is amazing thank you @robbiet480 for putting this together! I think that the main features that people would want shared between the 2 platforms will be aligned with this proposal. It will allow for users to use notify.group as they expect without needing additional service calls unless they really need to add OS specifics. Looking forward to the future developments we come up with on this!

[dshokouhi]

This might also be a good time to maybe enforce some of these standards through the notify platform itself? It seems that when it comes to things like image several integrations do different things. I know the goal here is to keep mobile_app in sync but if the goal is to intermix via notify.group we may want to consider all possibilities?

https://www.home-assistant.io/integrations/slack#slack-service-data
https://www.home-assistant.io/integrations/discord#example-service-call
https://www.home-assistant.io/integrations/hangouts/#service-hangoutssend_message
https://www.home-assistant.io/integrations/html5#data
https://www.home-assistant.io/integrations/nfandroidtv#service-data-for-sending-images

[JBassett]

Looks like someone else brought up the notification delema in architecture as well.
home-assistant/architecture#329

It would be nice if all notifications were a little more consistent.

[robbiet480]

I've updated the suggested payload to cover most suggestions made by @JBassett. I also moved actions into the notification dictionary.

Specific responses for Justin:

• For vibrate timings, I'm following the docs for the REST API since it's what the Admin SDK actually uses.
• action_data can be filled by the user when sending the notification with whatever they want. That data is sent back when a notification action is tapped, verbatim. I don't have a good use case for this right now, but I know people are using it in the iOS app.
• The maximum payload size is currently and will continue to be 4kB. If a user tries to send more than that they will get a error in their console. This behavior is currently in place.

I'm torn on matching the actions entries to HTML5 notification payloads because I don't like how it uses the action key in place of my currently chosen id key. I think it gets confusing. Furthermore, those entries will have to be expanded to support iOS specific things like authentication_required, destructive, etc.

Changing HAC behavior to unify notifications for anything other than the official apps is outside the scope of this proposal.

[robbiet480]

I still haven't come up with a good way to let users provide values that must be sent in the clear, e.g. APNS headers, priority and ttl.

[mario-tux]

I have to say that the new HA wait_for_trigger action is very helpful in order to keep low (one?!) the number of automation to manage the interactive notification of some kind of event (for example: arrival with request to open the door).

[SeanPM5]

So with the upcoming Blueprints feature in Home Assistant, a lot of people are going to be building re-usable and shareable automations that contain notifications involving Mobile App.

Some examples that come to mind:

• Notify when printer ink falls beneath a certain threshold
• Notify when battery level on a device falls beneath a certain threshold
• Notify when robot vacuum needs a replacement part (filter, brush, etc)

For those above examples, it would be useful to simply tap the notification and get taken to an Amazon product page for the relevant product, so that the user doesn't have to go up on a ladder just to find out their door sensor takes a CR2032 battery or whatever. But you cannot make Blueprints like this currently (with tappable link notifications), because the way the iOS and Android companion apps handle URL's are inconsistent.

Other popular blueprints will likely be "water leak detected" and "smoke detected" etc where you'd want to receive a critical notification, but critical notifications differ too depending on platform. Same goes for things like images etc.

With Blueprints just around the corner, unifying these as much as possible would be extremely beneficial, that way the community can create shareable automations that work regardless of what mobile platform they use. This proposal was a great idea before, but it'd be even more useful now in the Blueprint era.

[robbiet480]

Revisiting this... we’re finally doing it. Moving away from FCM to SNS and switching to e2e notifications. Discussing with @zacwest now.

[mdegat01]

Revising an old issue here. For a very long time I used scripts and an automation to normalize schema of sending notifications and receiving notification actions. I audited it this month to see what all I could drop after the efforts to harmonize notifications. I dropped a lot but as of 2022.4.0 here are the things that I still have to normalize between platforms:

1. data.clickAction (android) vs. data.url (ios)
2. In mobile_app_notification_action events, android sets the action to event_data.action, ios sets it to event_data.actionName EDIT: I'm not seeing this one anymore but I swear I was when I wrote this. Confused how this could've changed since I don't see a PR about it but glad for the consistency I suppose.
3. In mobile_app_notification_action events, android includes the tag as event_data.tag, ios does not include it
4. In mobile_app_notification_action events, ios includes anything placed in data.action_data in the original notification in event_data.action_data, android does not
5. In mobile_app_notification_action events, if the action is REPLY, android includes the entered text in event_data.reply_text, ios puts it in event_data.response_info
6. Android supports mdi icons on notifications if set to data.notification_icon. Ios does not support mdi icons
7. To include data from a camera entity, ios wants the camera entity's id in data.entity_id. Android wants data.image set to /api/camera_proxy/{camera entity id}
8. To convey urgency of a notification, android wants you to set data.importance. ios wants you to set data.push.interruption-level. These two have different values as well so you must map them.
9. For image icons, android wants the url in data.icon_url. Ios wants the url in data.image.
   • Note: this one is particularly frustrating because you cannot send a notification to android with the same url in data.icon_url and data.image. If you do the android notification will expand into a low-res version of the icon which you don't want. So you have to send different notification data to the ios and android services for this one.
10. If you want newlines in your notification message, android wants <br> tags, ios wants \n characters

Since I made the list I thought I would share to capture the current state here.