Usage of Homematic IP API (according to customer support)

[maxigs]

Not a direct issue per-se but i thought it would be interesting to share:

I have been in contact with the customer support of Homematic in the last few days. At the start unrelated to the API (question about notifications in the Homematic app), but it quickly turned into a discussion about my API usage (via home-assistant), based on this library.

Originally, in the beginning of the year, when i switched to Homematic IP (Cloud), i asked the support about using it with non-approved third party integrations, and they seemed ok with it. No official support, but also no problem with it, and they have server side meassures in place to prevent it to cause propblems (the well known rate limit). So all good.

Now, with my contact this week, it turned into something less favourable. Aparently they had a pretty high increase in API traffic and have to start limiting things further. They did not tell me to stop using it, but strongly encuraged me to think about alternative ways, to avoid interruptions.
It's a bit sad, because the homematic devices work incredibly well and easy to set up, especially thanks to this library. Not sure if there is any way to "fix" this, but i would hate to see it break completely.

Note : I currently use it as read only, to display values in home-assistant, not for any updates into the homematic system. But i do have a pretty high number of devices (~50).

[hahn-th]

Hey, thank you for your insights.

It think this is related to an error in this library in april (?) where all of us were banned by HmIP because of an error. I fixed the error. This was my first contact with this library. But there are more libraries around like ioBroker Integration, which provides basically the same functionallity.

I think i should integrate something like not try to reconnect every few milliseconds, when an error occured. Because this is when we get blocked by HmIP because of high API traffic.

If they warned you so strongly, maybe they are working in a fix which makes access by a third party lib impossible. Hopefully not! I mean i can unterstand them. There are a few libraries which are spaming their servers when errors occuring and creating load on their infrastructure. I would not like that either. On the other hand, for them it's a free opertunity to make their products more available.

What did they say about button presses?

[maxigs]

I think i should integrate something like not try to reconnect every few milliseconds

If this is the case it would indeed be very bad. Normally clients should have some logic to give the server a break in such cases. Typically what's called exponential backoff.

The last message i received did not really sound like they would kill it. Not sure if there is anything in the works to further/stronger limit the API usage. There are a couple of alternative implementations/systems, like the local CCU. I'm not sure about the differences though, their cloud service was working perfectly fine and i put all my heating schedules there. But i sill like to have the data in home-assistant for display and analysis.

What did they say about button presses?

This was no topic for me, as i have none. I just mentioned it, as the docs state button presses are causing a lot traffic which might acuse the ip blocking.

[hahn-th]

This was no topic for me, as i have none. I just mentioned it, as the docs state button presses are causing a lot traffic which might acuse the ip blocking.

Ah okay. I mixed something up. This is about pushing buttons in HomeAssistant multiple times. I think it is not my responsiblity to avoid that. But indeed its a point.

Normally clients should have some logic to give the server a break in such cases. Typically what's called exponential backoff.

I gonna have a look at that! If you are firm with python, feel free to create a push request 😄

[maxigs]

I gonna have a look at that! If you are firm with python, feel free to create a push request

Unfortunately my python is very basic, but i will have a look. Need to figure out how the connection handling works here in the first place.

[hahn-th]

It should be function api_call in class AsyncConnection.
If you have discord: agonist#6159

[maxigs]

Just had a quick look over the project. Not 100% sure but i think the issue is here:

https://github.com/hahn-th/homematicip-rest-api/blob/master/homematicip/home.py#L779

This is doing a immediate re-connect when the websocket fails, no waiting time.

The AsyncConnection seems to be used further down, or directly by usig the CLI to make calls – but those are one-off exectutions. The Home class seems to be what is taking care of a persistent connection, used in upstream systems (like home-assistant).

This seems to be a good example of how exponential backoff in python can look like:
https://keestalkstech.com/2021/03/python-utility-function-retry-with-exponential-backoff/

I will try to test it. Might take a few days though, until i manage to get it set up and running for testing though.

[josvo]

Hi

I just found this issue. Looks like some kind of (IP?) Rate limiting is already in place. When I do a basic query for my temperature sensors, I get an error from time to time and the query fails.
The error did not lead me to the right direction

  File "/usr/local/lib/python3.9/site-packages/homematicip/home.py", line 349, in _get_devices
    for id_, raw in json_state["devices"].items():
TypeError: string indices must be integers

However I got a message in the Homeatic Android App, saying that rate limiting is in place and I have to wait 15 minutes before trying again.

A stub of my query (I have about 10 devices in total):

    ...
    global home
    home.get_current_state()
        for g in home.groups:
            if g.groupType == "META":
                for device in g.devices:
                    if isinstance(device, TemperatureHumiditySensorDisplay):
                        write_temperaturehumiditysensor(g.label, device)
                    elif isinstance(device, TemperatureHumiditySensorOutdoor):
                        write_temperaturehumiditysensoroutdoor(g.label, device)
            time.sleep(300)

This is reproducible at the moment.

[hahn-th]

How often do you loop thru your code?
Looks like the json_state string contains wrong data.

[josvo]

How often do you loop thru your code?

Once every 5 min and then I check the values for every device (about 10 devices). I'm logging the current temperatures and humidity to a database.

Looks like the json_state string contains wrong data.

That's the problem. The json_state string is empty (empty string), I checked in debug mode. I get no answer from the API.
I contacted the customer support and ask what exactly the limits are and how I could avoid being limited.

[h3tz]

Same here I just can request every 5-10min all devices to avoid touching limits. That's a mess.

[tschissler]

I also find it annoying that there is this limitation. I think other vendors are managing it pretty well to allow reading data more frequently. I personally would also prefer if I could just read the data directly from my access point but did not find a way to do so yet.

[helje5]

AFAIK the access point is just a bridge to the cloud service, it doesn't run the app server. To have something local, you need to use a CCU3, which you can query as often as you like (but using different protocols as of today).

[tschissler]

Thanks for the quick feedback. I was fearing that I have to buy a CCU3 to properly handle my scenario. I really like the approach Shelly (https://www.shelly.cloud/) is taking here :-) Maybe I should with over to their platform ;-)
Still great what you folks are doing here.

[h3tz]

I am also using Shelly plugs to measure the solar power input, but only for the solar panels. For other devices, i am still using homematic and the 5min availability to keep track of them. I am thinking about using Wireshark to observ the homematic AP conversation. Maybe there is something like a black channel they are using because just using the app 5-second poll is not a problem.

I am just using this one for shellyPlugs: https://github.com/Jan200101/ShellyPy

[helje5]

I would guess that they (the app) uses a WebSocket connection to get value updates from the server. The AP is probably just a proxy (or VPN) and has not a lot of own logic.