get_device_reading sometimes returns all readings instead of single reading

[d1nd141]

Hi,
If use f.ex.
temp_1_wday = fh.get_device_reading("myWeather", "fc1_day_of_week")
temp_1_wday = temp_1_wday['day_of_week']
temp_1_wday = temp_1_wday['Value']

or
temp_1_code = fh.get_device_reading("myWeather", "fc1_code")
temp_1_code = temp_1_code['fc1_code']
temp_1_code = temp_1_code['Value']

to get a specific reading.
Unfortunately sometimes (not always, and also not everytime the same device) i get all readings.
Not sure if it's FHEM related or python-fhem:

DEBUG:Fhem:Connected, sending...
INFO:Fhem:Sent msg, len=25
ERROR:Fhem:Failed to decode json, exception raised. {
  "Arg":"NAME~myWeather",
  "Results": [
  {
    "Name":"myWeather",
    "PossibleSets":"update",
    "PossibleAttrs":"alias comment:textField-long eventMap:textField-long group room suppressReading userReadings:textField-long verbose:0,1,2,3,4,5 disable event-aggregator event-min-interval event-on-change-reading event-on-update-reading oldreadings stateFormat:textField-long timestamp-on-change-reading DbLogExclude DbLogInclude SplunkLogExclude SplunkLogInclude SyslogLogExclude SyslogLogInclude alexaName alexaRoom cmdIcon devStateIcon devStateStyle fp_Keller fp_Keller_T fp_RDC fp_RDC_T fp_Upstairs fp_Upstairs_T genericDeviceType:security,ignore,switch,outlet,light,blind,thermometer,thermostat,contact,garage,window,lock homebridgeMapping:textField-long icon sortby webCmd webCmdLabel:textField-long widgetOverride userattr",
    "Internals": {
      "API": "YahooWeatherAPI",
      "APIOPTIONS": "transport:https,cachemaxage:600",
      "DEF": "979629",
      "INTERVAL": "3600",
      "LANG": "en",
      "LOCATION": "979629",
      "NAME": "myWeather",
      "NOTIFYDEV": "global",
      "NR": "358",
      "NTFY_ORDER": "50-myWeather",
      "STATE": "T: 1  H: 77  W: 2  P: 995",
      "TYPE": "Weather",
      "UNITS": "c"
    },
    "Readings": {
      "city": { "Value":"Howald", "Time":"2018-12-26 13:46:36" },
      "code": { "Value":"28", "Time":"2018-12-26 13:46:36" },
      "condition": { "Value":"mostly cloudy", "Time":"2018-12-26 13:46:36" },
      "country": { "Value":"Luxembourg", "Time":"2018-12-26 13:46:36" },
      "current_date_time": { "Value":"Wed, 26 Dec 2018 01:00 PM CET", "Time":"2018-12-26 13:46:36" },
      "day_of_week": { "Value":"Wed", "Time":"2018-12-26 13:46:36" },
      "description": { "Value":"Yahoo! Weather for Howald, Luxemburg, LU", "Time":"2018-12-26 13:46:36" },
      "fc10_code": { "Value":"28", "Time":"2018-12-26 13:46:36" },
      "fc10_condition": { "Value":"mostly cloudy", "Time":"2018-12-26 13:46:36" },
      "fc10_date": { "Value":"04 Jan 2019", "Time":"2018-12-26 13:46:36" },
      "fc10_day_of_week": { "Value":"Fri", "Time":"2018-12-26 13:46:36" },
      "fc10_high_c": { "Value":"0", "Time":"2018-12-26 13:46:36" },
      "fc10_icon": { "Value":"mostlycloudy", "Time":"2018-12-26 13:46:36" },
      "fc10_low_c": { "Value":"-2", "Time":"2018-12-26 13:46:36" },
      "fc1_code": { "Value":"28", "Time":"2018-12-26 13:46:36" },
      "fc1_condition": { "Value":"mostly cloudy", "Time":"2018-12-26 13:46:36" },
      "fc1_date": { "Value":"26 Dec 2018", "Time":"2018-12-26 13:46:36" },
      "fc1_day_of_week": { "Value":"Wed", "Time":"2018-12-26 13:46:36" },
      "fc1_high_c": { "Value":"1", "Time":"2018-12-26 13:46:36" },
      "fc1_icon": { "Value":"mostlycloudy", "Time":"2018-12-26 13:46:36" },
      "fc1_low_c": { "Value":"-3", "Time":"2018-12-26 13:46:36" },
      "fc2_code": { "Value":"28", "Time":"2018-12-26 1 Unterminated string starting at: line 45 column 42 (char 2883)
temp_1_code:{}
Traceback (most recent call last):
  File "own.py", line 75, in <module>
    temp_1_code = temp_1_code['fc1_code']
KeyError: 'fc1_code'

or

DEBUG:Fhem:Sending: jsonlist2 NAME~myWeather
DEBUG:Fhem:Connected, sending...
INFO:Fhem:Sent msg, len=25
ERROR:Fhem:Failed to decode json, exception raised. {
  "Arg":"NAME~myWeather",
  "Results": [
  {
    "Name":"myWeather",
    "PossibleSets":"update",
    "PossibleAttrs":"alias comment:textField-long eventMap:textField-long group room suppressReading userReadings:textField-long verbose:0,1,2,3,4,5 disable event-aggregator event-min-interval event-on-change-reading event-on-update-reading oldreadings stateFormat:textField-long timestamp-on-change-reading DbLogExclude DbLogInclude SplunkLogExclude SplunkLogInclude SyslogLogExclude SyslogLogInclude alexaName alexaRoom cmdIcon devStateIcon devStateStyle fp_Keller fp_Keller_T fp_RDC fp_RDC_T fp_Upstairs fp_Upstairs_T genericDeviceType:security,ignore,switch,outlet,light,blind,thermometer,thermostat,contact,garage,window,lock homebridgeMapping:textField-long icon sortby webCmd webCmdLabel:textField-long widgetOverride userattr",
    "Internals": {
      "API": "YahooWeatherAPI",
      "APIOPTIONS": "transport:https,cachemaxage:600",
      "DEF": "979629",
      "INTERVAL": "3600",
      "LANG": "en",
      "LOCATION": "979629",
      "NAME": "myWeather",
      "NOTIFYDEV": "global",
      "NR": "358",
      "NTFY_ORDER": "50-myWeather",
      "STATE": "T: 1  H: 77  W: 2  P: 995",
      "TYPE": "Weather",
      "UNITS": "c"
    },
    "Readings": {
      "city": { "Value":"Howald", "Time":"2018-12-26 13:46:36" },
      "code": { "Value":"28", "Time":"2018-12-26 13:46:36" },
      "condition": { "Value":"mostly cloudy", "Time":"2018-12-26 13:46:36" },
      "country": { "Value":"Luxembourg", "Time":"2018-12-26 13:46:36" },
      "current_date_time": { "Value":"Wed, 26 Dec 2018 01:00 PM CET", "Time":"2018-12-26 13:46:36" },
      "day_of_week": { "Value":"Wed", "Time":"2018-12-26 13:46:36" },
      "description": { "Value":"Yahoo! Weather for Howald, Luxemburg, LU", "Time":"2018-12-26 13:46:36" },
      "fc10_code": { "Value":"28", "Time":"2018-12-26 13:46:36" },
      "fc10_condition": { "Value":"mostly cloudy", "Time":"2018-12-26 13:46:36" },
      "fc10_date": { "Value":"04 Jan 2019", "Time":"2018-12-26 13:46:36" },
      "fc10_day_of_week": { "Value":"Fri", "Time":"2018-12-26 13:46:36" },
      "fc10_high_c": { "Value":"0", "Time":"2018-12-26 13:46:36" },
      "fc10_icon": { "Value":"mostlycloudy", "Time":"2018-12-26 13:46:36" },
      "fc10_low_c": { "Value":"-2", "Time":"2018-12-26 13:46:36" },
      "fc1_code": { "Value":"28", "Time":"2018-12-26 13:46:36" },
      "fc1_condition": { "Value":"mostly cloudy", "Time":"2018-12-26 13:46:36" },
      "fc1_date": { "Value":"26 Dec 2018", "Time":"2018-12-26 13:46:36" },
      "fc1_day_of_week": { "Value":"Wed", "Time":"2018-12-26 13:46:36" },
      "fc1_high_c": { "Value":"1", "Time":"2018-12-26 13:46:36" },
      "fc1_icon": { "Value":"mostlycloudy", "Time":"2018-12-26 13:46:36" },
      "fc1_low_c": { "Value":"-3", "Time":"2018-12-26 13:46:36" },
      "fc2_code": { "Value":"28", "Time":"2018-12-26 13:46:36" },
      "fc2_condition": { "Value":"mostly cloudy", "Time":"2018-12-26 13:46:36" },
      "fc2_date": { "Value":"27 Dec 2018", "Time":"2018-12-26 13:46:36" },
      "fc2_day_of_week": { "Value":"Thu", "Time":"2018-12-26 13:46:36" },
      "fc2_high_c": { "Value":"2", "Time":"2018-12-26 13:46:36" },
      "fc2_icon": { "Value":"mostlycloudy", "Time":"2018-12-26 13:46:36" },
      "fc2_low_c": { "Value":"-1", "Time":"2018-12-26 13:46:36" },
      "fc3_code": { "Value":"30", "Time":"2018-12-26 13:46:36" },
      "fc3_condition": { "Value":"partly cloudy", "Time":"2018-12-26 13:46:36" },
      "fc3_date": { "Value":"28 Dec 2018", "Time":"2018-12-26 13:46:36" },
      "fc3_day_of_week": { "Value":"Fri", "Time":"2018-12-26 13:46:36" },
      "fc3_high_c": { "Value":"2", "Time":"2018-12-26 13:46:36" },
      "fc3_icon": { "Value":"partly_cloudy", "Time":"2018-12-26 13:46:36" },
      "fc3_low_c": { "Value":"-1", "Time":"2018-12-26 13:46:36" },
      "fc4_code": { "Value":"26", "Time":"2018-12-26 13:46:36" },
      "fc4_condition": { "Value":"cloudy", "Time":"2018-12-26 13:46:36" },
      "fc4_date": { "Value":"29 Dec 2018", "Time":"2018-12-26 13:46:36" },
      "fc4_day_of_week": { "Value":"Sat", "Time":"2018-12-26 13:46:36" },
      "fc4_high_c": { "Value":"2", "Time":"2018-12-26 13:46:36" },
      "fc4_icon": { "Value":"cloudy", "Time":"2018-12-26 13:46:36" },
      "fc4_low_c": { "Value":"0", "Time":"2018-12-26 13:46:36" },
      "fc5_code": { "Value":"26", "Time":"2018-12-26 13:46:36" },
      "fc5_condition": { "Value":"cloudy", "Time":"2018-12-26 13:46:36" },
      "fc5_date": { "Value":"30 Dec 2018", "Time":"2018-12-26 13:46:36" },
      "fc5_day_of_week": { "Value":"Sun", "Time":"2018-12-26 13:46:36" },
      "fc5_high_c": { "Value":"5", "Time":"2018-12-26 13:46:36" },
      "fc5_icon": { "Value":"cloudy", "Time":"2018-12-26 13:46:36" },
      "fc5_low_c": { "Value":"2", "Time":"2018-12-26 13:46:36" },
      "fc6_code": { "Value":"28", "Time":"2018-12-26 13:46:36" },
      "fc6_condition": { "Value":"mostly cloudy", "Time":"2018-12-26 13:46:36" },
      "fc6_date": { "Value":"31 Dec 2018", "Time":"2018-12-26 13:46:36" },
      "fc6_day_of_week": { "Value":"Mon", "Time":"2018-12-26 13:46:36" },
      "fc6_high_c": { "Value":"5", "Time":"2018-12-26 13:46:36" },
      "fc6_icon": { "Value":"mostlycloudy", "Time":"2018-12-26 13:46:36" },
      "fc6_low_c": { "Value":"2", "Time":"2018-12-26 13:46:36" },
      "fc7_code": { "Value":"28", "Time":"2018-12-26 13:46:36" },
      "fc7_condition": { "Value":"mostly cloudy", "Time":"2018-12-26 13:46:36" },
      "fc7_date": { "Value":"01 Jan 2019", "Time":"2018-12-26 13:46:36" },
      "fc7_day_of_week": { "Value":"Tue", "Time":"2018-12-26 13:46:36" },
      "fc7_high_c": { "Value":"1", "Time":"2018-12-26 13:46:36" },
      "fc7_icon": { "Value":"mostlycloudy", "Time":"2018-12-26 13:46:36" },
      "fc7_low_c": { "Value":"-1", "Time":"2018-12-26 13:46:36" },
      "fc8_code": { "Value":"28", "Time":"2018-12-26 13:46:36" },
      "fc8_condition": { "Value":"mostly cloudy", "Time":"2018-12-26 13:46:36" },
      "fc8_date": { "Value":"02 Jan 2019", "Time":"2018-12-26 13:46:36" },
      "fc8_day_of_week": { "Value":"Wed", "Time":"2018-12-26 13:46:36" },
      "fc8_high_c": { "Value":"1", "Time":"2018-12-26 13:46:36" },
      "fc8_icon": { "Value":"mostlycloudy", "Time":"2018-12-26 13:46:36" },
      "fc8_low_c": { "Value":"-1", "Time":"2018-12-26 13:46:36" },
      "fc9_code": { "Value":"28", "Time":"2018-12-26 13:46:36" },
      "fc9_condition": { "Value":"mostly cloudy", "Time":"2018-12-26 13:46:36" },
      "fc9_date": { "Value":"03 Jan 2019", "Time":"2018-12-26 13:46:36" },
      "fc9_day_of_week": { "Value":"Thu", "Time":"2018-12-26 13:46:36" },
      "fc9_high_c": { "Value":"1", "Time":"2018-12-26 13:46:36" },
      "fc9_icon": { "Value":"mostlycloudy", "Time":"2018-12-26 13:46:36" },
      "fc9_low_c": { "Value":"-2", "Time":"2018-12-26 13:46:36" },
      "humidity": { "Value":"77", "Time":"2018-12-26 13:46:36" },
      "icon": { "Value":"mostlycloudy", "Time":"2018-12-26 13:46:36" },
      "isConverted": { "Value":"0", "Time":"2018-12-26 13:46:36" },
      "lastError": { "Value":"", "Time":"2018-12-26 13:46:36" },
      "lat": { "Value":"49.584999", "Time":"2018-12-26 13:46:36" },
      "long": { "Value":"6.14139", "Time":"2018-12-26 13:46:36" },
      "pressure": { "Value":"995", "Time":"2018-12-26 13:46:36" },
      "pressure_trend": { "Value":"0", "Time":"2018-12-26 13:46:36" },
      "pressure_trend_sym": { "Value":"=", "Time":"2018-12-26 13:46:36" },
      "pressure_trend_txt": { "Value":"steady", "Time":"2018-12-26 13:46:36" },
      "pubDate": { "Value":"Wed, 26 Dec 2018 01:00 PM CET", "Time":"2018-12-26 13:46:36" },
      "pubDateComment": { "Value":"okay", "Time":"2018-12-26 13:46:36" },
      "pubDateRemote": { "Value":"Wed, 26 Dec 2018 01:00 PM CET", "Time":"2018-12-26 13:46:36" },
      "pubDateTs": { "Value":"1545825600", "Time":"2018-12-26 13:46:36" },
      "region": { "Value":" Luxemburg", "Time":"2018-12-26 13:46:36" },
      "state": { "Value":"T: 1  H: 77  W: 2  P: 995", "Time":"2018-12-26 13:46:36" },
      "temp_c": { "Value":"1", "Time":"2018-12-26 13:46:36" },
      "temperature": { "Value":"1", "Time":"2018-12-26 13:46:36" },
      "validity": { "Value":"up-to-date", "Time":"2018-12-26 13:46:36" },
      "visibility": { "Value":"16", "Time":"2018-12-26 13:46:36" },
      "wind": { "Value":"2", "Time":"2018-12-26 13:46:36" },
      "wind_chill": { "Value":"1", "Time":"2018-12-26 13:46:36" },
      "wind_condition": { "Value":"Wind: SE 2 km/h", "Time":"2018-12-26 13:46:36" },
      "wind_direction": { "Value":"135", "Time":"2018-12-26 13:46:36" },
      "wind_speed": { "Value":"2", "Time":"2018-12-26 13:46:36" Expecting object: line 125 column 63 (char 8687)
temp_1_wday:{}
Traceback (most recent call last):
  File "own.py", line 68, in <module>
    temp_1_wday = temp_1_wday['day_of_week']
KeyError: 'day_of_week'

DEBUG:Fhem:Connected, sending...
INFO:Fhem:Sent msg, len=25
ERROR:Fhem:Failed to decode json, exception raised. {
  "Arg":"NAME~myWeather",
  "Results": [
  {
    "Name":"myWeather",
    "PossibleSets":"update",
    "PossibleAttrs":"alias comment:textField-long eventMap:textField-long group room suppressReading userReadings:textField-long verbose:0,1,2,3,4,5 disable event-aggregator event-min-interval event-on-change-reading event-on-update-reading oldreadings stateFormat:textField-long timestamp-on-change-reading DbLogExclude DbLogInclude SplunkLogExclude SplunkLogInclude SyslogLogExclude SyslogLogInclude alexaName alexaRoom cmdIcon devStateIcon devStateStyle fp_Keller fp_Keller_T fp_RDC fp_RDC_T fp_Upstairs fp_Upstairs_T genericDeviceType:security,ignore,switch,outlet,light,blind,thermometer,thermostat,contact,garage,window,lock homebridgeMapping:textField-long icon sortby webCmd webCmdLabel:textField-long widgetOverride userattr",
    "Internals": {
      "API": "YahooWeatherAPI",
      "APIOPTIONS": "transport:https,cachemaxage:600",
      "DEF": "979629",
      "INTERVAL": "3600",
      "LANG": "en",
      "LOCATION": "979629",
      "NAME": "myWeather",
      "NOTIFYDEV": "global",
      "NR": "358",
      "NTFY_ORDER": "50-myWeather",
      "STATE": "T: -3  H: 93  W: 5  P: 994",
      "TYPE": "Weather",
      "UNITS": "c"
    },
    "Readings": {
      "city": { "Value":"Howald", "Time":"2018-12-26 20:46:45" },
      "code": { "Value":"29", "Time":"2018-12-26 20:46:45" },
      "condition": { "Value":"partly cloudy", "Time":"20 Unterminated string starting at: line 26 column 54 (char 1445)
Traceback (most recent call last):
  File "own.py", line 104, in <module>
    temp_2_low = temp_2_low['Value']
KeyError: 'Value'

[Andre0512]

Hello,
the python interface always loads all data of a device, because FHEM offers no other possibility. The data is then filtered by python-fhem.
Therefore, from a performance perspective, it is better to filter the readings yourself if you need several readings from one device, like this:

data = fh.get_device_reading("myWeather", ["fc1_day_of_week", "fc1_code"], value_only=True)
temp_1_code = data["temp_1_code"]
temp_1_wday = data["temp_1_wday"]

Your problem is, that the json data coming from FHEM are not well-formed and can not be parsed by python. I assume that this error is in jsonlist2 of FHEM or in telnet transmission.
Did you test if the error also occurs with http?

Maybe this could be catched by requesting the data again when a decode error occours @domschl ? Maybe with a parameter that sets how often?

To catch this problem for the moment you could also do something like this:

# Try 5 times to get correct data
for x in range(5):
    data = fh.get_device_reading("myWeather", ["fc1_day_of_week", "fc1_code"], value_only=True)
    if data:
        break

[domschl]

This is most probably caused by a performance-problem with telnet. In telnet, there are no markers that indicate the actual end of a block, so timing has to be used to infer end-of-transmission. [I can reproduce this error, if I set fh.get_device_reading('myWeather', 'fc1_day_of_week', timeout=0.001), reading via telnet]

The errors indicate a JSON decode of an incomplete block, so somehow the system doesn't finish the transmission within timeout=0.1 default constraints.

I would try:

• Set timeout=0.5 (experiment with the value):
  e.g. fh.get_device_reading('myWeather', 'fc1_day_of_week', timeout=0.5)
• Use @Andre0512 recommendations to 'bundle' requests.
• Consider switching to HTTP, which doesn't have those protocol limitations (and also performs better.)
• Alternatively: If all you want to do is reading values, have a look at FhemEventQueue. You can set it up with server-side filters to deliver only the values you are interested in, only when Fhem actively sends them.

In the end, when using telnet, proper configuration of timeout is a requirement, which unfortunately cannot be avoided.

[d1nd141]

Hi again,
i tried with http, and everything works fine.
Will probably stay on http connect.
As i'm reading at least 26 values, i will check with Queue. (to display on a 7.5 e-ink display)

Thanks 4 your help and explications!

[domschl]

@d1nd141, you're welcome, and thanks @Andre0512 for the help and advice.