[sx126x,sx127x] FineOffset 433/868MHz RF sensor protocol/gateway

[LorbusChris]

Describe the problem you have/What new integration you would like

I'm using the the below yaml to read data from a FineOffset WH65 weather station using an sx127x which was whipped up by @swoboda1337 in #2886. That already works great!

However I think it'd be great to extend this to support more sensors. FineOffset sensors are sold under many brand names including Ecowitt, Froggit, Ambient Weather and many more. I only have access to the one 868MHz WH65 weather station, but I've extended the yaml with some TODOs - they shouldn't be hard to implement with https://github.com/merbanan/rtl_433 as a reference.

Eventually, this could live in a dedicated component so that discovered sensors on the same band could be recognized and added dynamically.

Other protocols have in the past been added to remote_receiver/remote_transmitter, maybe going that route would be applicable here too?

Please describe your use case for this integration and alternatives you've tried:

esphome:
  name: "fineoffset-rf-sensor-gateway"
  friendly_name: FineOffset RF Sensor Gateway

esp32:
  board: adafruit_feather_esp32_v2
  framework:
    type: arduino

spi:
  clk_pin: GPIO5
  mosi_pin: GPIO19
  miso_pin: GPIO21

sx127x:
  dio0_pin: GPIO27 # IRQ
  cs_pin: GPIO33
  rst_pin: GPIO15
  frequency: 868350000
  modulation: FSK
  bandwidth: 100_0kHz
  bitrate: 17240
  bitsync: true
  rx_floor: -50
  rx_start: true
  sync_value: [0x2d, 0xd4]
  preamble_size: 5
  preamble_detect: 2
  preamble_errors: 8
  preamble_polarity: 0xAA
  payload_length: 17
  packet_mode: true
  on_packet:
    then:
      - lambda: |-
          ESP_LOGD("lambda", "Received message: %s", format_hex_pretty(x).c_str());
          ESP_LOGD("fineoffset", "Type: 0x%02X", x[0]);
          ESP_LOGD("fineoffset", "ID: %d", x[1]);


          // Type 0x90
          // Fine Offset Electronics WS90 Weather Station
          if (x[0] == 0x90) {
            // TODO
          }


          // Type 0x80
          // Fine Offset Electronics WS80 Weather Station
          if (x[0] == 0x80) {
            // TODO
          }


          // Type 0x68
          // Fine Offset Electronics WS68 Wind Sensor
          if (x[0] == 0x68) {
            // TODO
          }


          // Type 0x57
          // Fine Offset Electronics WH57 Lightning Sensor
          // Ambient Weather WH31L
          if (x[0] == 0x57) {
            // TODO
          }


          // Type 0x55
          // Fine Offset Electronics WH55 Water Leak Sensor
          if (x[0] == 0x55) {
            // TODO
          }


          // Type 0x53
          // Fine Offset Electronics WH53 Temperature Sensor
          // Ecowitt WH53/WH0280/WH0281A.
          if (x[0] == 0x53) {
            // TODO
          }


          // Type 0x52
          // Fine Offset Electronics WH31E Radio Controlled Clock (Datetime Data)
          // Ambient Weather WH31E, Alecto WS-1200 (v2 DCF)
          if (x[0] == 0x52) {
            // TODO
          }


          // Type 0x51
          // Fine Offset Electronics WH51 Soil Moisture Sensor
          // Ecowitt WH51
          if (x[0] == 0x51) {
            uint8_t sum = 0;
            for (uint32_t i = 0; i < 13; i++) {
              sum += x[i];
            }
            if (sum == x[13]) {
              uint8_t crc = 0;
              for (uint8_t byte = 0; byte < 12; byte++) {
                crc ^= x[byte];
                for (uint8_t bit = 0; bit < 8; bit++) {
                    if (crc & 0x80) {
                        crc = (crc << 1) ^ 0x31;
                    } else {
                        crc = (crc << 1);
                    }
                }
              }
              if (crc == x[12]) {
                uint32_t boost = (x[4] & 0xe0) >> 5;
                uint32_t battery_mv = (x[4] & 0x1f) * 100;
                uint32_t ad_raw = (((int16_t)x[7] & 0x01) << 8) | (int16_t)x[8];
                uint32_t moisture = x[6];
                float battery_ok = (battery_mv - 700) / 900.0f; // assume 1.6V (100%) to 0.7V (0%) range
                ESP_LOGD("lambda", "WH51: boost %u, battery_ok %.1f, battery_mv %u, ad_raw %u, moisture %u",
                         boost, battery_ok, battery_mv, ad_raw, moisture);
              }
            }
          }


          // Type 0x46
          // Fine Offset Electronics WH46 Air Quality Sensor
          // Ecowitt WH46
          if (x[0] == 0x46) {
            // TODO
          }


          // Type 0x45
          // Fine Offset Electronics WH45 Air Quality Sensor
          // Ecowitt WH45, Ecowitt WH0295, Froggit DP250, Ambient Weather AQIN
          if (x[0] == 0x45) {
            // TODO
          }


          // Type 0x42
          // Fine Offset Electronics WH0290 Air Quality Sensor
          // Ecowitt WH0290, Ecowitt WH41, Ambient Weather PM25, Misol PM25
          if (x[0] == 0x42) {
            // TODO
          }


          // Type 0x40
          // Fine Offset Electronics WH40/WH5360
          // Ecowitt WH40, Ecowitt WH5360B
          if (x[0] == 0x40) {
            // TODO
          }


          // Type 0x30, Type 0x37
          // Fine Offset Electronics WH31E/WH31B Temperature/Humidity Sensor
          // Ecowitt WH31, Ambient Weather WH31E
          if (x[0] == 0x30 || x[0] == 0x37) {
            uint8_t sum = 0;
            for (uint32_t i = 0; i < 6; i++) {
              sum += x[i];
            }
            if (sum == x[6]) {
              uint8_t crc = 0;
              for (uint8_t byte = 0; byte < 6; byte++) {
                crc ^= x[byte];
                for (uint8_t bit = 0; bit < 8; bit++) {
                  if (crc & 0x80) {
                    crc = (crc << 1) ^ 0x31;
                  } else {
                    crc = (crc << 1);
                  }
                }
              }
              if (crc == 0) {
                uint32_t batt_low = ((x[2] & 0x04) >> 2);
                uint32_t channel  = ((x[2] & 0x70) >> 4) + 1;
                uint32_t temp_raw = ((x[2] & 0x03) << 8) | (x[3]);
                uint32_t humidity = x[4];
                float temp_c = (temp_raw - 400) * 0.1f;
                ESP_LOGD("lambda", "WH31: batt_low %u, channel %u, humidity %u, temp %.1f",
                         batt_low, channel, humidity, temp_c);
              }
            }
          }


          // Type 0x34
          // Fine Offset Electronics WN34 Temperature Sensor
          if (x[0] == 0x34) {
            // TODO
          }


          // Type 0x24
          // Fine Offset Electronics WH24/WH65 Weather Station
          // Ecowitt WH65, Ecowitt WS69, Misol WS2320
          if (x[0] == 0x24 && x.size() >= 17) {
            uint8_t sum = 0;
            for (uint32_t i = 0; i < 16; i++) {
              sum += x[i];
            }
            if (sum == x[16]) {
              uint8_t crc = 0;
              for (uint8_t byte = 0; byte < 15; byte++) {
                crc ^= x[byte];
                for (uint8_t bit = 0; bit < 8; bit++) {
                  if (crc & 0x80) {
                    crc = (crc << 1) ^ 0x31;
                  } else {
                    crc = (crc << 1);
                  }
                }
              }
              if (crc == x[15]) {
                int wind_direction  = x[2] | (x[3] & 0x80) << 1; // range 0-359 deg, 0x1ff if invalid

                int battery_alert   = (x[3] & 0x08) >> 3;

                int temp_raw        = (x[3] & 0x07) << 8 | x[4]; // 0x7ff if invalid
                float temperature   = (temp_raw - 400) * 0.1f; // range -40.0-60.0 C

                int humidity        = x[5];                      // 0xff if invalid

                float wind_speed_factor, rain_cup_count;
                // Wind speed factor is 1.12 m/s (1.19 per specs?) for WH24, 0.51 m/s for WH65
                // Rain cup each count is 0.3mm for WH24, 0.01inch (0.254mm) for WH65
                // m/s = 3.6 km/h
                // TODO figure this out automatically
                if (0) { // WH24
                  wind_speed_factor = 1.12f * 3.6f;
                  rain_cup_count = 0.3f;
                } else { // WH65
                  wind_speed_factor = 0.51f * 3.6f;
                  rain_cup_count = 0.254f;
                }

                int wind_speed_raw  = x[6] | (x[3] & 0x10) << 4; // 0x1ff if invalid
                // Wind speed is scaled by 8, wind speed = raw / 8 * 1.12 m/s (0.51 for WH65)
                float wind_speed = wind_speed_raw * 0.125f * wind_speed_factor;

                int gust_speed_raw  = x[7];             // 0xff if invalid
                // Wind gust is unscaled, multiply by wind speed factor 1.12 m/s
                float gust_speed = gust_speed_raw * wind_speed_factor;

                int rainfall_raw    = x[8] << 8 | x[9]; // rain tip counter
                float rainfall   = rainfall_raw * rain_cup_count; // each tip is 0.3mm / 0.254mm

                int uv_raw          = x[10] << 8 | x[11];               // range 0-20000, 0xffff if invalid
                // UV value   UVI
                // 0-432      0
                // 433-851    1
                // 852-1210   2
                // 1211-1570  3
                // 1571-2017  4
                // 2018-2450  5
                // 2451-2761  6
                // 2762-3100  7
                // 3101-3512  8
                // 3513-3918  9
                // 3919-4277  10
                // 4278-4650  11
                // 4651-5029  12
                // >=5230     13
                int uvi_upper[] = {432, 851, 1210, 1570, 2017, 2450, 2761, 3100, 3512, 3918, 4277, 4650, 5029};
                int uv_index = 0;
                while (uv_index < 13 && uvi_upper[uv_index] < uv_raw) ++uv_index;

                int light_raw     = x[12] << 16 | x[13] << 8 | x[14]; // 0xffffff if invalid
                // Lux = value/10
                double light_intensity = light_raw * 0.1; // range 0.0-300000.0lux

                ESP_LOGD("WH65", "Temperature: %.1f °C", temperature);
                ESP_LOGD("WH65", "Humidity: %u %%", humidity);
                ESP_LOGD("WH65", "Wind Speed: %.1f km/h", wind_speed);
                ESP_LOGD("WH65", "Gust Speed: %.1f km/h", gust_speed);
                ESP_LOGD("WH65", "Wind Direction: %.1f °", wind_direction);
                ESP_LOGD("WH65", "Rainfall: %.1f mm", rainfall);
                ESP_LOGD("WH65", "UV Index: %d", uv_index);
                ESP_LOGD("WH65", "Light Intensity: %.1f lx", light_intensity);
                ESP_LOGD("WH65", "Battery Alert: %d", battery_alert);

                // known sensor ID  
                if (x[1] == 0xF1) {
                  id(sensor_temperature)->publish_state(temperature);
                  id(sensor_humidity)->publish_state(humidity);
                  id(sensor_wind_speed)->publish_state(wind_speed);
                  id(sensor_gust_speed)->publish_state(gust_speed);
                  id(sensor_wind_direction)->publish_state(wind_direction);
                  id(sensor_rainfall)->publish_state(rainfall);
                  id(sensor_uv_index)->publish_state(uv_index);
                  id(sensor_light_intensity)->publish_state(light_intensity);
                  id(sensor_battery_alert)->publish_state(battery_alert);
                }
              }
            }
          }


          // Type 0xE
          // Fine Offset Electronics WH26 Temperature/Humidity/Pressure Sensor
          // Ecowitt WH32 (WH32 outdoor), Ecowitt WN32, Froggit DP40, Ambient Weather WH32E
          if ((x[0] & 0xf0) == 0xe0) {
            // TODO
          }


          // Type 0xD
          // Fine Offset Electronics WH25 Temperature/Humidity Sensor
          // Ecowitt WH32B (WH32 indoor), Ecowitt WN32P, Ambient Weather WH32B, Garni G090HP
          if ((x[0] & 0xf0) == 0xd0) {
            // TODO
          }


          // Type 0xB
          // Fine Offset Electronics WH1080/WH3080 Weather Station (Datetime Data)
          if ((x[0] & 0xf0) == 0xb0) {
            // TODO
          }


          // Type 0xA
          // Fine Offset Electronics WH1080/WH3080 Weather Station (Weather Data)
          if ((x[0] & 0xf0) == 0xa0) {
            // TODO
          }


          // Type 0x7
          // Fine Offset Electronics WH1080/WH3080 Weather Station (UV/Light Data)
          if ((x[0] & 0xf0) == 0x70) {
            // TODO
          }


          // Type 0x6
          // Fine Offset Electronics WH1050 Weather Station (Datetime Data)
          // TFA Dostmann 30.3151
          if ((x[0] & 0xf0) == 0x60) {
            // TODO
          }


          // Type 0x5
          // Fine Offset Electronics WH1050 Weather Station (Weather Data)
          // TFA Dostmann 30.3151
          if ((x[0] & 0xf0) == 0x50) {
            // TODO
          }


          // Type 0x4
          // Fine Offset Electronics WH2/WH5 Temperature/Humidity/Rain Sensor
          // Ecowitt WH2, Ecowitt WH5, Agimex Rosenborg 66796, ClimeMET CM9088, TFA Dostmann 30.3157
          if ((x[0] & 0xf0) == 0x40) {
            // TODO
          }


          // Type 0x3
          // Fine Offset Electronics WH0530 Temperature/Rain Sensor
          // Agimex Rosenborg 35926, Alecto WS-1200 (v1, v2)
          if ((x[0] & 0xf0) == 0x30) {
            // TODO
          }


          // Type 0x0C
          // Alecto WS3500/WS4500 Weather Station (Rain Data)
          // Unitec W186-F
          if ((x[0] & 0x0f) == 0x0c) {
            // TODO
          }


          // Type 0x08
          // Alecto WS3500/WS4500 Weather Station (Wind Data)
          // Unitec W186-F
          if ((x[0] & 0x0f) == 0x08) {
            // TODO
          }


          // Type 0x05
          // Ambient Weather F007TH/F012TH Temperature/Humidity Sensor
          // TFA Dostmann 30.3208.02, SwitchDoc F016TH
          if ((x[0] & 0x0f) == 0x05) {
            // TODO
          }

logger:

api:
  encryption:
    key: !secret api_encryption_key

ota:
  - platform: esphome
    password: !secret ota_password

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

binary_sensor:
  - platform: template
    name: Battery Alert
    id: sensor_battery_alert
    icon: mdi:battery-alert

sensor:
  - platform: template
    name: "Temperature"
    id: sensor_temperature
    state_class: "measurement"
    unit_of_measurement: "°C"
    device_class: "temperature"

  - platform: template
    name: "Humidity"
    id: sensor_humidity
    state_class: "measurement"
    unit_of_measurement: "%"
    device_class: "humidity"

  - platform: template
    name: "Wind Direction"
    id: sensor_wind_direction
    state_class: "measurement"
    unit_of_measurement: "°"
    device_class: "wind_direction"

  - platform: template
    name: "Wind Speed"
    id: sensor_wind_speed
    state_class: "measurement"
    unit_of_measurement: "km/h"
    device_class: "wind_speed"

  - platform: template
    name: "Gust Speed"
    id: sensor_gust_speed
    state_class: "measurement"
    unit_of_measurement: "km/h"
    device_class: "wind_speed"

  - platform: template
    name: "Rainfall"
    id: sensor_rainfall
    state_class: "measurement"
    unit_of_measurement: "mm"
    icon: "mdi:weather-rainy"

  - platform: template
    name: "UV Index"
    id: sensor_uv_index
    state_class: "measurement"
    icon: "mdi:sun-wireless"

  - platform: template
    name: "Light Intensity"
    id: sensor_light_intensity
    state_class: "measurement"
    unit_of_measurement: "lx"
    device_class: "illuminance"

  - platform: internal_temperature
    name: "Internal Temperature"

Additional context

[swoboda1337]

Could make a new API but also could pass data to a fine offset component like this, would work with any radio:

on_packet:
    then:
      - fine_offset.process_packet:
             data: !lambda 'return x;'

[LorbusChris]

Unfortunately on_packet swallows the preamble, the length of which is needed to determine what sensor data is received. Maybe this is easier done with the remote_receiver component? Or are there any reasons against using that?

[swoboda1337]

Using continuous mode and remote receiver with FSK is not really viable. The OOK receiver is gated by signal strength and doesn't toggle the GPIO when the signal isn't strong enough. The FSK receiver always sends data continuously on the gpio at a very fast rate (bitrate is faster than OOK). The receiver is never gated. You can sometimes decode FSK in continuous but it's not consistent and you miss most packets.

[LorbusChris]

Could the preamble length be exposed in continuous mode somehow?

[swoboda1337]

Could the preamble length be exposed in continuous mode somehow?

You would see the preamble in continuous mode but as I said above it really doesn't work. At least not reliability. You might get 5% of the packets if you are lucky. Probably less. Have to stick with packet mode.

[LorbusChris]

sorry, I meant to say: is there a way to get the recorded preamble length in packet mode?

[swoboda1337]

sorry, I meant to say: is there a way to get the recorded preamble length in packet mode?

No. Preamble is meant to help afc and agc. It is not meant to be decoded. The receiver will miss the first bit anyways before the control loops settle.

[CaptainDouche]

Hi Guys,
this reminds me on how to read TX29 thermo/hygro sensors using a RFM69:
in this module, you also enter into some registers, what frequency, etc, and especially as well the preamble and the sync word - which is also 2D,D4 btw.
For TX29+RFM69 there is no need to know any length indication hidden in the preamble:
once the RFM has seen the sync word, it records all that comes in, and when its Fifo is full, you get an interrupt, read out the entire Fifo, and then your "hi level" firmware looks at the entire fifo - and since it knows the packet format, its no problem to find out where the packet ends in the fifo.
You dont miss any packets that way, ju just need to know when the packet ends and the rest of the fifo only contains noise.
Note that even after end of the transmission, the receiver fills the fifo (with noise), so it is guaranteed that after a packet began, fifo gets full, you get informed, and see all the packets. the not-knowing where the packet ends is as i said no problem, software can easy detect it.