[sx126x,sx127x] Hoymiles HMS Inverter RF protocol/gateway

[LorbusChris]

Describe the problem you have/What new integration you would like

I'd like to use esphome as a gateway to communicate with an Hoymiles HMS solar inverter on 865MHz using an sx126x/sx127x RF transceiver in order to:

• read power data
• set power limit
• on/off/restart

Please describe your use case for this integration and alternatives you've tried:

I need support configuring the sx127x radio and decoding the Hoymiles RF data.
I'm using the following yaml:

esphome:
  name: "hoymiles-gateway"
  friendly_name: Hoymiles Inverter RF Gateway

esp32:
  board: adafruit_feather_esp32_v2
  framework:
    type: arduino

spi:
  clk_pin: GPIO5
  mosi_pin: GPIO19
  miso_pin: GPIO21

sx127x:
  dio0_pin: GPIO27 # IRQ
  #dio1_pin: GPIO26 # A0
  #dio2_pin: GPIO25 # A1
  cs_pin: GPIO33
  rst_pin: GPIO15
  frequency: 865000000
  modulation: FSK
  shaping: GAUSSIAN_BT_0_5
  bitrate: 20000
  bitsync: true
  deviation: 20000
  #crc_enable: true
  rx_floor: -90
  rx_start: true
  sync_value: [0x4D, 0x48, 0x5A, 0x48]
  preamble_size: 30
  preamble_detect: 2
  preamble_errors: 8
  preamble_polarity: 0xAA
  payload_length: 0
  packet_mode: true
  on_packet:
    then:
      - lambda: |-
          std::string res;
          size_t len = x.size();
          char buf[5];
          for (size_t i = 0; i < len; i++) {
            if (i > 0) {
              res += " ";
            }
            sprintf(buf, "%02X", x[i]);
            res += buf;
          }
          ESP_LOGD("lambda", "%s", res.c_str());

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: !secret api_encryption_key

ota:
  - platform: esphome
    password: !secret ota_password

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

#sensor:
#  - platform: template
#    name: "Temperature (°C)"
#    id: inverter_temperature
#  - platform: template
#    name: "Power (W)"
#    id: inverter_power

Additional context

Two implementation for Hoymiles gateways are:

• https://github.com/lumapu/ahoy
• https://github.com/tbnobody/OpenDTU

There a CMT2300A transceiver initialized with the following config is used:

Details

;---------------------------------------
;  CMT2300A Configuration File
;  Generated by CMOSTEK RFPDK 1.53_Update3
;  2023.03.24 17:29
;---------------------------------------
; Mode                      = Advanced
; Part Number               = CMT2300A
; Frequency                 = 860.000 MHz
; Xtal Frequency            = 26.0000 MHz
; Demodulation              = GFSK
; AGC                       = On
; Data Rate                 = 20.0 kbps
; Deviation                 = 20.0 kHz
; Tx Xtal Tol.              = 19 ppm
; Rx Xtal Tol.              = 20 ppm
; TRx Matching Network Type = 20 dBm
; Tx Power                  = +20 dBm
; Gaussian BT               = 0.5
; Bandwidth                 = Auto-Select kHz
; CDR Type                  = Counting
; CDR DR Range              = NA
; AFC                       = On
; AFC Method                = Auto-Select
; Data Representation       = 0:F-low 1:F-high
; Rx Duty-Cycle             = Off
; Tx Duty-Cycle             = Off
; Sleep Timer               = Off
; Sleep Time                = NA
; Rx Timer                  = Off
; Rx Time T1                = NA
; Rx Time T2                = NA
; Rx Exit State             = STBY
; Tx Exit State             = STBY
; SLP Mode                  = Disable
; RSSI Valid Source         = PJD
; PJD Window                = 8 Jumps
; LFOSC Calibration         = On
; Xtal Stable Time          = 155 us
; RSSI Compare TH           = NA
; Data Mode                 = Packet
; Whitening                 = Disable
; Whiten Type               = NA
; Whiten Seed Type          = NA
; Whiten Seed               = NA
; Manchester                = Disable
; Manchester Type           = NA
; FEC                       = Enable
; FEC Type                  = x^3+x^2+1
; Tx Prefix Type            = 0
; Tx Packet Number          = 1
; Tx Packet Gap             = 32
; Packet Type               = Variable Length
; Node-Length Position      = First Node, then Length
; Payload Bit Order         = Start from msb
; Preamble Rx Size          = 2
; Preamble Tx Size          = 30
; Preamble Value            = 170
; Preamble Unit             = 8-bit
; Sync Size                 = 4-byte
; Sync Value                = 1296587336
; Sync Tolerance            = None
; Sync Manchester           = Disable
; Node ID Size              = NA
; Node ID Value             = NA
; Node ID Mode              = None
; Node ID Err Mask          = Disable
; Node ID Free              = Disable
; Payload Length            = 32
; CRC Options               = IBM-16
; CRC Seed                  = 0 crc_seed
; CRC Range                 = Entire Payload
; CRC Swap                  = Start from MSB
; CRC Bit Invert            = Normal
; CRC Bit Order             = Start from bit 15
; Dout Mute                 = Off
; Dout Adjust Mode          = Disable
; Dout Adjust Percentage    = NA
; Collision Detect          = Off
; Collision Detect Offset   = NA
; RSSI Detect Mode          = At PREAM_OK
; RSSI Filter Setting       = 32-tap
; RF Performance            = High
; LBD Threshold             = 2.4 V
; RSSI Offset               = 26
; RSSI Offset Sign          = 0

esphome output from yaml above:

[21:09:06][D][lambda:050]: 70 6A E8 C6 9C B4 2E 8C B9 A8 C0 06 CB 00 D0 04 68 C0 00 5C 00 01 A4 66 9A 1A 65
[21:09:06][D][lambda:050]: 72 5A E8 C6 9C B4 2E 8C B9 A8 C0 06 CB 00 D0 04 60 03 40 00 00 34 00 D1 B7 00 0D
[21:09:06][D][lambda:050]: 72 5A E8 C6 9C B4 2E 8C B9 A8 C0 06 CB 00 D0 04 60 05 F9 65 B9 2C 00 00 0D 1C E8
[21:09:06][D][lambda:050]: E6 5A E8 C6 9C B4 2E 8C B9 A8 C0 06 CB 00 D0 04 68 C6 80 00 00 00 00 00 00 00

expected (logged from OpenDTU):

21:09:06.075 > Nothing received, resend whole request
21:09:06.075 > TX RealTimeRunData 865.00 MHz --> 15 83 76 24 23 80 19 06 08 80 0B 00 68 6D 6C D1 00 00 00 00 00 00 00 00 EC 56 F9 
21:09:06.122 > Interrupt received
21:09:06.128 > RX 865.00 MHz --> 95 83 76 24 23 80 19 06 08 01 00 01 01 1B 01 1A 00 01 00 01 00 04 00 03 00 05 F3 | -78 dBm
21:09:06.167 > Interrupt received
21:09:06.174 > RX 865.00 MHz --> 95 83 76 24 23 80 19 06 08 02 EC B9 00 06 7D E8 02 65 02 62 09 16 13 88 00 00 B7 | -78 dBm
21:09:06.215 > Interrupt received
21:09:06.226 > RX 865.00 MHz --> 95 83 76 24 23 80 19 06 08 83 00 00 00 00 00 00 00 BD 00 11 CA 69 7C | -78 dBm
21:09:06.582 > RX Period 

so

// actual (logged on esphome/sx127x):
msg 1: 70 6A E8 C6 9C B4 2E 8C B9 A8 C0 06 CB 00 D0 04 68 C0 00 5C 00 01 A4 66 9A 1A 65
msg 2: 72 5A E8 C6 9C B4 2E 8C B9 A8 C0 06 CB 00 D0 04 60 03 40 00 00 34 00 D1 B7 00 0D
msg 3: 72 5A E8 C6 9C B4 2E 8C B9 A8 C0 06 CB 00 D0 04 60 05 F9 65 B9 2C 00 00 0D 1C E8
msg 4: E6 5A E8 C6 9C B4 2E 8C B9 A8 C0 06 CB 00 D0 04 68 C6 80 00 00 00 00 00 00 00

// expected (logged on OpenDTU):
msg 1: 15 83 76 24 23 80 19 06 08 80 0B 00 68 6D 6C D1 00 00 00 00 00 00 00 00 EC 56 F9 
msg 2: 95 83 76 24 23 80 19 06 08 01 00 01 01 1B 01 1A 00 01 00 01 00 04 00 03 00 05 F3
msg 3: 95 83 76 24 23 80 19 06 08 02 EC B9 00 06 7D E8 02 65 02 62 09 16 13 88 00 00 B7
msg 4: 95 83 76 24 23 80 19 06 08 83 00 00 00 00 00 00 00 BD 00 11 CA 69 7C

It recognizes the sync word, but I'm probably missing something obvious from the CMT2300A init, like the x^3+x^2+1 polynomial FEC (Forward Error Correction). Any guidance on how to properly receive/decode/parse this data would be appreciated.

ref: lumapu/ahoy#1630

@swoboda1337 @lumapu @stefan123t

[swoboda1337]

Ok I got it:

data = [0x70, 0x6A, 0xE8, 0xC6, 0x9C, 0xB4, 0x2E, 0x8C, 0xB9, 0xA8, 0xC0, 0x06, 0xCB, 0x00, 0xD0, 0x04, 0x68, 0xC0, 0x00, 0x5C, 0x00, 0x01, 0xA4, 0x66, 0x9A, 0x1A, 0x65]
out = []

for byte in data:
	for i in range(8):
		out.append((byte >> 7) & 1)
		byte = byte << 1

# we have to drop the first 6 bits because the first byte is missing and was interpreted as the length by the sx127x
# since 8 bits are missing and we need to drop 14 there are 6 bits left to drop, wont have to do this when using a fixed
# length packet as we can properly decode the length and the first byte wont be lost
for i in range(6):
	out.pop(0)

for i in range(int(len(out) / 14)):
	byte = 0
	for j in range(4):
		byte = byte << 1
		byte = byte | out.pop(0)
	for i in range(3):
		out.pop(0)
	for j in range(4):
		byte = byte << 1
		byte = byte | out.pop(0)
	for i in range(3):
		out.pop(0)		
	print("%02x" % (byte & 0xFF))

That code prints this which matches your expected data:

15
83
76
24
23
80
19
06
08
80
0b
00
68
6d
6c

That code is messy but its late and I just figured it out. The data is just 4 bits message then 3 bits of parity.

Unfortunately the length is encoded as well. So you can't use variable length packets in the sx127x, which is fine, set payload_length: 60 for now. Even if it goes beyond the length of the real packet its fine we can ignore the garbage later after decoding it.

You then just have to keep 4 bits of message and drop 3 bits of parity. Repeat that for the full message. After decoding it the first byte will be the length followed by the payload.

Will have to figure out the CRC next and possibly how to use the parity (and encode if you want to transmit). It seems to be a hamming code.

[swoboda1337]

If you set payload_length: 60 and do something like this it should match your expected (just drops parity bits for now):

  on_packet:
    then:
      - lambda: !lambda |-
          std::vector<uint8_t> decoded;
          uint32_t remainder = 0;
          uint32_t data = 0;
          for (uint8_t byte : x) {
            data = (data << 8) | byte;
            remainder += 8;
            if (remainder >= 14) {
              decoded.push_back(((data >> (remainder - 8)) & 0xF0) | ((data >> (remainder - 11)) & 0x0F));
              remainder -= 14;
            }
          }
          ESP_LOGD("lambda", "packet %s", format_hex(decoded).c_str());

[LorbusChris]

Thanks, that's an awesome start! I'll post updates here in the coming days.

[stefan123t]

@swoboda1337 that is some serious bit banging you are doing here. 🎩 Chapeau!

@LorbusChris love seeing SX127x LoRaWAN chipsets being put to purpose here using ESPHome.
What is the reason for this misalignment of bits and bytes ?
Are we missing the GPIO3 IRQ which was used on the CMT2300A ?

@DanielR92 and @lumapu hacked together the initial CMT implementation.
Maybe they can shed some light, but this might be just some SC127x specific issue in the first place ?

[swoboda1337]

If you look at the CMT2300A datasheet:

You can see the packet length is FEC encoded. The SX127x doesn't have support for FEC and @LorbusChris had configured it to use a variable packet length. This obviously can't work since it won't be able to decode the length, which is why it was misaligned, it ate the first byte assuming it was length.

If a fixed packet length is used nothing will be misaligned because the SX127x won't eat the first byte. The FEC will just have to be decoded in software. The pattern is simply MMMMPPP and that repeats for the entire message (M is a message bit and P is a parity bit).

EDIT: Note a fix length packet will work fine, just set it longer than it needs to be, will ignore the garbage at the end after the real length is decoded in software.

EDIT: All my lambda above does is strip out the party bits leaving only the message. Don't have to correct errors if the signal is good enough but shouldn't be too hard to do if required.

[LorbusChris]

Thanks! I can confirm that there's an extra byte at the beginning when using a fixed length.

I'll also be able to test this on an SX126x in the coming days.

[swoboda1337]

Thanks! I can confirm that there's an extra byte at the beginning when using a fixed length.

I'll also be able to test this on an SX126x in the coming days.

Did you try with that lambda? It matches the expected?

[stefan123t]

@swoboda1337 thanks for trying to enlighten and educate me, much appreciated. I think I got at least the gist of it 😅 You guys definitely rock!

The maximum packet size is 27 bytes for a full frame. Due to the maximum buffer size of the earlier used NRF24L01+ and the limitations of the Nordic Semiconductors Enhanced ShockBurst protocol.

@LorbusChris here a wavedrom DevInfo 0x15 MainCmd/MID for the RealTimeRunDebug 0x0b SubCmd Command:

{reg: [
    {bits: 1, name: '(0x7e)',        attr: '(SOF)',       type: 1},
    {bits: 1, name: '0x15',          attr: 'MID',         type: 2},
    {bits: 4, name: '0x12 34 56 78', attr: ['TargetAddr#', '1164 801 12345678', 'HMS-1660-4T Address'], type: 4},
    {bits: 4, name: '0x98 76 54 32', attr: ['SourceAddr#', '10FD 801 98765432', 'DTU Pro S Address'], type: 6},
    {bits: 1, name: '0x80',          attr: 'Idx',         type: 5},
    {bits: 2, name: '0x0b 00',       attr: 'SubCmd + ?',  type: 2},
    {bits: 4, name: '0x65 72 06 B8', attr: 'TimeStamp',   type: 3},
    {bits: 2, name: '0x00 00',       attr: 'Gap',         type: 3},
    {bits: 2, name: '0x00 00',       attr: 'Seq?',        type: 3},
    {bits: 4, name: '0x00 00 00 00', attr: 'Password',    type: 3},
    {bits: 2, name: '0xC9 72',       attr: 'CRC16',       type: 1},
    {bits: 1, name: '0x07',          attr: 'CRC8',        type: 1},
    {bits: 1, name: '(0x7f)',        attr: '(EOF)',       type: 1},
], config: {vflip: true, hspace: (29*40)}}

@LorbusChris please note, the GD32/STM32 to NRF24L01+ / CMT2300A communication employed on the Hoymiles DTU boards uses some start and stop bytes 0x7F SOF and 0x7E EOF for each frame, though this does not ((n)ever) go over the air.
So when looking at some of the raw onboard traces we logged you may need to replace 0x7d5([d-f]) -> 0x7\1 and vice versa to escape these three special bytes, see here tbnobody/OpenDTU#1421 (comment)

The Source and Target Adresses are using only the last 8 digits / 4 bytes in literal Hex-Encoding.
I.e. the first four digits / two bytes, i.e. the Inverter / DTU Model Type and the 3 digits / 1.5 bytes for the Production Year and Calendar Week of the Production run are neither sent over the air ever. This leaves you with the last 8 digits of the usually 15 digit serial IDs.

You may want to refer to some of the official the DTU / Inverter Model Type Prefixes e,g, 10FD a DTU Pro S and 1164 for an HMS-1600-4T.
E.g 0x801 is 8 = 2022, 9 = 2023, 0 = 2024, A = 2025, etc. and CW 01.

[Jumperm]

Thanks! I can confirm that there's an extra byte at the beginning when using a fixed length.

I'll also be able to test this on an SX126x in the coming days.

Hi LorbusChris,
do you get an solution for decoding the protocol of your HMS?

I'm interested in decoding the protocol of an HM300 with an NRF24L01+ over ESP Home. But I only manage to get the NRF to Work with the ESP Home. Maybe we can work together on adding the decoding LIB.

FYI: There was also an feature request for an NRF24L01

#1002 (comment)

BR Jumper

[stefan123t]

@Jumperm there is a whole library for the NRF which we use in OpenDTU and Ahoy!DTU projects: https://github.com/nRF24/RF24
It is based on GPLv2 in case you are curious.

I got the question on our Discord why one would want to query the Hoymiles MI/HM (NRF24L01+ based) HMS/HMT (CMT2300A) Inverters with ESPHome in the first place ? My answer would be because it should be possible, it is OpenSource after all, but what is your reasoning for trying it ?

[LorbusChris]

@Jumperm I haven't found the time to work on this yet unfortunately.

@stefan123t my reasoning is that with esphome one can fully control and build in only the features that one requires into the firmware, and compose multiple sensors/io and software/api features into one device, which is nice. I use esphome for some sensors already, and for me just having the inverter be another one, and being able to build and trigger firmware updates from Home Assistant myself easily would make an esphome based solution preferable to me. (there's another whole point of esphome supporting zigbee/thread/IEEE 802.15.4 networking, which I prefer over wifi for sensor stuff and which esphome makes super easy).

[stefan123t]

@LorbusChris thanks for explaining your goals of standardization and maintenance. I understand that it will also allow to refine and refactor the actual parsing and rf comms code which is a boon if multiple projects use and share the same code base. I hope that your changes will find their way back into the other projects.

I believe @lumapu built a library at one point for decoding Hoymiles packets in Ahoy which was eventually reused by @tbnobody in OpenDTU. I do not know for sure if they still share the exact same code base, but it would be good if they did sync their changes some day.

[Jumperm]

@Jumperm I haven't found the time to work on this yet unfortunately.

Thanks for answering.

I got the question on our Discord why one would want to query the Hoymiles MI/HM (NRF24L01+ based) HMS/HMT (CMT2300A) Inverters with ESPHome in the first place ? My answer would be because it should be possible, it is OpenSource after all, but what is your reasoning for trying it ?

@stefan123t That was me ;)
I nearly have the same goal. I have some espHome sensor and like the easy way to enhance the esp with sensors or on device automations.

@LorbusChris I hope we can go ahead with this project