RMT Buffer Allocation Fix (Thread-Safe) for Issue #375

[teknynja]

NOTE: This is a thread-safe version of pull-request #392: It is essentially the same code with a mutex added to protect the code from multiple threads.

This pull request addresses issue #375 [pixel.show() crash with more than 73 pixel on ESP32s3]

After reviewing the code and also getting some helpful feedback from the Espressif Forums (https://www.esp32.com/viewtopic.php?f=13&t=40270) and @robertlipe it was determined that the code in esp.c for handling the RMT item buffers when using the IDF v5 framework was allocating too much space on the stack when using more than 70ish pixels.

This pull request addresses that issue by allocating the RMT buffers from the heap instead. It will attempt to allocate a single block of memory to accommodate the largest configured instance (sharing the buffer between instances is fine, as the buffer is completely populated each each time the espShow() method is called).

I also took the time to improve the channel allocation management, previously the RMT channels were initialized on each call to espShow(), now the RMT channels are only de-initialized and re-initialized whenever the output pin is changed.

Finally, I was concerned about problems that may be caused by allocating large buffers on the heap without giving the user any way to free that memory, so the code allows a user to free that memory (and also release the RMT channels) by setting the number of pixels to zero using .updateLength(0) and then calling .show(). This will de-allocate the heap memory used for the RMT buffers and release the RMT channels held by driver. They will automatically be re-allocated if needed when setting the number of pixels back to a non-zero value.

[teknynja]

Updated code to move espShow()'s mutex initialization to the Adafruit_NeoPixel constructor, allowing users to avoid a race condition.

[robertlipe]

I like this. Thank you, teknynja.

…

On Sat, Jun 15, 2024 at 2:07 PM teknynja ***@***.***> wrote: Updated code to move espShow()'s mutex initialization to the Adafruit_NeoPixel constructor, allowing users to avoid a race condition. — Reply to this email directly, view it on GitHub <#394 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD3ZNOUTWXBT74UFP2Z3ZHSGHVAVCNFSM6AAAAABJGXOECWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZQGUZTKMJXGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[egnor]

NOTE NOTE NOTE I AM ALSO JUST A KIBITZER DO NOT ASSUME I KNOW WHAT I AM DOING

I think this change can be simplified a great deal, and doing so will make it much easier to review and merge. See the individual comments, but in short

• remove spurious whitespace changes
• use a function-local static to initialize the mutex
• simplify the interaction with rmtInit() and rmtDeinit()

[egnor]

[minor] Again, maybe always have an espInit() for ESP32, just have it do nothing as appropriate based on ESP-IDF version? BUT see comments below, I think we can actually remove espInit() entirely?

[dhalbert]

later...

[egnor]

CONSIDER instead, for diagnostics and conciseness:

const int requiredBytes = requiredSize * sizeof(rmt_data_t);
led_data = (rmt_data_t *)realloc(led_data, requiredBytes);
if (!led_data) {
  log_e("NeoPixel RMT buffer allocation (%d bytes) failed", requiredBytes);
  xSemaphoreGive(show_mutex);
  return;
}

[dhalbert]

later...

[robertlipe]

It's unfortunate that the inertia behind this (much needed) PR seems to have been lost...again.

Is there anyone with commit privileges that can deliver specific reasons this shouldn't be applied or that can produce a list of required changes to get it applied? I don't particularly have a direct interest in this project, but if it "just" needs a programmer to jump through acceptance hoops and confirm that a specific test case works, I'm willing to build up a test fixture and help. Crashing at 70-odd pixels on a robust SOC is kinda embarrassing.

I just hate to see this languish for another couple of quarters. How can we move this PR toward that that "merge" button?

I'm willing to commit to (probably)more-than-moderately-qualified brain (and equipment) cycles to moving this PR forward. What will it take? Let's get the kibitizers (?) and the approvers together and get some action on this, please.

I'd like to see someone with submit approval comment on what it would take to move this to completion. Whether Tekninja or another of us takes the PR and moves it to the goal, the ESP32 world really needs to see this landed.

How can we all help get this submitted?

[ednieuw]

I forked your changes and tried it with an Arduino Nano ESP32 with ESP32 board 3.0.4
But the Arduino Nano ESP32 crashes with my program.
When I use the Arduino board 2.0.13 the modification does not crash the Arduino. But with the original Neopixel also works fine.
So there is still a problem with the Neopixel library on ESP32 board 3.0.4

[teknynja]

@ednieuw - The only thing I'm noticing when doing a quick look at your code is that you are creating three instances of the NeoPixel driver. The first one isn't really allocating much memory because it's using default constructor, but the other two are allocating space for 256 pixels each at 3 & 4 bytes per pixel. That's probably not too much memory for the driver's normal pixel buffers, but it does end up trying to allocate 32768 bytes from the heap for the RMT buffer to be shared by both instances - I don't know if that's a problem for the underlying libraries or not, but you might try paring back the number of pixels to the required minimum to see if that helps.

Digging into this problem did reveal another problem in my changes though (although it's likely not related to your issue). If a user creates all instances of the NeoPixel driver using the default constructor and then configures/uses those instance(s) later on in the code, the mutex object is not instantiated, which would probably negatively impact the operation of the esp32 RMT driver. The state of my patch seems to be in limbo at this point 🤷, so unless somebody hits this particular code path, I'm probably just going to leave things as they are for now.

[ednieuw]

I found out the size of the RMT-buffer is large enough for 512 LEDs.

Espressif has a nice example with WS2812 LED strips:
https://docs.espressif.com/projects/arduino-esp32/en/latest/api/rmt.html

I tested it with a with 256 and 512 WS2812 LEDs and that works.
It takes 8 sec to write 512 LEDs 512 times. so apprx 16 ms to write 512 LEDs.
The strip is connected to Arduino pin D5 = GPIO8. Compile with pin numbering: By GPIO numbering

My effort to adapt the coding for a SK6812 strip needs some more time. The timing is not correct and it still write 3 bytes instead of four. But it is not on my priority list yet.
https://github.com/ednieuw/RMT_WS2812-SK6812-ESP32

// Copyright 2024 Espressif Systems (Shanghai) PTE LTD
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at

//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/**
 * @brief This example demonstrates usage of RGB LED driven by RMT
 *
 * The output is a visual WS2812 RGB LED color moving in a 8 x 4 LED matrix
 * Parameters can be changed by the user. In a single LED circuit, it will just blink.
 */
#define PIN_LED_RGB 8

#define NR_OF_LEDS     16 * 16
#define NR_OF_ALL_BITS 24 * NR_OF_LEDS

//
// Note: This example uses a board with 32 WS2812b LEDs chained one
//      after another, each RGB LED has its 24 bit value
//      for color configuration (8b for each color)
//
//      Bits encoded as pulses as follows:
//
//      "0":
//         +-------+              +--
//         |       |              |
//         |       |              |
//         |       |              |
//      ---|       |--------------|
//         +       +              +
//         | 0.4us |   0.85 0us   |
//
//      "1":
//         +-------------+       +--
//         |             |       |
//         |             |       |
//         |             |       |
//         |             |       |
//      ---+             +-------+
//         |    0.8us    | 0.4us |

rmt_data_t led_data[NR_OF_ALL_BITS];

void setup() {
  Serial.begin(115200);
  if (!rmtInit(PIN_LED_RGB, RMT_TX_MODE, RMT_MEM_NUM_BLOCKS_1, 10000000)) {
    Serial.println("init sender failed\n");
  }
  Serial.println("real tick set to: 100ns");
}

int color[] = {0x11, 0x99, 0x11};  // Green Red Blue values
int led_index = 0;

void loop() 
{
  // Init data with only one led ON
  int led, col, bit;
  int i = 0;
  for (led = 0; led < NR_OF_LEDS; led++) {
    for (col = 0; col < 3; col++) {
      for (bit = 0; bit < 8; bit++) {
        if ((color[col] & (1 << (7 - bit))) && (led == led_index)) {
          led_data[i].level0 = 1;
          led_data[i].duration0 = 8;
          led_data[i].level1 = 0;
          led_data[i].duration1 = 4;
        } else {
          led_data[i].level0 = 1;
          led_data[i].duration0 = 4;
          led_data[i].level1 = 0;
          led_data[i].duration1 = 8;
        }
        i++;
      }
    }
  }
  // make the led travel in the panel
  if ((++led_index) >= NR_OF_LEDS) {
    led_index = 0;
  }
  // Send the data and wait until it is done
  rmtWrite(PIN_LED_RGB, led_data, NR_OF_ALL_BITS, RMT_WAIT_FOR_EVER);
//  delay(1);
}

[ednieuw]

Not willing to wait long for an working update of the Neopixel library I wrote a library for WS2812 and SK6812 LEDs to be used with an Arduino Nano ESP32 with Espressif board ESP32 V3.0.
It uses the RMT driver and It will probably also work with ESP32-S3 and other ESP32's
https://github.com/ednieuw/EdSoftLED
It is crude version 1.0.0

[robertlipe]

Nice, Ed. Since this library seems abandoned, working around it seems appropriate.

…

On Mon, Sep 23, 2024 at 8:38 AM Ed Nieuwenhuys ***@***.***> wrote: Not willing to wait long for an working update of the Neopixel library I wrote a library for WS2812 and SK6812 LEDs to be used with an Arduino Nano ESP32 with Espressif board ESP32 V3.0. It uses the RMT driver and It will probably also work with ESP32-S3 and other ESP32's https://github.com/ednieuw/EdSoftLED It is crude version 1.0.0 — Reply to this email directly, view it on GitHub <#394 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCSD36JE6YOTQXJTSPMDS3ZYAKW3AVCNFSM6AAAAABJGXOECWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRYGI4TKMBUGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bobdoah]

Thanks @teknynja, I was struggling with my build of https://github.com/BikeBeamer/BikeBeamer because it uses 256 WS2812B LEDs. This fixed my issue!

[dhalbert]

I am looking at #402, which provoked #392 and #394. @teknynja and @robertlipe: I have some questions for you (and anyone else who has technical suggestions).

• Do you know if ESP32 Arduino in general is thread-safe, for things like I2C, etc.? I am wondering if that is an expectation in general for Arduino.
• We have another implementation of NeoPixel writing in CircuitPython, https://github.com/adafruit/circuitpython/blob/main/ports/espressif/common-hal/neopixel_write/__init__.c, which does not do explicit storage allocation at all. Instead it usesrmt_new_bytes_encoder()and rmt_transmit(), which take care of storage management under the covers. That implementation also does not hold on the RMT peripheral between writes, so the RMT peripheral and the storage needed are grabbed and released each time. Those are not terribly expensive operations. Do you see some reason not to switch to that style of implementation?
• This PR appears to still be in process, in the sense that there are a number of unresolved review comments. I think we saw it in process and expected to wait until it had settled down, which didn't quite happen.

Thanks for your comments.

[teknynja]

Do you know if ESP32 Arduino in general is thread-safe, for things like I2C, etc.? I am wondering if that is an expectation in general for Arduino.

I guess thread-safety is a general expectation I have for these kinds of libraries, but it may not be an issue for most use-cases in Arduino

We have another implementation of NeoPixel writing in CircuitPython, https://github.com/adafruit/circuitpython/blob/main/ports/espressif/common-hal/neopixel_write/__init__.c, which does not do explicit storage allocation at all. Instead it usesrmt_new_bytes_encoder()and rmt_transmit(), which take care of storage management under the covers. That implementation also does not hold on the RMT peripheral between writes, so the RMT peripheral and the storage needed are grabbed and released each time. Those are not terribly expensive operations. Do you see some reason not to switch to that style of implementation?

This PR was really intended as more of a patch to get things working for people trying to use the library until something more elegant came along. Without looking at the details, it sounds like using the higher-level rmt calls for dealing with allocation is probably the better solution to this in the long run.

This PR appears to still be in process, in the sense that there are a number of unresolved review comments. I think we saw it in process and expected to wait until it had settled down, which didn't quite happen.

I think this PR ran out of steam once people starting discussing more extensive, long-term solutions to the problem and the lack of consensuses on how to move forward. It sounds like going with something like the implementation from CircuitPython is probably the way to actually resolve this issue.

[egnor]

• Agreed that Arduino libraries are broadly not thread safe, and in many cases actively thread hostile. It's nice if a library is thread safe (and says so!) but it's a little hard to discern even which ESP-IDF calls are or are not thread safe. Do not let that stop you from moving forward!!

• Also agreed that using a streaming encoding callback is a better approach BUT BUT BUT BEWARE-- I have run into significant issues, especially with single core ESP32 variants, where the RMT peripheral's buffer size is small enough that it can't get filled in time leading to annoying glitches especially when wi-fi is active. WORSE YET the RMT buffer size is directly opposed to the number of RMT channels you can have, AND ESP-IDF has removed the ability to change pins on the fly, so if you want to drive more than 2 pins of LED strips it's a messsssss.

  HOWEVER HOWEVER all of that also applies to the "make a big buffer of RMT symbols" approach, so really that's not reason to use an encoding callback, it's just something to be aware of. Definitely make sure the encoding callback has the least blockers on running (put it in IRAM, etc). And on chips that do support RMT DMA (and thus larger buffers and more pins), make sure to enable that. (I think that's only the ESP32-S3 at this point?)

  ALSO ALSO the API for encoder callbacks is subtle with the way it represents nested encoder state, I had a bug in my version for a while where it was skipping the final delay which meant that usually things would work but when two streams go out back to back it's a mess. But given that you have CircuitPython code to crib from (and also other implementations) that makes things easier. (I see the CircuitPython version has its own approach to the end delay though I think doing it in the encoder is much better because it means the call doesn't have to delay -- but the CircuitPython code delays until the whole thing is sent anyway so whatever.)

@dhalbert I would encourage you to use your judgment to move as fast as you can to a fix, whether it be adopting this PR as a stopgap (it's a fine stopgap imho), or making your own PR similar to this one as a stopgap, or just heading straight to a streaming solution. I think we're all on the same page here.

Here's my implementation for reference (depends on some of my utility libraries so not easy to run directly, though I'm happy to share those if you care for some reason)

#include "ok_led_strip.h"

#include "Arduino.h"

#include "ok_logging.h"
#include "ok_logging_esp.h"

static OkLoggingContext const OK_CONTEXT("ok_led_strip");

#if defined(ARDUINO_ARCH_ESP32)
  #include "esp_idf_version.h"
  #include "soc/soc_caps.h"

  #if SOC_RMT_SUPPORTED && ESP_IDF_VERSION_MAJOR >= 5
    #define INCLUDE_ESP32_RMT_DRIVER 1
  #endif
#endif

#if INCLUDE_ESP32_RMT_DRIVER
  #include <atomic>

  #include "driver/gpio.h"
  #include "driver/rmt_tx.h"
  #include "esp32-hal-periman.h"
  #include "esp_attr.h"
  #include "esp_check.h"

  static constexpr uint32_t RMT_RESOLUTION = 10000000;  // 1 MHz

  static constexpr rmt_symbol_word_t END_SYM = {
    .duration0 = RMT_RESOLUTION * 280 / 1000000, .level0 = 0,  // 280us
    .duration1 = 0, .level1 = 0,
  };

  struct context {
    rmt_encoder_t main_encoder;   // So rmt_encoder_t* casts to context*
    rmt_encoder_t* data_encoder;  // Byte encoder for LED data
    rmt_encoder_t* end_encoder;   // Copy encoder that delays a bit at the end
    rmt_channel_handle_t rmt_channel;
    bool at_end;
    std::atomic<bool> running;
  };

  static size_t IRAM_ATTR on_encode(
      rmt_encoder_t* encoder, rmt_channel_handle_t channel,
      void const* data, size_t data_size,
      rmt_encode_state_t* status) {
    auto* const ctx = (context*) encoder;
    size_t symbols = 0;

    *status = RMT_ENCODING_RESET;
    if (!ctx->at_end) {
      rmt_encode_state_t data_status = RMT_ENCODING_RESET;
      symbols += ctx->data_encoder->encode(
          ctx->data_encoder, channel, data, data_size, &data_status);
      if (data_status & RMT_ENCODING_COMPLETE) ctx->at_end = true;
      if (data_status & RMT_ENCODING_MEM_FULL) *status = RMT_ENCODING_MEM_FULL;
    }

    if (ctx->at_end && !(*status & RMT_ENCODING_MEM_FULL)) {
      symbols += ctx->end_encoder->encode(
          ctx->end_encoder, channel, &END_SYM, sizeof(END_SYM), status);
    }

    return symbols;
  }

  static esp_err_t IRAM_ATTR on_reset_encoder(rmt_encoder_t* encoder) {
    auto* const ctx = (context*) encoder;
    if (ctx->data_encoder != nullptr) rmt_encoder_reset(ctx->data_encoder);
    if (ctx->end_encoder != nullptr) rmt_encoder_reset(ctx->end_encoder);
    ctx->at_end = false;
    return ESP_OK;
  }

  static bool IRAM_ATTR on_trans_done(
      rmt_channel_handle_t channel,
      rmt_tx_done_event_data_t const* ev, void* vctx) {
    auto* const ctx = (context*) vctx;
    OK_FATAL_IF(!ctx->running);
    ctx->at_end = false;
    ctx->running = false;  // Atomic write allows main code to continue
    return false;  // No task woken up.
  }

  class OkLedStripEsp32Rmt : public OkLedStrip {
   public:
    OkLedStripEsp32Rmt() {}

    ~OkLedStripEsp32Rmt() {
      OK_DETAIL("Stopping RMT LED strip driver (pin %d)", pin);
      if (ctx.rmt_channel != nullptr && enabled) rmt_disable(ctx.rmt_channel);
      if (ctx.rmt_channel != nullptr) rmt_del_channel(ctx.rmt_channel);
      if (ctx.data_encoder != nullptr) rmt_del_encoder(ctx.data_encoder);
      if (ctx.end_encoder != nullptr) rmt_del_encoder(ctx.end_encoder);
    }

    bool setup(int pin) {
      OK_FATAL_IF(ctx.running);
      OK_FATAL_IF(this->pin != -1);
      OK_FATAL_IF(pin < 0);
      this->pin = pin;

      OK_DETAIL("Preparing RMT LED strip driver (pin %d)", pin);
      if (OK_ERROR_IF(!perimanClearPinBus(pin))) return false;

      ctx.main_encoder.encode = on_encode;
      ctx.main_encoder.reset = on_reset_encoder;

      rmt_bytes_encoder_config_t const data_cf = {
        .bit0 = {
          .duration0 = RMT_RESOLUTION * 3 / 10000000, .level0 = 1,  // 0.3us
          .duration1 = RMT_RESOLUTION * 9 / 10000000, .level1 = 0,  // 0.9us
        },
        .bit1 = {
          .duration0 = RMT_RESOLUTION * 9 / 10000000, .level0 = 1,  // 0.9us
          .duration1 = RMT_RESOLUTION * 3 / 10000000, .level1 = 0,  // 0.3us
        },
        .flags = {.msb_first = 1},
      };
      if (OK_LOG_ESP_ERRORS(rmt_new_bytes_encoder(&data_cf, &ctx.data_encoder)))
        return false;

      rmt_copy_encoder_config_t const end_cf = {};
      if (OK_LOG_ESP_ERRORS(rmt_new_copy_encoder(&end_cf, &ctx.end_encoder)))
        return false;

      rmt_tx_channel_config_t const rmt_cf = {
        .gpio_num = (gpio_num_t) pin,
        .clk_src = RMT_CLK_SRC_DEFAULT,
        .resolution_hz = RMT_RESOLUTION,
        .mem_block_symbols = 256,  // Max on ESP32-S2
        .trans_queue_depth = 1,
        // .flags = {.with_dma = true},
      };
      if (OK_LOG_ESP_ERRORS(rmt_new_tx_channel(&rmt_cf, &ctx.rmt_channel)))
        return false;

      rmt_tx_event_callbacks_t const cb = {.on_trans_done = on_trans_done};
      if (
        OK_LOG_ESP_ERRORS(rmt_tx_register_event_callbacks(
            ctx.rmt_channel, &cb, &ctx)) ||
        OK_LOG_ESP_ERRORS(rmt_enable(ctx.rmt_channel))) {
        return false;
      }

      OK_DETAIL("RMT LED strip driver enabled (pin %d)", pin);
      enabled = true;
      return true;
    }

    virtual void start_sending(std::span<uint8_t const> data) override {
      if (ctx.running.exchange(true)) {
        OK_FATAL("Pin %d is still busy", pin);
      }

      rmt_transmit_config_t transmit_cf = {};
      if (OK_LOG_ESP_ERRORS(rmt_transmit(
              ctx.rmt_channel, &ctx.main_encoder,
              &data[0], data.size(), &transmit_cf))) {
        ctx.running = false;
      }
    }

    virtual bool busy() const override { return ctx.running; }

   private:
    int pin = -1;
    bool enabled = false;
    context ctx = {};
  };
#endif  // INCLUDE_ESP32_RMT_DRIVER

std::unique_ptr<OkLedStrip> ok_led_strip(int pin) {
  #if INCLUDE_ESP32_RMT_DRIVER
    auto esp32_rmt_strip = std::make_unique<OkLedStripEsp32Rmt>();
    if (esp32_rmt_strip->setup(pin)) return std::move(esp32_rmt_strip);
  #endif

  OK_ERROR("No LED strip driver available for pin %d", pin);
  return {};
}

[robertlipe]

I think that everyone with a qualified opinion was pretty happy with the second patch. I don't know if the 'resolved' button was clicked on everything, but the last time I looked at it, I was as happy as I get. We just can't get anyone with appropriate permission to press the 'merge' button, leading me to suspect this library has become abandoned. (The last commit was six months ago.)

ESP32 users may thus be better served by Espressif's own WS281x library, which, of course, tracks all the latest thrash in the ESP-IDF RMT layer: https://components.espressif.com/components/espressif/led_strip/
The latest FastLED builds have moved to that library as a backend to do the actual diode scribbling. This has the benefit of bypassing the Arduino 3.0 layer which broke so many Arduino packages, many of which have also fallen into disrepair.

Threading, TMK, doesn't exist in pure Arduino Land because it's geared for 8-bit processors with tiny RAM. Threading on ESP32, however, is extremely common, as most of those units are even multi-core, and all of them are fast enough to multithread around waiting for peripherals and such.

Users continue to suffer:
#402
I've sent multiple people to this PR for a solution, and I don't even have any projects using this code.

If you need a working version of THIS code, I would totally go with Tekninja's PR. Threading or not, what's in the trunk now is totally broken for most non-trival cases.

[dhalbert]

I tested this code on a spliced-together strip of 24+30+30 Neopixels, with strandtest. It works fine. We'll merge and release this now, and check on adapting the CircuitPython code to keep it simpler. We'll also check on the current ESP-IDF library.

[dhalbert]

later...

[dhalbert]

later...

[dhalbert]

@tyeth I've tested and merged this PR, but not bumped the version number. Could you release this? Thanks.

@ladyada

[ladyada]

all good, tyeth will do so when up next :)

[tyeth]

Thanks Dan, released as v.1.12.4