ESP-IDF V5 #4838
ESP-IDF V5 #4838
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
I'm not sure if renaming Network is the right approach to fix the conflict @willmmiles |
|
I am also unsure why we are seeing linker errors for things like mbedtls_sha1_starts |
Hm, it looks like it's just a bag of stateless utility functions. Probably it should be a namespace instead of a class. I don't think there's a practical solution to avoid the collision other than renaming, unless the newer core offers these utilities itself. (In fact I'd also suggest pulling them in to wled00/network.cpp , given the use of WLED-specific #defines.) (Best pratice would've been that the upstream libraries put everything they do in a namespace, eg.
I believe that the Tasmota platform omits mbedtls to save flash space, as they've gone over to BearSSL. Looks like they're still exposing the headers though - I don't know if that's an IDF bug or something wrong with their build process. AsyncWebServer needs an SHA1 implementation as it is required for WebSockets. It looks like the upstream AWS folks have switched over to IDFv5 functions and vendored in Espressif's implementation when building on IDFv4. I'll see if I can pull those patches. |
|
With few fixes it builds. MQTT mbedTLS WiFi wled.cpp Compare IPAddress to a value, not to 0 (prevents pointer overload → memcmp on nullptr) and clamp index (prevents out-of-range access cascading into bad compares). @@ void WLED::initConnection()
- if (multiWiFi[selectedWiFi].staticIP != 0U && multiWiFi[selectedWiFi].staticGW != 0U) {
- WiFi.config(multiWiFi[selectedWiFi].staticIP, multiWiFi[selectedWiFi].staticGW, multiWiFi[selectedWiFi].staticSN, dnsAddress);
- } else {
- WiFi.config(IPAddress((uint32_t)0), IPAddress((uint32_t)0), IPAddress((uint32_t)0));
- }
+ if (multiWiFi.empty()) { // guard: handle empty WiFi list safely
+ WiFi.config(IPAddress((uint32_t)0), IPAddress((uint32_t)0), IPAddress((uint32_t)0));
+ } else {
+ if (selectedWiFi >= multiWiFi.size()) selectedWiFi = 0; // guard: ensure valid index
+ if (multiWiFi[selectedWiFi].staticIP != IPAddress((uint32_t)0) &&
+ multiWiFi[selectedWiFi].staticGW != IPAddress((uint32_t)0)) { // guard: compare as IPAddress to avoid pointer overload
+ WiFi.config(multiWiFi[selectedWiFi].staticIP, multiWiFi[selectedWiFi].staticGW, multiWiFi[selectedWiFi].staticSN, dnsAddress);
+ } else {
+ WiFi.config(IPAddress((uint32_t)0), IPAddress((uint32_t)0), IPAddress((uint32_t)0));
+ }
+ }ESP32 MAC fallback (fix Info tab showing 000000000000) @@ (top of file)
+#ifdef ARDUINO_ARCH_ESP32
+#include <esp_mac.h> // fallback MAC read from efuse
+#endif@@ // generate module IDs must be done before AP setup
-escapedMac = WiFi.macAddress();
+escapedMac = WiFi.macAddress();
+#ifdef ARDUINO_ARCH_ESP32
+if (escapedMac == "00:00:00:00:00:00") { // fix: fallback to efuse MAC if WiFi not started
+ uint8_t m[6] = {0};
+ if (esp_read_mac(m, ESP_MAC_WIFI_STA) == ESP_OK) {
+ char buf[18];
+ sprintf(buf, "%02X:%02X:%02X:%02X:%02X:%02X", m[0], m[1], m[2], m[3], m[4], m[5]);
+ escapedMac = buf;
+ }
+}
+#endif
escapedMac.replace(":", "");
escapedMac.toLowerCase();network.cpp
@@ int findWiFi(...)
- if (multiWiFi.size() <= 1) {
- DEBUG_PRINTF_P(PSTR("WiFi: Defaulf SSID (%s) used.\n"), multiWiFi[0].clientSSID);
- return 0;
- }
+ if (multiWiFi.size() <= 1) {
+ if (multiWiFi.empty()) return 0; // guard: handle empty list safely
+ DEBUG_PRINTF_P(PSTR("WiFi: Defaulf SSID (%s) used.\n"), multiWiFi[0].clientSSID);
+ return 0;
+ }
@@ for (size_t n = 0; n < multiWiFi.size(); n++)
- bool foundBSSID = memcmp(multiWiFi[n].bssid, WiFi.BSSID(o), 6) == 0;
+ const uint8_t* _scanBSSID = WiFi.BSSID(o);
+ bool _cfgHasBSSID = false; for (int _i = 0; _i < 6; _i++) _cfgHasBSSID |= multiWiFi[n].bssid[_i];
+ bool foundBSSID = (_scanBSSID && _cfgHasBSSID) && memcmp(multiWiFi[n].bssid, _scanBSSID, 6) == 0; // guard: safe memcmp against nullptr/empty BSSIDUsermods Modified um_manager.cpp: #include "wled.h"
#include <assert.h>
/*
* Registration and management utility for v2 usermods
*
* Discovery paths, in this order:
* 1) Legacy dtors window: .dtors.tbl.usermods.0 ... .99 (older toolchains)
* 2) Dynamic fallback: usermods call registerUsermod(Usermod*) from a ctor
*/
// -------- Legacy dtors window (keep for backward compatibility) --------
static Usermod * const _usermod_table_begin[0]
__attribute__((section(".dtors.tbl.usermods.0"), unused)) = {};
static Usermod * const _usermod_table_end[0]
__attribute__((section(".dtors.tbl.usermods.99"), unused)) = {};
struct UMSpan {
Usermod* const* begin;
Usermod* const* end;
};
static inline UMSpan reg_dtors_span() {
UMSpan s{_usermod_table_begin, _usermod_table_end};
return s;
}
// ----- Dynamic fallback: usermods can push themselves here at boot -----
static Usermod* g_dynMods[16];
static size_t g_dynCnt = 0;
// Weak, so a stronger definition elsewhere (if any) wins.
// Usermods may call this from a constructor to guarantee registration.
extern "C" void registerUsermod(Usermod* m) __attribute__((weak));
extern "C" void registerUsermod(Usermod* m) {
if (m && g_dynCnt < (sizeof(g_dynMods)/sizeof(g_dynMods[0]))) {
g_dynMods[g_dynCnt++] = m;
}
}
// ----- Common iteration helpers -----
// Accept any callable (incl. capturing lambdas/functors)
template<typename Fn>
static inline void forEachMod(Fn&& fn) {
// 1) legacy dtors table
{
UMSpan s = reg_dtors_span();
for (auto p = s.begin; p < s.end; ++p) {
Usermod* m = *p;
if (m) fn(m);
}
}
// 2) dynamic list
for (size_t i = 0; i < g_dynCnt; ++i) {
if (g_dynMods[i]) fn(g_dynMods[i]);
}
}
static inline size_t countMods() {
size_t count = 0;
// dtors window
{
UMSpan s = reg_dtors_span();
count += (size_t)(s.end - s.begin);
}
// dynamic
count += g_dynCnt;
return count;
}
// --------- Usermod Manager methods (iterate via forEachMod) ---------
void UsermodManager::setup() { forEachMod([](Usermod* m){ m->setup(); }); }
void UsermodManager::connected() { forEachMod([](Usermod* m){ m->connected(); }); }
void UsermodManager::loop() { forEachMod([](Usermod* m){ m->loop(); }); }
void UsermodManager::handleOverlayDraw() { forEachMod([](Usermod* m){ m->handleOverlayDraw(); }); }
void UsermodManager::appendConfigData(Print& dest) {
forEachMod([&](Usermod* m){ m->appendConfigData(dest); });
}
bool UsermodManager::handleButton(uint8_t b) {
bool overrideIO = false;
forEachMod([&](Usermod* m){
if (m->handleButton(b)) overrideIO = true;
});
return overrideIO;
}
bool UsermodManager::getUMData(um_data_t **data, uint8_t mod_id) {
bool got = false;
forEachMod([&](Usermod* m){
if (got) return;
if (mod_id > 0 && m->getId() != mod_id) return;
if (m->getUMData(data)) got = true;
});
return got;
}
void UsermodManager::addToJsonState(JsonObject& obj) {
forEachMod([&](Usermod* m){ m->addToJsonState(obj); });
}
void UsermodManager::addToJsonInfo(JsonObject& obj) {
auto um_id_list = obj.createNestedArray("um");
forEachMod([&](Usermod* m){
um_id_list.add(m->getId());
m->addToJsonInfo(obj);
});
}
void UsermodManager::readFromJsonState(JsonObject& obj) {
forEachMod([&](Usermod* m){ m->readFromJsonState(obj); });
}
void UsermodManager::addToConfig(JsonObject& obj) {
forEachMod([&](Usermod* m){ m->addToConfig(obj); });
}
bool UsermodManager::readFromConfig(JsonObject& obj) {
bool allComplete = true;
forEachMod([&](Usermod* m){
if (!m->readFromConfig(obj)) allComplete = false;
});
return allComplete;
}
#ifndef WLED_DISABLE_MQTT
void UsermodManager::onMqttConnect(bool sessionPresent) {
forEachMod([&](Usermod* m){ m->onMqttConnect(sessionPresent); });
}
bool UsermodManager::onMqttMessage(char* topic, char* payload) {
bool handled = false;
forEachMod([&](Usermod* m){
if (!handled && m->onMqttMessage(topic, payload)) handled = true;
});
return handled;
}
#endif // WLED_DISABLE_MQTT
#ifndef WLED_DISABLE_ESPNOW
bool UsermodManager::onEspNowMessage(uint8_t* sender, uint8_t* payload, uint8_t len) {
bool handled = false;
forEachMod([&](Usermod* m){
if (!handled && m->onEspNowMessage(sender, payload, len)) handled = true;
});
return handled;
}
#endif // WLED_DISABLE_ESPNOW
void UsermodManager::onUpdateBegin(bool init) {
forEachMod([&](Usermod* m){ m->onUpdateBegin(init); });
}
void UsermodManager::onStateChange(uint8_t mode) {
forEachMod([&](Usermod* m){ m->onStateChange(mode); });
}
Usermod* UsermodManager::lookup(uint16_t mod_id) {
Usermod* found = nullptr;
forEachMod([&](Usermod* m){
if (!found && m->getId() == mod_id) found = m;
});
return found;
}
size_t UsermodManager::getModCount() {
return countMods();
}
/* Usermod v2 interface shim for oappend */
Print* Usermod::oappend_shim = nullptr;
void Usermod::appendConfigData(Print& settingsScript) {
assert(!oappend_shim);
oappend_shim = &settingsScript;
this->appendConfigData();
oappend_shim = nullptr;
}At the end of usermod_v2_RF433.cpp I added: // --- Registration via dynamic fallback (no custom sections) ---
static RF433Usermod usermod_v2_RF433;
REGISTER_USERMOD(usermod_v2_RF433); // OK to keep for legacy dtors builds
extern "C" void registerUsermod(Usermod*) __attribute__((weak));
__attribute__((constructor, used))
static void __rf433_ctor_register(void) {
if (registerUsermod) registerUsermod(&usermod_v2_RF433);
}NeoPixelBus --- a/src/internal/methods/ESP/ESP32/NeoEsp32RmtXMethod.h
+++ b/src/internal/methods/ESP/ESP32/NeoEsp32RmtXMethod.h
@@ -1,5 +1,7 @@
#pragma once
+// Needed for ESP_* macros that take a log_tag
+#include "esp_log.h"
+static const char* TAG = "NeoEsp32RmtX";And I added a temporary shim in wled00/bus_wrapper.h. Since ESP-IDF 5 switched to the new RMT “driver_v2,” Makuna renamed all of the “N” (legacy/driver_v1) variants to “X” (v2) under internal/methods/ESP/ESP32/* // --- temporary shim for NeoPixelBus CORE3 / RMT driver_v2 ------------------
#if __has_include(<NeoPixelBus.h>)
#define NeoEsp32RmtNWs2812xMethod NeoEsp32RmtXWs2812xMethod
#define NeoEsp32RmtNSk6812Method NeoEsp32RmtXSk6812Method
#define NeoEsp32RmtN400KbpsMethod NeoEsp32RmtX400KbpsMethod
#define NeoEsp32RmtNTm1814Method NeoEsp32RmtXTm1814Method
#define NeoEsp32RmtNTm1829Method NeoEsp32RmtXTm1829Method
#define NeoEsp32RmtNTm1914Method NeoEsp32RmtXTm1914Method
#define NeoEsp32RmtNApa106Method NeoEsp32RmtXApa106Method
#define NeoEsp32RmtNWs2805Method NeoEsp32RmtXWs2805Method
#endif
// --------------------------------------------------------------------------- |
|
Thanks for helping with this @sombree Could you please open a PR against the V5 branch and we can then pull in your fixes that way, maintaining attribution for your work |
…(ESP-IDF 5.3.4 / arduino-esp32 v3.1.4). Based on wled#4838.
…(ESP-IDF 5.3.4 / arduino-esp32 v3.1.4). Based on wled#4838.
| @@ -1,3 +1,4 @@ | |||
| #if false | |||
There was a problem hiding this comment.
TODO: remove once this has been updated for newer ESP-IDF
|
|
||
| #endif | ||
|
|
||
| #endif // NEOE_SP32_RMT_HI_METHODS_H |
There was a problem hiding this comment.
TODO: remove once this has been updated for newer ESP-IDF
| @@ -1,3 +1,5 @@ | |||
| #if false | |||
There was a problem hiding this comment.
TODO: remove once this has been updated for newer ESP-IDF
| } | ||
|
|
||
| #endif | ||
| #endif |
There was a problem hiding this comment.
TODO: remove once this has been updated for newer ESP-IDF
| ldscript_2m1m = eagle.flash.2m1m.ld | ||
| ldscript_4m1m = eagle.flash.4m1m.ld | ||
|
|
||
| default_usermods = ;; TODO: add back audioreactive once V5 compatible |
There was a problem hiding this comment.
TODO: add back audioreactive once V5 compatible
There was a problem hiding this comment.
This might be a "longer march" because I2S has received a completly different API in V5, so I'll need to rewrite a lot of code.
@troyhacks you mentioned that there is some kind of legacy I2S driver in IDF version5 🤔 Is this included the vanilla arduino-esp32 3.0.x ? Or we need our own arduino-esp32 / esp-idf build to use the legacy I2S driver in V5?
There was a problem hiding this comment.
In audioreactive, we will most likely lose "analog mics" when moving to V5.
The "ADC over I2S" feature that we use today was completely removed in the new V5 framework.
If we manage to port #4761 to V5, this might allow us to bring back Analog ADC microphones for V5.
| esp32s3dev_8MB_opi | ||
| esp32s3_4M_qspi | ||
| usermods | ||
| ; usermods |
There was a problem hiding this comment.
TODO: enable once the usermods have been ported to ESP-IDF V5
| fastled/FastLED @ 3.6.0 | ||
| IRremoteESP8266 @ 2.8.2 | ||
| fastled/FastLED @ 3.10.1 | ||
| ; IRremoteESP8266 @ 2.8.2 |
There was a problem hiding this comment.
TODO: fix IRremoteESP8266 for V5
| ; -D PIO_FRAMEWORK_ARDUINO_MMU_CACHE16_IRAM48_SECHEAP_SHARED ;; (experimental) adds some extra heap, but may cause slowdown | ||
| -D NON32XFER_HANDLER ;; ask forgiveness for PROGMEM misuse | ||
|
|
||
| -D WLED_DISABLE_MQTT ;; TODO: remove once we have updated library for V5 |
There was a problem hiding this comment.
TODO: remove once we have updated library for V5
| [esp32_all_variants] | ||
| lib_deps = | ||
| esp32async/AsyncTCP @ 3.4.7 | ||
| esp32async/AsyncTCP @ 3.4.6 |
There was a problem hiding this comment.
TODO: 3.4.7 breaks, so need to report issue upstream so we don't have to downgrade
There was a problem hiding this comment.
All AsyncTCP 3.x releases have full support for Arduino 3.x cores and current IDF releases; it's tested by our CI. AsyncTCP 3.4.6 has a serious memory bug and should never be used. (The only change in 3.4.7 is the bug fix.) Can you add any details as to what problem you're having?
| -D WLED_DISABLE_INFRARED ;; TODO: remove once we have updated library for V5 | ||
| -D WLED_DISABLE_MQTT ;; TODO: remove once we have updated library for V5 | ||
| -D WLED_ENABLE_DMX_INPUT | ||
| -D WLED_USE_SHARED_RMT ;; Use standard RMT method instead of incompatible NeoEsp32RmtHI - until updated for V5 |
There was a problem hiding this comment.
NeoEsp32RmtHI is not V5 compatible @willmmiles
There was a problem hiding this comment.
Yes, that's correct. The underlying problem it works around is a latency issue in the FreeRTOS task scheduler. We should re-test the reference drivers to confirm that the problem still exists before spending time re-integrating the alternative driver.
| https://github.com/netmindz/esp_dmx/#esp-idf-v5-fixes | ||
| ${env.lib_deps} | ||
| lib_ignore = | ||
| NeoESP32RmtHI |
There was a problem hiding this comment.
TODO: remove once NeoESP is fixed
|
@sombree - can you have a look at the commits I have made on this PR vs your changes please? |
|
It would be good to apply your V5 experience here @troyhacks |
This is a known incompatibility. Root Case: a new "ws2812 leds" driver in arduino-esp32, which pulls in the new RMT code that conflicts with any "legacy RMT" code. The solution is to disable the arduino driver (we don't need it) with the following build_flags, to prevent crashes at boot: -DESP32_ARDUINO_NO_RGB_BUILTIN ;; avoids RMT driver abort on startup "E (98) rmt(legacy): CONFLICT! driver_ng is not allowed to be used with the legacy driver" |
ouch 🤌 this looks ugly and fragile ... we should not mingle with linker sections, unless there is no other way. @sombree @willmmiles did you check the gcc documentation for other solutions, like |
As far as I'm aware, there isn't any other approach for constructing a static list across translation units at build time. This technique was called "Distributed Arrays" when formal language support was proposed. It's used in many places; an example in ESP-IDF is ESP_SYSTEM_INIT_FN. The key advantage is that it requires no RAM; the list exists entirely in code memory. It is my hope to leverage this technique for other static lists in the future, such as effects and bus drivers. What's changed is that someone at Espressif realized that the ".dtors" section will typically never be called upon in an embedded system, and removed it from the linker map to save some space. I'd packed the table in the ".dtors" section for that reason, and also because it's sorted by the linker script, so an order could be enforced. Now that it's been removed, however, all that data is discarded. For IDF >=v5.3, we'll need to add a linker fragment to embed the dynamic sections. We can keep the ".dtor.lst" name for compatibility with older systems. |
Update the code to handle at least being able to compile against V5
Edit (softhack007): espressif has "silently" changed a lot of APIs, and - in many cases - replacements are not behaving the same way as before. We are lucky that a few "legacy APIs" are still available, like the "V4" I2S driver. So even when "it compiles", there may be a lot more adaptations needed before "it really works".
Helpful information and migration guidance
coding guides
We should make sure that the "V5" branch will still compile for "V4" as a backup solution.
When adding code that only works in the new framework, it should be conditionally compiled, like in the examples below
related