Support Airplay 2 for audio reactive mode.

[digaus]

Is your feature request related to a problem? Please describe.
Feature request
Describe the solution you'd like
Add airplay 2 support so audio reactive can directly access music feed:
https://github.com/henriklied/esphome-airplay2

[softhack007]

@coderabbitai please evaluate if the suggested esphome library might be compatible with WLED audioreactive, or the core algorithms might be portable to WLED - assume a dual core MCU with PSRAM as the hardware target.

Include these aspects into your assessment. Extend the list based on your code review conclusions.

• AR needs a batch of 512 samples each 21ms, we cannot tolerate decoding delays of more than one batch; down-sampling to 22kHz is needed, this could be done by the audioSource class.
• WLED cannot re-configure esp-idf, we need to use the configuration as it is in tasmota arduino-esp32
• bluetooth, HTTPs encryption, or other non-standard protocols increase program size.
• is the AAC / ALAC decoder fast enough for realtime decoding, considering that we should keep the CPU load for Airplay below 50% of one core
• any potential compatibility issue with respect to asyncWebserver, asyncTCP, asyncUDP or WiFiUDP

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== GitHub issue ==='
gh api repos/wled/WLED/issues/5842 \
  --jq '{title, body, state, html_url, labels: [.labels[].name]}'

echo '=== GitHub issue comments ==='
gh api --paginate repos/wled/WLED/issues/5842/comments \
  --jq '.[] | {user: .user.login, created_at, body, html_url}'

echo '=== Candidate source and build files ==='
fd -HI -t f . | rg -i '(^|/)(platformio[^/]*|.*audio.*reactive.*|audio.*source.*|usermod.*audio.*|wled00/.*(platformio|.*audio.*))(\.[^/]+)?$' | head -200

echo '=== ESP32 build configuration references ==='
rg -n -i -C 3 'arduino-esp32|tasmota|psram|CONFIG_|sdkconfig|AsyncWebServer|AsyncTCP|AsyncUDP|WiFiUDP' \
  platformio.ini wled00 2>/dev/null | head -500

Length of output: 36542

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== esphome-airplay2 repository metadata ==='
gh api repos/henriklied/esphome-airplay2 \
  --jq '{default_branch, description, license: .license.spdx_id, archived, pushed_at, size}'

echo '=== esphome-airplay2 root tree ==='
gh api repos/henriklied/esphome-airplay2/git/trees/main?recursive=1 \
  --jq '.tree[] | select(.type == "blob") | .path' | head -300

echo '=== esphome-airplay2 manifest and dependency/configuration references ==='
for f in README.md manifest.json library.json platformio.ini CMakeLists.txt; do
  echo "--- $f ---"
  gh api "repos/henriklied/esphome-airplay2/contents/$f" \
    --jq '.content' 2>/dev/null | base64 -d 2>/dev/null || true
done

echo '=== esphome-airplay2 code indicators ==='
gh api repos/henriklied/esphome-airplay2/git/trees/main?recursive=1 --jq -r \
  '.tree[] | select(.type == "blob") | .path' |
while IFS= read -r f; do
  case "$f" in
    *.c|*.cc|*.cpp|*.h|*.hpp|*.py|*.yaml|*.yml)
      content="$(gh api "repos/henriklied/esphome-airplay2/contents/$f" --jq '.content' | base64 -d)"
      if grep -nEi 'aac|alac|decoder|esp[_-]?idf|idf[_-]?version|esp[_-]?adf|bluetooth|bt|mbedtls|tls|https|AsyncTCP|AsyncUDP|WiFiUDP|WiFiClient|lwip|udp|task|xTaskCreate|i2s|sample|buffer' <<<"$content" >/dev/null; then
        echo "--- $f ---"
        grep -nEi -C 2 'aac|alac|decoder|esp[_-]?idf|idf[_-]?version|esp[_-]?adf|bluetooth|bt|mbedtls|tls|https|AsyncTCP|AsyncUDP|WiFiUDP|WiFiClient|lwip|udp|task|xTaskCreate|i2s|sample|buffer' <<<"$content"
      fi
      ;;
  esac
done

echo '=== WLED AudioReactive structure ==='
ast-grep outline usermods/audioreactive/audio_reactive.cpp --items all --view expanded
ast-grep outline usermods/audioreactive/audio_source.h --items all --view expanded

echo '=== WLED AudioReactive timing, buffers, and source interface ==='
rg -n -i -C 5 '512|21|22050|22[.]?0|sample|buffer|fft|task|xTaskCreate|i2s|audioSource|read|process|psram' \
  usermods/audioreactive/audio_reactive.cpp usermods/audioreactive/audio_source.h

echo '=== Relevant WLED PlatformIO definitions ==='
sed -n '257,345p' platformio.ini

Length of output: 18576

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

repo="henriklied/esphome-airplay2"
ref="main"
files=(
  "components/airplay_receiver/manifest.json"
  "components/airplay_receiver/README.md"
  "components/airplay_receiver/airplay_receiver.cpp"
  "components/airplay_receiver/airplay_receiver.h"
  "components/airplay_receiver/platform/esp32/config.h"
  "components/airplay_receiver/platform/esp32s3/config.h"
  "components/airplay_receiver/audio/audio_decode_worker.cpp"
  "components/airplay_receiver/audio/audio_engine_v2.cpp"
  "components/airplay_receiver/audio/audio_receiver.cpp"
  "components/airplay_receiver/audio/audio_resample.cpp"
  "components/airplay_receiver/decoder/audio_decoder.cpp"
  "components/airplay_receiver/decoder/decoder_module.cpp"
  "components/airplay_receiver/crypto/crypto_module.cpp"
  "components/airplay_receiver/transport/transport_module.cpp"
  "components/airplay_receiver/transport/rtsp_conn.cpp"
  "components/airplay_receiver/transport/socket_utils.cpp"
  "components/airplay_receiver/timing/timing_module.cpp"
)

for f in "${files[@]}"; do
  echo "===== $f ====="
  gh api "repos/$repo/contents/$f?ref=$ref" --jq '.content' | base64 -d
done

echo '===== WLED audio_reactive.cpp outline ====='
ast-grep outline usermods/audioreactive/audio_reactive.cpp --items all --view expanded

echo '===== WLED audio_source.h outline ====='
ast-grep outline usermods/audioreactive/audio_source.h --items all --view expanded

echo '===== WLED AudioReactive source interface, timing, and processing ====='
rg -n -i -C 8 'class |struct |virtual |audioSource|512|21|22050|22[.]?0|sample|buffer|fft|task|xTaskCreate|i2s|process|read|psram' \
  usermods/audioreactive/audio_reactive.cpp usermods/audioreactive/audio_source.h

Length of output: 202260

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

repo="henriklied/esphome-airplay2"
ref="main"
files=(
  "components/airplay_receiver/manifest.json"
  "components/airplay_receiver/platform/esp32/config.h"
  "components/airplay_receiver/platform/esp32s3/config.h"
  "components/airplay_receiver/audio/audio_decode_worker.cpp"
  "components/airplay_receiver/audio/audio_engine_v2.cpp"
  "components/airplay_receiver/audio/audio_receiver.cpp"
  "components/airplay_receiver/audio/audio_resample.cpp"
  "components/airplay_receiver/decoder/audio_decoder.cpp"
  "components/airplay_receiver/decoder/decoder_module.cpp"
  "components/airplay_receiver/crypto/crypto_module.cpp"
  "components/airplay_receiver/transport/transport_module.cpp"
  "components/airplay_receiver/transport/rtsp_conn.cpp"
  "components/airplay_receiver/transport/socket_utils.cpp"
  "components/airplay_receiver/timing/timing_module.cpp"
)

pattern='esp[_-]?idf|idf_component|espressif/|esp_|freertos|xTask|task|core|affinity|aac|alac|decoder|faad|libav|buffer|psram|heap_caps|malloc|resampl|44100|22050|i2s|lwip|socket|bind|listen|accept|recvfrom|sendto|tcp|udp|mdns|mbedtls|tls|https|bluetooth'

for f in "${files[@]}"; do
  content="$(gh api "repos/$repo/contents/$f?ref=$ref" --jq '.content' | base64 -d)"
  echo "===== $f ====="
  printf '%s\n' "$content" | grep -nEi -C 3 "$pattern" || true
done

Length of output: 187261

---

@softhack007, I assessed the library proposed in #5842: henriklied/esphome-airplay2.

Conclusion

The library is not directly compatible with WLED.

A selective port is technically possible on a dual-core ESP32 or ESP32-S3 with PSRAM. The practical scope is large. The highest risks are framework compatibility, license compatibility, I2S ownership, real-time scheduling, and flash/RAM use.

The best reuse boundary is decoded PCM. Do not attempt to insert the library as an ESPHome component or as a normal WLED usermod.

Area	Assessment
Direct ESPHome component reuse	Not compatible
Port AirPlay 2 transport and decoder to WLED Arduino	Possible, high effort and high risk
Reuse the PCM-to-FFT integration concept	Feasible
Reuse AAC/ALAC decoder through esp_audio_codec	Not portable to all WLED targets without component integration work
Meet the 21 ms AudioReactive deadline	Possible only after target measurements
Keep AirPlay below 50% of one core	Unproven
Merge upstream code into WLED	Blocked unless license permission is obtained

Framework and ESP-IDF compatibility

The library explicitly requires:

• ESP-IDF v5.5 or later.
• ESPHome.
• ESP-IDF component manager.
• espressif/esp_audio_codec.
• espressif/mdns.
• espressif/libsodium.
• esp_driver_i2s and the ESP-IDF standard I2S driver.

WLED uses Arduino builds from the Tasmota ESP32 platform. WLED has environments based on ESP-IDF 4.4.8 and 5.5.4. WLED cannot enable or tune ESP-IDF Kconfig options as this library expects.

This blocks direct integration for two reasons:

1. WLED’s IDF v4 environments do not meet the library’s ESP-IDF v5.5 requirement.
2. WLED’s Arduino build does not provide the ESPHome component-manager integration path.

The ESP-IDF v5 WLED environments reduce the API-version gap. They do not remove the component-manager, linker, Kconfig, and Arduino/ESPHome integration work.

Recommendation: support only WLED ESP-IDF v5 targets if development proceeds. Do not target IDF v4 builds for the AirPlay receiver.

AudioReactive timing and PCM handoff

WLED AudioReactive requires 512 mono samples at 22,050 Hz. This is approximately 23.22 ms of source audio. FFT_MIN_CYCLE is 21 ms. The existing FFTcode() task runs on core 0 and calls:

audioSource->getSamples(valFFT, samplesFFT);

This gives a suitable abstraction boundary.

A new AirPlaySource can implement AudioSource::getSamples() and supply down-sampled mono PCM to FFTcode().

The implementation must meet these requirements:

• The decoder must never run inside getSamples().
• getSamples() must never wait for network input.
• getSamples() must never wait for a decoder mutex.
• getSamples() must copy from a pre-filled PCM ring only.
• The PCM ring must hold more than one 512-sample batch.
• The system must define a deterministic underrun policy. Silence or repetition is safer than blocking.
• The handoff must preserve ordering across seek, pause, flush, and reconnect events.
• The producer must down-mix stereo before or during resampling.
• The source must reset filter and resampler state after an AirPlay flush or stream change.

The ESPHome library uses a queued decode worker and a large timeline buffer. That design is correct in principle. It must be reduced and adapted for WLED’s memory budget and AudioReactive latency requirement.

Down-sampling to 22,050 Hz

The AirPlay library normally uses 44,100 Hz stereo PCM. Conversion from 44,100 Hz to 22,050 Hz is an exact 2:1 ratio.

The library currently uses a 32-tap floating-point sinc resampler. This is portable code, but it is not the best first option for WLED AudioReactive.

For WLED, use a dedicated 2:1 low-pass decimator:

1. Down-mix 44,100 Hz stereo to mono.
2. Apply an anti-alias low-pass filter.
3. Keep every second sample.
4. Write 512-sample batches to the AirPlaySource ring.

This avoids general-purpose floating-point resampling cost and reduces the PCM bandwidth by 75% relative to 44.1 kHz stereo.

Do not use simple “take every second sample” decimation without low-pass filtering. It will alias content above 11.025 kHz into AudioReactive FFT bands.

The resampler also adds filter delay. The delay is acceptable for visual effects if it is stable. It must not cause the PCM ring to miss a 512-sample deadline.

AAC and ALAC real-time performance

The library does not include measured decode-time data for AAC or ALAC.

It calls esp_aac_dec_decode() and esp_alac_dec_decode() from espressif/esp_audio_codec. Source inspection alone cannot prove that either decoder stays below the requested 50% of one core.

The library’s task design shows that decode can be costly:

• It uses a dedicated priority-6 decode task.
• It gives RTP/control tasks priority 7 and 8.
• It gives I2S playback priority 9.
• It pins the decoder to core 1.
• It uses a queue of up to 16 compressed packets.
• It states that inline AAC decoding could stall socket reads.

This is a good architecture for an audio receiver. It is not evidence that the 50% target is met on WLED hardware.

Required benchmark

Measure each codec separately on the intended MCU, compiler flags, and WLED feature set.

Record:

• Mean and maximum AAC decode time per frame.
• Mean and maximum ALAC decode time per frame.
• Mean and maximum decrypt time per packet.
• Mean and maximum 44.1 kHz stereo to 22.05 kHz mono conversion time.
• Queue depth high-water mark.
• PCM ring low-water mark.
• Audio underrun count.
• AudioReactive missed-batch count.
• Core 0 and core 1 idle time.
• Flash size and free internal DRAM.
• PSRAM allocation and peak use.
• Wi-Fi throughput and packet-loss behavior during LED output.

Acceptance criteria should include:

• No AirPlaySource::getSamples() delay above the 21 ms processing interval.
• No PCM-ring underrun during at least 30 minutes of AAC and ALAC playback.
• AirPlay processing below 50% of its assigned core at sustained playback.
• AudioReactive FFT remains within its existing timing budget.
• No loss of WLED web UI, OTA, MQTT, UDP sync, or Wi-Fi stability.

ALAC may be more predictable because it is lossless and commonly uses 352-sample frames. AAC uses 1024-sample frames in this library. AAC can produce burstier decode work. The decode queue and PCM ring must absorb that burst without allowing a delayed frame to reach getSamples().

Core allocation

The existing WLED AudioReactive FFT task is pinned to core 0.

The proposed library pins audio receive, decode, and playback work to core 1. It places RTSP and Wi-Fi-facing tasks on core 0.

This allocation is reasonable only if measurements confirm it:

Core	Suggested work
Core 0	WLED main loop, AsyncTCP, AsyncWebServer, Wi-Fi/lwIP, AudioReactive FFT
Core 1	AirPlay decode, down-mix, decimation, PCM producer

Do not run AirPlay decoding on core 0. It would compete directly with WLED’s FFT task and the network stack.

The library itself notes that an unpinned decoder can move to core 0 and starve the main loop. WLED must retain explicit core affinity.

Task priority also needs review. WLED documents FFTTASK_PRIORITY as 1 by default. The AirPlay library uses priority 6 for decoding and higher priorities for receive and playback. Priority changes must not starve WLED LED timing or AsyncTCP processing.

I2S conflict

This is a major integration issue.

WLED AudioReactive configures and owns I2S_NUM_0 for microphone input. The AirPlay library owns an I2S output path for an external DAC. It expects the ESP-IDF standard I2S driver.

A microphone and DAC cannot both independently install and reconfigure the same I2S port.

The AirPlay-to-AudioReactive feature does not need an I2S DAC. Therefore:

• Remove the AirPlay I2S output module from the WLED port.
• Do not initialize the library’s audio-output task or its I2S standard driver.
• Export decoded PCM only to a WLED AirPlaySource.
• Make AirPlay input mutually exclusive with local microphone input unless the hardware and driver model support separate I2S peripherals.

If future work also requires AirPlay speaker output, use a separate I2S peripheral and validate the target-specific peripheral count and pin routing.

AsyncWebServer, AsyncTCP, AsyncUDP, and WiFiUDP

The library does not use AsyncWebServer, AsyncTCP, AsyncUDP, or WiFiUDP for its AirPlay transport.

It uses native lwIP/POSIX sockets directly:

• TCP listener on port 7000 for RTSP.
• TCP event listener.
• TCP buffered audio stream.
• UDP realtime audio stream.
• UDP timing and control sockets.
• mDNS advertisement.
• PTP and NTP timing sockets.

This can coexist with WLED’s AsyncTCP and AsyncWebServer at the API level. It does not guarantee runtime compatibility.

The important risks are:

1. lwIP socket count.
   The library comments reference CONFIG_LWIP_MAX_SOCKETS of 10. WLED cannot change this through ESP-IDF configuration. WLED already uses HTTP, WebSocket, OTA, UDP, MQTT, and other network features. Socket exhaustion is possible.

2. Port conflicts.
   RTSP port 7000 must be reserved. Dynamic stream ports must not conflict with existing WLED services.

3. Wi-Fi task contention.
   Native socket receive loops and AirPlay timing traffic increase load on the same Wi-Fi/lwIP subsystem used by AsyncTCP and WiFiUDP.

4. Multicast traffic.
   WLED AudioReactive uses WiFiUDP multicast for sound sync. AirPlay uses mDNS and PTP multicast. Test multicast joins, AP mode, STA reconnect, and Wi-Fi roaming together.

5. Blocking behavior.
   The AirPlay library has blocking accept() and recv() loops in dedicated tasks. This is acceptable only if task priorities and shutdown behavior remain isolated from WLED tasks.

6. mDNS service registration.
   WLED and AirPlay must share one mDNS subsystem. Duplicate initialization or incompatible mDNS wrappers can fail.

7. Network recovery.
   Test Wi-Fi disconnect, reconnect, DHCP renewal, sender handover, stream seek, and OTA while a session is active.

The preferred approach is to keep native sockets for the AirPlay protocol. Rewriting the RTSP and RTP portions on AsyncTCP/AsyncUDP would increase scope and risk. The port must include hard socket limits, deterministic close behavior, and diagnostics for open descriptors.

Flash, internal RAM, and PSRAM

The library adds substantial code and data:

• esp_audio_codec.
• libsodium.
• mDNS.
• mbedTLS pairing and control encryption.
• AAC and ALAC decoders.
• RTSP parser and binary-plist parser.
• PTP/NTP timing.
• Audio buffering.
• Resampling.

Bluetooth is not required by this library. It is disabled for ESP32-S3 profiles. Do not enable Bluetooth for this feature.

HTTPS is not a required AirPlay transport dependency. AirPlay pairing and encrypted RTSP control are required for AirPlay 2. They cannot be removed without changing protocol support.

The library’s ESP32-S3 profile uses a 1000-frame PCM ring. It documents this as approximately 1.42 MiB. It also documents an engine allocation of approximately 790 KiB. This is feasible with 8 MiB PSRAM. It is not suitable as-is for all WLED PSRAM boards.

PSRAM can hold long-lived compressed and PCM buffers. Do not put these real-time data structures in PSRAM without measurement:

• DMA buffers and descriptors.
• The 512-sample AudioReactive handoff buffer.
• Decoder working buffers that are accessed continuously.
• Queue metadata.
• Synchronization objects.
• Stack-critical data.

The library itself has an option to put decoder scratch buffers in PSRAM. This reduces internal DRAM pressure but can increase cache misses and decode jitter. Evaluate both configurations.

License compatibility

This is a blocking project issue.

The library is marked non-commercial. Its README says commercial use requires upstream written permission. It also says its terms are inherited from rbouteiller/airplay-esp32.

WLED cannot merge, distribute, or derive from this code until the project confirms license compatibility and obtains any required permission. A technical port should not start before this is resolved.

An independent clean-room implementation can avoid code-license reuse. It does not remove AirPlay protocol, patent, interoperability, and maintenance considerations.

Recommended scope

Use a staged approach.

1. Resolve license permission before source reuse.
2. Limit the first target to ESP32-S3 with at least 8 MiB PSRAM and WLED IDF v5.
3. Remove speaker/I2S-output support from the AirPlay port.
4. Create an AirPlay PCM producer on core 1.
5. Implement AirPlaySource : AudioSource for WLED AudioReactive.
6. Use an internal-DRAM, bounded PCM ring between the producer and FFTcode().
7. Implement a fixed 44.1 kHz to 22.05 kHz mono decimator.
8. Keep WLED FFT on core 0.
9. Disable local microphone input while AirPlay input is active.
10. Add counters for decode failures, packet drops, queue depth, PCM underruns, FFT missed batches, socket allocation failures, and reconnects.
11. Benchmark AAC and ALAC before enabling the feature by default.
12. Test with AsyncWebServer, AsyncTCP, WiFiUDP sound sync, OTA, MQTT, and Wi-Fi roaming.

The PCM handoff and fixed-ratio decimation are portable. The full AirPlay 2 receiver is not a small library dependency. It is a separate networked audio subsystem that must be adapted to WLED’s fixed Arduino/Tasmota build configuration.

You are interacting with an AI system.

[softhack007]

@digaus sorry, but I think that at least the licensing situation is a blocker for WLED.

🕮 Licensing

• Personal and hobby use, modification and redistribution are permitted.
• Commercial use -- selling devices with this firmware, or shipping it in a product -- requires written permission from Remi Bouteiller.

We cannot prevent that WLED is used in a commercial environment - actually the WLED license was chosen so it allows to redistribute WLED with commercial hardware devices that are sold for money. The WLED development team also has no control over who sells such devices - even a generic "free to distribute with WLED" permission would not work, according to the autor's terms. For the same reasons, a "click to accept terms" approach is impossible - it does not satisfy the requirement of a "written permission" for each commercial distributor.

🏗️ Heavy Lift

According to the AI analysis from above, we cannot use the espHome component as a library, because our build environment is not compatible. We would need to port parts of the source code from esphome-airplay2 into WLED audioreactive, which brings us back to the licensing problem.