Async encode/decode, and encoding an AudioBuffer

[eliot-akira]

Hello, thank you for creating this library. I had some questions about how to use it.

From what I understand in wasm/index.js, the encode/decode functions are synchronous and they block the main thread in browser/node. Is that true?

If so, is it possible to load this library from a worker, to make it async/non-blocking? I'll see if I can achieve it and report back here if successful.

Looking at other similar libraries, wasm-media-encoders, wasm-audio-decoders, and opusscript, I see their API is synchronous and blocking also. ffmpeg.wasm and lame-wasm are async.

That's interesting because I would have expected, for most purposes, async encode/decode is preferred for both browser and server side, so as not to block the main thread. For example, how AudioContext.decodeAudioData() works.

---

Currently, I'm encountering some difficulties trying to encode (and decode back) an instance of AudioBuffer.

First, I noticed that if the sample rate is 44100, the Encoder/Decoder constructor throws an error, opus: invalid argument.

const { encode } = new Encoder({
  channels: audioBuffer.numberOfChannels,
  sample_rate: audioBuffer.sampleRate
})

If I force it to 48000, it doesn't throw.

Next, I learned how to extract PCM data from an AudioBuffer using getChannelData().

const buffers = [] // Float32Array per channel

for (let i=0, len=audioBuffer.numberOfChannels; i < len; i++) {
  buffers.push(audioBuffer.getChannelData(i))
}

When I pass a channel's data to encode(), it throws RangeError: source array is too long.

The error is from the first line of RawEncoder.encode() in @evan/wasm/target/opus/deno.js.

encode(buffer) {
    bptrs.set(buffer = new Uint8Array(buffer.buffer, buffer.byteOffset, buffer.byteLength));
    return pptrs.slice(0, err(wasm.opus_encode(this.#ptr, bptr, buffer.length / 2 / this.channels, pptr, pptrl)));
}

..So I should convert my Float32Array to an Uint8Array before passing it to encode. Is that correct?

const channelData = buffers[0].buffer // Get ArrayBuffer from Float32Array
const pcm = new Uint8Array(channelData)
const opusData = encode(pcm)

The conversion seems to work, but still results in the same error, RangeError: source array is too long.

EDIT: I'm guessing it's because I set the wrong sample rate when instantiating the encoder. And probably it's impossible (or unwise, haha) to try to up-scale from 44.1 to 48 kHz.

---

Well, I'll try to figure out these questions on my own, but if you have any thoughts, I'd appreciate it.

[eliot-akira]

For the async question, and about loading the library from a worker, it looks like I'm thinking in the right direction.

Why Put Your WebAssembly Code in a Web Worker?

The critical aspect of putting a WebAssembly module in a Web Worker is that it removes the overhead of fetching, compiling and initialising a WebAssembly module off the main thread, and in turn calling the given functions in a module. This keeps the main browser thread free to continue rendering and handling user interactions. Considering that WebAssembly is often used for processing intensive code, pairing it with Web Workers can be a great combination.

However, there are some drawbacks to this approach. Transferring data from the main thread to the worker thread could be costly depending on the size of the data in question. There is also additional complexity and logic to handle when using WebAssembly in a Web Worker. Placing a WebAssembly module in a Web Worker makes interacting with the WASM module code asynchronous because the message passing mechanism leverages event listeners and callbacks.

From: Using WebAssembly with Web Workers

So it sounds useful if there's an async and non-blocking wrapper for the library, as long as the data being passed to and from the worker is a transferable object like ArrayBuffer.

---

The library wasm-audio-decoders mentioned earlier actually has both sync/async APIs.

WASM Audio Decoders is a collection of Web Assembly audio decoder libraries that are highly optimized for browser use. Each module supports synchronous decoding on the main thread as well as asynchronous (threaded) decoding through a built in Web Worker implementation.

[eliot-akira]

About the sample rate, I found the answer in the Opus codec manual.

Sampling rate of input signal (Hz) This must be one of 8000, 12000, 16000, 24000, or 48000.

From: opus_encoder_create()

That makes sense why 44100 is an invalid argument for the encoder.