-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Add first implementation of Wasm Audio Worklets, based on Wasm Workers. Fix Chrome not continuing the AudioWorklet processing if a number 1 is returned from the callback - must return 'true' specifically. Add new tone generator sample. Adjust comment. Use MessagePort.onmessage instead of add/removeEventListener(), since onmessage .start()s the MessagePort automatically. Fix name noise-generator to tone-generator Improve assertions. * Add src/audio_worklet.js to eslint ignore * Optimize code size * Add emscripten_current_thread_is_audio_worklet(), remove ENVIRONMENT_IS_AUDIO_WORKLET. * Fix to work with Closure * Simplify MINIMAL_RUNTIME shell module preamble generation. * Fix Closure and simple AudioWorklet creation * Fix Module import for AudioWorklets, and move towards globalThis * Disable -sAUDIO_WORKLET + -sTEXTDECODER=2 combo * Fix shell case * Mark AUDIO_WORKLET and SINGLE_FILE not mutually compatible * Default runtime support * Revert unnecessary code * Fix merge * AudioWorklets functioning in default runtime. * Remove #if WASM_WORKERS checks since Wasm Workers is unconditionally depended on. * Add more interactive tests * Test Audio Worklets with Closure
- Loading branch information
Showing
33 changed files
with
1,525 additions
and
53 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,188 @@ | ||
.. _wasm_audio_worklets: | ||
|
||
======================= | ||
Wasm Audio Worklets API | ||
======================= | ||
|
||
The AudioWorklet extension to the `Web Audio API specification | ||
<https://webaudio.github.io/web-audio-api/#AudioWorklet>`_ enables web sites | ||
to implement custom AudioWorkletProcessor Web Audio graph node types. | ||
|
||
These custom processor nodes process audio data in real-time as part of the | ||
audio graph processing flow, and enable developers to write low latency | ||
sensitive audio processing code in JavaScript. | ||
|
||
The Emscripten Wasm Audio Worklets API is an Emscripten-specific integration | ||
of these AudioWorklet nodes to WebAssembly. Wasm Audio Worklets enables | ||
developers to implement AudioWorklet processing nodes in C/C++ code that | ||
compile down to WebAssembly, rather than using JavaScript for the task. | ||
|
||
Developing AudioWorkletProcessors in WebAssembly provides the benefit of | ||
improved performance compared to JavaScript, and the Emscripten | ||
Wasm Audio Worklets system runtime has been carefully developed to guarantee | ||
that no temporary JavaScript level VM garbage will be generated, eliminating | ||
the possibility of GC pauses from impacting audio synthesis performance. | ||
|
||
Audio Worklets API is based on the Wasm Workers feature. It is possible to | ||
also enable the `-pthread` option while targeting Audio Worklets, but the | ||
audio worklets will always run in a Wasm Worker, and not in a Pthread. | ||
|
||
Development Overview | ||
==================== | ||
|
||
Authoring Wasm Audio Worklets is similar to developing Audio Worklets | ||
API based applications in JS (see `MDN: Using AudioWorklets <https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API/Using_AudioWorklet>`_), with the exception that users will not manually implement | ||
the JS code for the ScriptProcessorNode files in the AudioWorkletGlobalScope. | ||
This is managed automatically by the Emscripten Wasm AudioWorklets runtime. | ||
|
||
Instead, application developers will need to implement a small amount of JS <-> Wasm | ||
(C/C++) interop to interact with the AudioContext and AudioNodes from Wasm. | ||
|
||
Audio Worklets operate on a two layer "class type & its instance" design: | ||
first one defines one or more node types (or classes) called AudioWorkletProcessors, | ||
and then, these processors are instantiated one or more times in the audio | ||
processing graph as AudioWorkletNodes. | ||
|
||
Once a class type is instantiated on the Web Audio graph and the graph is | ||
running, a C/C++ function pointer callback will be invoked for each 128 | ||
samples of the processed audio stream that flows through the node. | ||
|
||
This callback will be executed on a dedicated separate audio processing | ||
thread with real-time processing priority. Each Web Audio context will | ||
utilize only a single audio processing thread. That is, even if there are | ||
multiple audio node instances (maybe from multiple different audio processors), | ||
these will all share the same dedicated audio thread on the AudioContext, | ||
and will not run in a separate thread of their own each. | ||
|
||
Note: the audio worklet node processing is pull-mode callback based. Audio | ||
Worklets do not allow the creation of general purpose real-time prioritized | ||
threads. The audio callback code should execute as quickly as possible and | ||
be non-blocking. In other words, spinning a custom `for(;;)` loop is not | ||
possible. | ||
|
||
Programming Example | ||
=================== | ||
|
||
To get hands-on experience with programming Wasm Audio Worklets, let's create a | ||
simple audio node that outputs random noise through its output channels. | ||
|
||
1. First, we will create a Web Audio context in C/C++ code. This is achieved | ||
via the ``emscripten_create_audio_context()`` function. In a larger application | ||
that integrates existing Web Audio libraries, you may already have an | ||
``AudioContext`` created via some other library, in which case you would instead | ||
register that context to be visible to WebAssembly by calling the function | ||
``emscriptenRegisterAudioObject()``. | ||
|
||
Then, we will instruct the Emscripten runtime to initialize a Wasm Audio Worklet | ||
thread scope on this context. The code to achieve these tasks looks like: | ||
|
||
.. code-block:: cpp | ||
#include <emscripten/webaudio.h> | ||
uint8_t audioThreadStack[4096]; | ||
int main() | ||
{ | ||
EMSCRIPTEN_WEBAUDIO_T context = emscripten_create_audio_context(0); | ||
emscripten_start_wasm_audio_worklet_thread_async(context, audioThreadStack, sizeof(audioThreadStack), | ||
&AudioThreadInitialized, 0); | ||
} | ||
2. When the worklet thread context has been initialized, we are ready to define our | ||
own noise generator AudioWorkletProcessor node type: | ||
|
||
.. code-block:: cpp | ||
void AudioThreadInitialized(EMSCRIPTEN_WEBAUDIO_T audioContext, EM_BOOL success, void *userData) | ||
{ | ||
if (!success) return; // Check browser console in a debug build for detailed errors | ||
WebAudioWorkletProcessorCreateOptions opts = { | ||
.name = "noise-generator", | ||
}; | ||
emscripten_create_wasm_audio_worklet_processor_async(audioContext, &opts, &AudioWorkletProcessorCreated, 0); | ||
} | ||
3. After the processor has initialized, we can now instantiate and connect it as a node on the graph. Since on | ||
web pages audio playback can only be initiated as a response to user input, we will also register an event handler | ||
which resumes the audio context when the user clicks on the DOM Canvas element that exists on the page. | ||
|
||
.. code-block:: cpp | ||
void AudioWorkletProcessorCreated(EMSCRIPTEN_WEBAUDIO_T audioContext, EM_BOOL success, void *userData) | ||
{ | ||
if (!success) return; // Check browser console in a debug build for detailed errors | ||
int outputChannelCounts[1] = { 1 }; | ||
EmscriptenAudioWorkletNodeCreateOptions options = { | ||
.numberOfInputs = 0, | ||
.numberOfOutputs = 1, | ||
.outputChannelCounts = outputChannelCounts | ||
}; | ||
// Create node | ||
EMSCRIPTEN_AUDIO_WORKLET_NODE_T wasmAudioWorklet = emscripten_create_wasm_audio_worklet_node(audioContext, | ||
"noise-generator", &options, &GenerateNoise, 0); | ||
// Connect it to audio context destination | ||
EM_ASM({emscriptenGetAudioObject($0).connect(emscriptenGetAudioObject($1).destination)}, | ||
wasmAudioWorklet, audioContext); | ||
// Resume context on mouse click | ||
emscripten_set_click_callback("canvas", (void*)audioContext, 0, OnCanvasClick); | ||
} | ||
4. The code to resume the audio context on click looks like this: | ||
|
||
.. code-block:: cpp | ||
EM_BOOL OnCanvasClick(int eventType, const EmscriptenMouseEvent *mouseEvent, void *userData) | ||
{ | ||
EMSCRIPTEN_WEBAUDIO_T audioContext = (EMSCRIPTEN_WEBAUDIO_T)userData; | ||
if (emscripten_audio_context_state(audioContext) != AUDIO_CONTEXT_STATE_RUNNING) { | ||
emscripten_resume_audio_context_sync(audioContext); | ||
} | ||
return EM_FALSE; | ||
} | ||
5. Finally we can implement the audio callback that is to generate the noise: | ||
|
||
.. code-block:: cpp | ||
#include <emscripten/em_math.h> | ||
EM_BOOL GenerateNoise(int numInputs, const AudioSampleFrame *inputs, | ||
int numOutputs, AudioSampleFrame *outputs, | ||
int numParams, const AudioParamFrame *params, | ||
void *userData) | ||
{ | ||
for(int i = 0; i < numOutputs; ++i) | ||
for(int j = 0; j < 128*outputs[i].numberOfChannels; ++j) | ||
outputs[i].data[j] = emscripten_random() * 0.2 - 0.1; // Warning: scale down audio volume by factor of 0.2, raw noise can be really loud otherwise | ||
return EM_TRUE; // Keep the graph output going | ||
} | ||
And that's it! Compile the code with the linker flags ``-sAUDIO_WORKLET=1 -sWASM_WORKERS=1`` to enable targeting AudioWorklets. | ||
|
||
Synchronizing audio thread with the main thread | ||
=============================================== | ||
|
||
Wasm Audio Worklets API builds on top of the Emscripten Wasm Workers feature. This means that the Wasm Audio Worklet thread is modeled as if it was a Wasm Worker thread. | ||
|
||
To synchronize information between an Audio Worklet Node and other threads in the application, there are two options: | ||
|
||
1. Leverage the Web Audio "AudioParams" model. Each Audio Worklet Processor type is instantiated with a custom defined set of audio parameters that can affect the audio computation at sample precise accuracy. These parameters are passed in the ``params`` array into the audio processing function. | ||
|
||
The main browser thread that created the Web Audio context can adjust the values of these parameters whenever desired. See `MDN function: setValueAtTime <https://developer.mozilla.org/en-US/docs/Web/API/AudioParam/setValueAtTime>`_ . | ||
|
||
2. Data can be shared with the Audio Worklet thread using GCC/Clang lock-free atomics operations, Emscripten atomics operations and the Wasm Worker API thread synchronization primitives. See :ref:`wasm_workers` for more information. | ||
|
||
3. Utilize the ``emscripten_audio_worklet_post_function_*()`` family of event passing functions. These functions operate similar to how the function family emscripten_wasm_worker_post_function_*()`` does. Posting functions enables a ``postMessage()`` style of communication, where the audio worklet thread and the main browser thread can send messages (function call dispatches) to each others. | ||
|
||
|
||
More Examples | ||
============= | ||
|
||
See the directory tests/webaudio/ for more code examples on Web Audio API and Wasm AudioWorklets. |
Oops, something went wrong.