Audio refactor

[Filoppi]

AUDIO CODE IS STILL WIP and will be split into multiple PRs.
This used to include input changes as well, but they have been moved to #9489.
Would benefit from: #9398

Audio:
-New audio mixer, capable of playing samples at any speed or sample rate, also has a lower latency overall.
-Audio now slows down when falling behind the target speed by "Audio Emu Speed Tolerance".
-Improved audio quality by changing from linear to cubic interpolation, I was able to tell the difference in multiple tests.
-When missing samples, in some occasions Dolphin now plays the last ~0.3ms of samples backwards, which are generally better than crackling and silence you would get during a larger stutter.
-New audio stretcher implementation, fixed quality loss at 100% speed, rarely ever crackles and is way more responsive than before. Also fixes the stretcher breaking at lower speeds like 20%.
-Mixer latency is now adaptive, meaning it has a range, from 0 to 40ms it is accepted, when it goes above 40ms it will try to go back to 20ms. This increases the quality overall as before the audio speed was constantly fluctuating around the target latency, without ever reaching it. Setting the max latency to 20-25ms is very doable if your emulation is fast and stable. You can customize the setting from the ini. It might be particularly useful in musical games.
-At higher emulation speeds, audio doesn't have an Aliasing filter applied (not even with stretching on), but you now can Mix at any sample rate, including up to 192kHz, fixing the problem.
-Mixing at higher sample rates is possible with all the Windows backends and is very easy to implement for other platforms, it gives the advantage of having a whole resampling without any losses (on Wii), for example, if mixing at 96kHz, 32kHz sources should now sound perfect (yes, it's hard to notice, but it's an improvement). The new setting is "Mix at OS Mixer sample rate".
-Fixed GC sample rates even more, they should now be exactly as they are on console, they were still using an approximated number and the DVD Stream audio rates were still wrong.
-DPLII fixes, also added an ini only setting to enable LTE/bass redirection, plus fixed DPLII + sound stretching combo. Added ability to enable and disable it at runtime. DPLII now automatically selects its best latency to be aligned with the backend one, to minimize the overall latency.
-New WASAPI implementation, fixed all the crashes, error checks/leaks, some compatibility problems, adding DPLII and loads of other stuff (like support for any sample rate).
-Improvements to all audio backends, especially regarding pausing/unpausing the emulation and memory leaks. Uniquely defined all sound backends in themselves, as a lot of their settings/capabilities were spread around the UI code.
-Added feature to cubeb and WASAPI to change all their settings while a game is running.
-Wii mote speaker should now sound better as it's not always on the brink of finishing samples (thus slowed down).
-4 emulated Wii mote speakers can now play at the same time.
-Removed any hardcoded references to output sample rate, AudioCommon::GetDefaultSampleRate() now holds the value (48kHz)
-Audio panel refactor and fixes (added info for when a backend or surround has failed starting)

What does this all mean?
If you have a machine that can play Dolphin games fine but struggles in some sections of the game, the audio quality should now be much better overall and be adaptive, meaning that when you lose frames it will follow the emulation speed, rarely ending up in cracking or missing samples/silence.
If you have an Android phone which plays game at a speed between 70 and 100%, this might improve the experience.
Similarly, if you want to play a game at 500% the game speed, you now can and it will sound great.

Porting to other Mac OS/Linux/Android is needed for:
-Get the OS mixer sample rate AudioCommon::GetOSMixerSampleRate()
-Testing cubeb latency support

[Filoppi]

TESTING

1. I would be grateful if a few people could test the new audio stretching, it's 98% done.
   Both on (very) low and high PCs and on Android.
   I need to find the best default value for a setting, which is the new "Emulation Speed Tolerance" in the Audio Panel.
   It is how much time the audio needs to fall behind the target emulation speed before falling back to the ACTUAL emulation speed itself.
   By target emulation speed I mean, the emulation speed the emulation tries to reach, whether 100%, 10% or 200%...
   By actual emulation speed I mean the speed at which the emulation is actually going, which is calculated by the audio mixer, counted from DMA samples pushes as had the best balance between number of pushes per second and accuracy.
   A value of about 10ms will immediately make the audio playback speed fallback to the actual emulation speed if there is a smaller stutter,
   which might be undesirable if you don't have sound stretching on, especially because it happens when you alt tab and stuff...
   But given that I have fixed the loss in quality from keeping the stretcher off, you might keep it on at all times now.
   Values too high will cause crackling in the first ms of a stutter, but it should never be as bad as before, for 2 reasons:
   When speed keeps going down, audio is slowed down dramatically so new sounds are always played, instead of padded silence,
   while is some other occasions, the last samples you had will play backwards, filling up the time with samples that sound good.
   If you are debugging and you don't want to be bothered with the audio being linked to the actual elapsed time, you can just disable it by sliding the slider left.

2. The changes in GC sample rates need testing, the main change is on the DVD Streaming sample rate, as they were still being pushed at 48kHz, not even the approximate GC value of 48043. While the DMA audio now uses the exact floating point sample rate from the GC. The difference from before is 1.000016, so 0.06 seconds per hour, not much, but still better than before.

3. Please test whether cubeb reacts to the latency settings on Linux, Max OS or if it ignores it.

4. Improved DPLII support for PulseAudio but I'm unable to test it.

This is the latest Win build if you can't build yourself https://github.com/Filoppi/dolphin/releases/tag/v1 (OUTDATED)

[Filoppi]

WASAPI changes:
Fix: WASAPI using wrong latency (device min latency + user selected latency, instead of the max of the 2)
Feature: WASAPI now works with 5.1/DPLII. It would be very easy to add support for any new audio formats now.
Feature: Added support to any sample rate to WASAPI (user selected, some are restricted but can be easily allowed).
Fix: WASAPI not initializing correctly in multiple cases, mostly related to latency.
Fix: Some WASAPI COM objects not being released.
Fix: WASAPI improved errors handling and log printing.
Fix: WASAPI crashing Dolphin if we changed the properties of the used device while a game was running, it now keeps trying to restarts WASAPI until Windows has finished reconfiguring it.
Fix: WASAPI stopping the game thread for the duration of the backend latency when the emulation was paused or stopped.
Fix: WASAPI not being fully thread safe. Crashing when pausing/unpausing, or stopping.
Fix: WASAPI COM initialization.
Fix: Dolphin hanging forever if WASAPI wasn't supported (same fix applies to all backends)
Fix: Possibly fixed other backends that create their own audio thread from crashing or hanging dolphin when pausing the emulation.
Feature: Restart WASAPI if a device has been changed or removed.
Feature: Automatically change the output device if the default device is changed from the Windows settings.
Feature: Automatically detect when latency is too low to keep up.
Feature: Automatically detect when a the input and output (Mixer against Audio Device) got out of sync and re-initialize WASAPI as there is no other way to avoid permanent crackling once you miss a period (can easily happen when the Dolphin process hangs for a bit).
Fix: WASAPI breaking when the process was being stopped (e.g. a breakpoint in VS) and then resumed.
Fix: memory leak/crash when closing dolphin when a game was running or stopping a game, with WASAPI on.
Feature: added the ability to change DPLII, latency, device and sample rate at runtime.
Feature: WASAPI now correctly follows the volume set to the application in the Windows mixer.

I doubt cubeb will ever implement exclusive mode, as it's not very useful for them, so this is good to have.

[JMC47]

@dolphin-emu-bot rebuild

[RinMaru]

no DPLII on HLE though?

[Filoppi]

no DPLII on HLE though?

No, this does not really touch the emulation side of the audio, just the mixing. I have no knowledge to fix that, I generally just use LLE anyway. DPLII is "greatly" improved though and it should work with audio stretching now (not yet, I will push the fix tonight).

[newperson1746]

Does this work transfer to linux also? I can very audibly tell audio quality degradation through dolphin vs the raw streams. For example, I can play the Mario Galaxy .asts converted to .wav (32khz) and theyll play at a perfect integer multiple (96khz) and sound excellent. Through dolphin, the high frequencies are all muddled; turns out this is because dolphin resamples to 48khz always.

[Filoppi]

Does this work transfer to linux also?

The audio conversion is independent from the OS, but there are a few things that are missing (or need testing) for Linux in this PR.
By the way work on this will resume in a few weeks.

[FishamanP]

I have great interest in DPLII and less stuttering while still maintaining low latency audio. Would love to see this PR polished and merged! Is any testing still needed from a higher-end Windows setup?

[Filoppi]

@altimumdelta thank you for you kind and understanding words... This PR is 98% finished (excluding review time) and just need to be split. I just have some other stuff going on in my life but I'm still following dolphin daily and finishing this is basically my highest priority in life as of now. So yeah I will finish it, I spent 3 months on that, I'm not letting it go to waste.
@FishamanP what needs testing is in a comment above.

[JosJuice]

@altimumdelta I would appreciate it if you stopped trying to onboard new contributors. The information you provide them with has frequently been wrong, and in this case, close to hostile.

[ghost]

@altimumdelta I would appreciate it if you stopped trying to onboard new contributors. The information you provide them with has frequently been wrong, and in this case, close to hostile.

Fine, though I just took the opportunity to state my opinion after I've put my extra of my own time into this which I wouldn't like to go to waste, if possible.

And could you specify what information is wrong in your view? I never received a complaint before.

[JosJuice]

And could you specify what information is wrong in your view? I never received a complaint before.

I can't point to very many specific things since I don't keep logs beyond the current day, but there does happen to be an example from the current day: The discussion about clang-format which started 5-6 hours ago. Part of the problem is that trying to correct everything you say would take up a lot of our time, which has been a frequent pattern since when you started trying to contribute a few years ago.

Anyway, I don't wish to derail the comments on this pull request further.

[Filoppi]

@altimumdelta I don't wanna carry on with this discussion, but personally, I feel like you don't fully know what you are talking about. Splitting this into multiple PR isn't really hard, most of the things don't overlap, I was just waiting to be done with everything before getting into the splitting part. And yes, I know that reviews take a long time, possibly months, as my older PRs are still to be merged, but there isn't much I can do about that.

Next time avoid saying stuff like:

Filoppi was in a huge rush getting some code done and hasn't familiarized himself with the contribution practices

because that is really not what happened? I just worked on different things at the same time because I felt like it? Then I was gonna split everything at the end, that's just the way I usually do and it works, it's not really a problem for others because I will split them once I'm done, and this PR is a draft.
So yes I could have planned it better from the start, but it's only my problem.
And saying I was in a rush to get code done? That couldn't be more wrong, I could have called it a day and submitted multiple times, but I target "perfection" and that's why I haven't finished it yet.

Let's stop the discussion here please. I will finish this once I do. Thanks.