WIP: Add basic audio mixer functionality

[levs42]

Description & motivation

• Feature: add audio mixing functionality

This is a work-in-progress PR to add audio mixing functionality. As IOAudioUnit is the main point to receive audio buffers, this PR adds an audio channel concept to the audio unit. Default channel is 0 and is handled by default audio resampler. When user wants to add an additional channel, they should use a different channel id. This will create a separate resampler. When the data is available from one of the resamplers, it's stored in a ringBuffer that is associated with every channel. Data from the default resampler triggers the mix() function that provides the mixed audio buffer back to the audio unit.

I need some assistance with this PR. Currently, I don't hear any audio on the live stream even when I copy the resampler data directly into the mix buffer. There are also some optimizations I would add later. For example, to avoid mixing when there is only one available channel. All comments are welcome!

Type of change

• New feature (non-breaking change which adds functionality)

[shogo4405]

1. The mix function has a high computational cost. Please use vDSP instead.
2. When considering ReplayKit, it's necessary to align the PTS (Presentation Time Stamp) since audio doesn't necessarily arrive continuously and may have gaps of several seconds. Additionally, you need to consider buffering for a certain period of time.
3. I thought an approach like changing the method signature, such as AudioMixer, would be like that.

[levs42]

Thank you for the feedback! For the first point, I've replaced the manual rendering with AVAudioMixerNode. AVAudioSourceNode is used to pass data in a callback style. It's available from macOS 10.15 so I had to bump the minimum version from 10.13; iOS also need a slight bump. For the second point, could you please help with the time stamp calculation? Current solution is mixing incoming audio but has an echo effect that I couldn't figure out yet how to fix. I wrote a simple audio renderer to provide a second source to the audio mixer and used it for macOS example.

[shogo4405]

1. When using AVAudioEngine, we understand that the AudioSession needs to be set to playAndRecord or higher. Due to compatibility issues with ReplayKit, we cannot adopt playAndRecord in HaishinKit.
2. The echo effect seems unrelated to timestamp issues. Can you clarify what specifically you mean? Are you referring to the situation when mixing audioApp and audioMic in ReplayKit? This occurs because the audio produced by audioApp is picked up by audioMic. The solution involves mixing only when headphones are plugged in.

[levs42]

1. I will try to use AVAudioEngine manual processing mode to see if can be a workaround.
2. The echo effect was appearing in headphones. I found out that main mixer is ignoring provided format here. The tap was installed for 48k bitrate, however the tap samples format was 41k. Looking for a fix.

[levs42]

I've added manual rendering and mixing works for macOS "Ingest Test" example. Mixing is triggered by the main channel. I used a mic and an audio file buffers from audio rendered. Didn't test it for iOS yet; buffering isn't implemented yet. Please take a look when having time.

[shogo4405]

1. I remember that with iOS, playAndRecord was required even for manual rendering. Did it work? I plan to try it out myself over the weekend.
2. Unfortunately, I don't have any knowledge regarding timestamp alignment using manual rendering with AVAudioEngine.

[levs42]

Just a small update, using AVAudioEngine indeed doesn't work nicely with ReplayKit. I'm getting an error when trying to start the engine:

AURemoteIO.cpp:1162 failed: 561015905 (enable 2, outf< 2 ch, 0 Hz, Float32, deinterleaved> inf< 2 ch, 0 Hz, Float32, deinterleaved>)

I will check if mixing can be done using Audio Toolbox before trying direct mixing.

[levs42]

I've replaced AVAudioEngine with Audio Units. I tested iOS, macOS and it works now. The audio isn't perfect, probably still need to add buffering. ReplayKit iOS example exceeds 50MB memory limit so need to do optimizations. Using AudioUnitConnection instead of deprecated AUGraph. CPU load looks good. No need to increase minimum targets now. I've added AudioFilePlayer and an audio file so you could try it out of the box. kAudioUnitSubType_MultiChannelMixer doesn't work on macOS and always produced empty buffers.

[levs42]

Somehow GitHub showing commits from main as being added to this branch. I'm going to reopen the PR.