Add multichannel audio mixer

[levs42]

Description & motivation

This is a PR based on #1380. It replaces audio resampler with an audio mixer. Everything works as before when there is just one input source. Once there is more, audio mixer creates an audio graph; adds a resampler for each input source; starts the mixing process. It is using plain AudioUnit components instead of AVAudioNode to avoid interacting with audio session and instead of AUAudioUnit to avoid deprecated AUGraph.

Users can provide dedicated audio settings for every input source with sourceSettings. This allows to set channelMap and other parameters independently for every resampler. However, output sampleRate and channels are enforced by the IOMixer to be the same as for the default resampler. This ensures that mixer node works with the same audio format from every source. Changes of settings and audio format of the default resampler are monitored for this purpose.

All interfaces and default behavior is kept the same.

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Screenshots:

[shogo4405]

The PR is large, so I'll need some time for the review.

[shogo4405]

I tried it with ReplayKit@Control Center at hand. Only the audioApp could be heard. What device was it tested on? Just for reference, I'll leave the FormatDescription here.

iPhone 15 Pro max + iOS17.3 a475e05

audioMic

CMSampleBuffer 0x102506830 retainCount: 4 allocator: 0x1e8f806a8
	invalid = NO
	dataReady = YES
	makeDataReadyCallback = 0x0
	makeDataReadyRefcon = 0x0
	formatDescription = <CMAudioFormatDescription 0x283311900 [0x1e8f806a8]> {
	mediaType:'soun' 
	mediaSubType:'lpcm' 
	mediaSpecific: {
		ASBD: {
			mSampleRate: 48000.000000 
			mFormatID: 'lpcm' 
			mFormatFlags: 0xc 
			mBytesPerPacket: 2 
			mFramesPerPacket: 1 
			mBytesPerFrame: 2 
			mChannelsPerFrame: 1 
			mBitsPerChannel: 16 	} 
		cookie: {(null)} 
		ACL: {(null)}
		FormatList Array: {
			Index: 0 
			ChannelLayoutTag: 0x640001 
			ASBD: {
			mSampleRate: 48000.000000 
			mFormatID: 'lpcm' 
			mFormatFlags: 0xc 
			mBytesPerPacket: 2 
			mFramesPerPacket: 1 
			mBytesPerFrame: 2 
			mChannelsPerFrame: 1 
			mBitsPerChannel: 16 	}} 
	} 
	extensions: {(null)}
}

audioApp

CMSampleBuffer 0x1047130f0 retainCount: 4 allocator: 0x1e8f806a8
	invalid = NO
	dataReady = YES
	makeDataReadyCallback = 0x0
	makeDataReadyRefcon = 0x0
	formatDescription = <CMAudioFormatDescription 0x283ab08c0 [0x1e8f806a8]> {
	mediaType:'soun' 
	mediaSubType:'lpcm' 
	mediaSpecific: {
		ASBD: {
			mSampleRate: 44100.000000 
			mFormatID: 'lpcm' 
			mFormatFlags: 0xe 
			mBytesPerPacket: 4 
			mFramesPerPacket: 1 
			mBytesPerFrame: 4 
			mChannelsPerFrame: 2 
			mBitsPerChannel: 16 	} 
		cookie: {(null)} 
		ACL: {(null)}
		FormatList Array: {
			Index: 0 
			ChannelLayoutTag: 0x650002 
			ASBD: {
			mSampleRate: 44100.000000 
			mFormatID: 'lpcm' 
			mFormatFlags: 0xe 
			mBytesPerPacket: 4 
			mFramesPerPacket: 1 
			mBytesPerFrame: 4 
			mChannelsPerFrame: 2 
			mBitsPerChannel: 16 	}} 
	} 
	extensions: {(null)}
}

[shogo4405]

[Q]
It's convenient for debugging, but I prefer to avoid having it in the library to prevent changing the default behavior. Will moving it to an example still not work as expected?

[levs42]

AudioNode is a new class so I implemented a custom description. I can rename it to something else. Is it more about reimplementing a custom description for AudioStreamBasicDescription? It's very handy to print mFormatFlags as a readable flags. For example,

mFormatFlags: 12 AudioFormatFlags(audio_IsPacked | audio_IsSignedInteger | ll_32BitSourceData | pcm_IsPacked | pcm_IsSignedInteger)

instead of just mFormatFlags: 12.

[shogo4405]

Regarding custom classes in HK, that's fine. However, please keep the Extension folder for Cocoa frame-only extensions. Please move it to the bottom of the definition file.

As for AudioStreamBasicDescription, I haven't adopted it because the default behavior changes. I acknowledge that it's convenient for debugging, but we often avoid it in application-level development due to the numerous problems it can cause.

[levs42]

Sounds good, I'll move it tomorrow. Can I keep AudioStreamBasicDescription extension while removing description override and rename it to something like verboseDescription for debug purpose?

[shogo4405]

[Must]

• createTrack -> makeTrack
  Following Apple's Cocoa naming conventions, factory methods adopt the 'make' convention.

[levs42]

Renamed to makeTrack.

[levs42]

For the original issue of mic audio being silent, I've pushed 7f837a7 with the fix. I was testing with custom audio settings, so resamplers were calling delegate methods and ring buffers were initialized. However, the delegate method wasn't called for default settings, so ring buffer was nil. Now ring buffer always has a default value for current output format.

[shogo4405]

I tried it with 44e506f using ReplayKit@Control Center, but this time the microphone audio was coming in, while the app audio was not. Please check if the same issue occurs on your end. The testing device is the same as mentioned above.

[levs42]

Please try the new version. I've tested Screencast example with and without mic enabled from the start.

[shogo4405]

Thank you very much. It's a great feature! The built-in microphone on the iPhone and external USB headphones seem to be working fine without any noticeable audio delay.

However, during the next round of testing, the following issues were discovered:

1. When streaming with a Bluetooth microphone, no microphone audio is captured.
2. During streaming, an incoming call was received. After dismissing the call without answering, the app's audio ceased to be captured.
   • In some cases, due to ReplayKit behavior, app audio may not be captured, which may be considered acceptable as per the specifications.

[levs42]

Awesome. Thank you for testing!

1. Do you have a recommendation for a Bluetooth microphone model? Also could you please provide the ASBD?
2. Does this only happen to ReplayKit? I did a test on 17.4 and have the same behavior. processSampleBuffer receives only video buffers after a phone call. I will submit a bug report to Apple next week.

[shogo4405]

The standard equipment adopted is the AirPods Pro. The ASBD (AudioStreamBasicDescription) is as follows:
<AVAudioFormat 0x2828fc690: 1 ch, 24000 Hz, Int16> numSamples = 1024
Some inexpensive Bluetooth microphones may operate at 16kHz, so I'd like to try at that frequency as well. However, I don't know the product name.

[levs42]

I've tested 3 headphones with my iPhone 11 Pro Max, iOS 17.4.1 and wasn't able to reproduce the issue. Here are my tests:

Summary

Test 1: starting with Apple AirPods Pro
channel:0, inputFormat:<AVAudioFormat 0x300fe54f0: 1 ch, 24000 Hz, Int16>
channel:1, inputFormat:<AVAudioFormat 0x300fe9c20: 2 ch, 44100 Hz, Int16, interleaved>
// Switched to built-in mic

Test 2: starting with built-in mic
channel:0, inputFormat:<AVAudioFormat 0x302435720: 1 ch, 48000 Hz, Int16>
channel:1, inputFormat:<AVAudioFormat 0x302435fe0: 2 ch, 44100 Hz, Int16, interleaved>
// Switched to AirPods

Test 3: start with Beats Fit Pro (Apple)
channel:0, inputFormat:<AVAudioFormat 0x3008a9c70: 1 ch, 24000 Hz, Int16>
channel:1, inputFormat:<AVAudioFormat 0x3008a9ef0: 2 ch, 44100 Hz, Int16, interleaved>
// Switched to built-in mic

Test 4: starting with built-in mic
channel:0, inputFormat:<AVAudioFormat 0x303bd5b80: 1 ch, 48000 Hz, Int16>
channel:1, inputFormat:<AVAudioFormat 0x303bd6210: 2 ch, 44100 Hz, Int16, interleaved>
// Switched to Beats

Test 5: start with non-Apple headphones (Bose QC 35 II)
channel:0, inputFormat:<AVAudioFormat 0x302765a90: 1 ch, 16000 Hz, Int16>
channel:1, inputFormat:<AVAudioFormat 0x303bd6210: 2 ch, 44100 Hz, Int16, interleaved>
// Switched to built-in mic

Test 6: starting with built-in mic
channel:0, inputFormat:<AVAudioFormat 0x303bd5b80: 1 ch, 48000 Hz, Int16>
channel:1, inputFormat:<AVAudioFormat 0x303bd6210: 2 ch, 44100 Hz, Int16, interleaved>
// Switched to Bose

I noticed that iOS keeps the initial audio format and performs the audio conversion before providing it to the ReplayKit sample handler when user changes the audio source. For example, using if users starts the broadcast with 16KHz headphones connected, disconnecting them doesn't switch back to 48KHz.

Could you please verify couple of things when using AirPods Pro?

1. The sample handler receives a buffer here
2. The audio mixer appends the audio buffer here
3. After appending it calls mix() function and the mix function calls the delegate method here
4. And the audio buffer for channel 0 actually has audio audioBuffer.mutableAudioBufferList.pointee.mBuffers.mData in XCode's Memory Browser

This covers the default mixing troubleshooting. Also please let me know if func audioMixer(_ audioMixer: IOAudioMixer, errorOccurred error: IOAudioUnitError) delegate method is getting called at any moment. Thank you.

[shogo4405]

I noticed that iOS keeps the initial audio format and performs the audio conversion before providing it to the ReplayKit sample handler when user changes the audio source. For example, using if users starts the broadcast with 16KHz headphones connected, disconnecting them doesn't switch back to 48KHz.

This can be switched with some actions. Here are the ones I know:

Triggering an alarm using the clock function.
Tapping the notch area to trigger a stop broadcast alert.
AudioSession is reset when moving apps, etc., and may switch to 48 kHz at some point.
I think it's unnecessary to support these actions up to this point.

[shogo4405]

I tried it just now and it worked as expected. Thank you.

Since the audio part is fundamental, merging it as it is and releasing it may lead to many criticisms.
Therefore, I plan to proceed with the following steps:

1. In version 1.7.x, make changes to method signatures and other related changes to accommodate this external API change.
2. In version 1.8.0,
   • Switch between this mixing feature and the existing audio processing, with the default being the existing audio.
   • Make adjustments to allow switching the processing via rtmpConnection.requireNetworkFramework as an example.
3. In version 1.9.0,
   • After confirming stability, make this mixing process the default.

Please let me work on the branch shogo4405:feature/audio-mixer for now.

[levs42]

This plan looks great! Thank you.