Audio enhancements

[leo150]

I want to share some ideas and discuss possible changes of the audio implementation:

1. Support for selecting external audio channels. While the recent PR fixed an issue for some external audio mixers, it didn't solve it for every device. For example, there's no audio when connecting PreSonus® AudioBox USB® 96 25th Anniversary. I'm going to order one to test the hypothesis that we should use other channels rather than the first two.
2. Support of the sample rate conversion. Currently, the AudioCodec is converting from PCM to AAC using the same sample rate. I tried to use a different sample rate for the output format, but it makes the audio to sound laggy. For example, 48000 to 16000. It might make sense when the server doesn't support dynamic sample rate change: user started the stream with build-in mic (48k) and then connected a headphones (16k). This so far looks like the hardest change, because it requires some tweaks to the ring buffer.
3. To send the updated metadata to the server when audio format was changed. While it might be server specific, Twitch stream dies or has no audio when switching the sample rate without sending the metadata onMetaData packet. RTMPMuxer attempts to handle the audio converter change by sending AudioSpecificConfig but it has hardcoded to 44k and is ignored according to flv spec.

[shogo4405]

1. According to the product specifications, it states 24-bit, 96kHz. If 96kHz is the issue, then option 2 might have the potential to resolve it. I would appreciate it if you could share the contents of ASBD once you obtain it.
2. There is a plan for option 2 in the next version.
3. I believe that by maintaining the same sampling rate during streaming, this issue could be indirectly resolved.

[leo150]

Posting here some formats.

Focusrite Clarett2Pro USB (works fine)
mSampleRate: 48000.0, mFormatID: 1819304813, mFormatFlags: 12, mBytesPerPacket: 24, mFramesPerPacket: 1, mBytesPerFrame: 24, mChannelsPerFrame: 12, mBitsPerChannel: 16, mReserved: 0

external microphone, unknown device
mSampleRate: 48000.0, mFormatID: 1819304813, mFormatFlags: 12, mBytesPerPacket: 4, mFramesPerPacket: 1, mBytesPerFrame: 4, mChannelsPerFrame: 2, mBitsPerChannel: 16, mReserved: 0
(had noise, fixed by 615dc2f)

audiobox usb 96
mSampleRate: 48000.0, mFormatID: 1819304813, mFormatFlags: 12, mBytesPerPacket: 2, mFramesPerPacket: 1, mBytesPerFrame: 2, mChannelsPerFrame: 1, mBitsPerChannel: 16, mReserved: 0

All of them have same sample rate. One way I could get 16KHz format is to use bluetooth headphones:
mSampleRate: 16000.0, mFormatID: 1819304813, mFormatFlags: 12, mBytesPerPacket: 2, mFramesPerPacket: 1, mBytesPerFrame: 2, mChannelsPerFrame: 1, mBitsPerChannel: 16, mReserved: 0

[shogo4405]

I have implemented the sample rate conversion. While it still requires thorough debugging before the official release, I have tested it with the major scenarios. Therefore, I will merge it into the 'main' branch. #1294

[leo150]

Great news! Thanks a lot. Can't wait to test this feature in production.

[shogo4405]

there's no audio when connecting PreSonus® AudioBox USB® 96 25th Anniversary

I have added support for the downmix feature in this PR. It might be more user-friendly, especially when dealing with multiple channels. #1298. However, it seems that this method (AVAudioChannelLayout(layoutTag: kAudioChannelLayoutTag_DiscreteInOrder | numberOfChannels)) fails to downmix. Currently, it supports 2, 4, 5, 6, and 8 channels.

HaishinKit.swift/Sources/Util/AVAudioFormatFactory.swift

Lines 33 to 49 in a6725f4

	private static func makeChannelLayout(_ numberOfChannels: UInt32) -> AVAudioChannelLayout? {
	guard 2 < numberOfChannels else {
	return nil
	}
	switch numberOfChannels {
	case 4:
	return AVAudioChannelLayout(layoutTag: kAudioChannelLayoutTag_AudioUnit_4)
	case 5:
	return AVAudioChannelLayout(layoutTag: kAudioChannelLayoutTag_AudioUnit_5)
	case 6:
	return AVAudioChannelLayout(layoutTag: kAudioChannelLayoutTag_AudioUnit_6)
	case 8:
	return AVAudioChannelLayout(layoutTag: kAudioChannelLayoutTag_AudioUnit_8)
	default:
	return AVAudioChannelLayout(layoutTag: kAudioChannelLayoutTag_DiscreteInOrder | numberOfChannels)
	}
	}