Enable multi-input to be mixed via Jamulus

[emlynmac]

I'd like to start working on expanding the multi-track capabilities of Jamulus.

I have a 16 channel firewire mixer and one of the things I'd like to be able to do is to have individual channels appear as users in the Jamulus mixer.

I'm going to start looking into the code as to how this could be done, but any help would be greatly appreciated or any insights that would make this a no-go also keenly wanted!

To be clear, this is about having multiple up streams to the server from the same host, from different channels of the same interface. On the macOS side, this will be about manipulating the core audio call backs into the stream to the server.
One question here is whether the audio up stream will be multiplexed or independent. Probably the latter, in order to avoid a change to the protocol but I have yet to look into it.

Just wanting to get a discussion going around this while I have a look at the how.

Thanks!

[dcorson-ticino-com]

Hi Emlyn,
Thanks for your initiative. I think that you have a very special case that not many users have, maybe something for a fork.
On the other hand, however, I think that everyone who has an external interface has two input channels that they might profit from being able to control separately. Maybe that would interest you to implement to get the ball rolling.

I have thought about this some myself, but haven't looked at the code at all about how to implement it.
My thoughts are the following:
A client has two input and two output channels. We now have:

• stereo input which places the two input channels on the respective output channels,
• mono input with adds the two inputs and places them in a mono stream to the server and
• mono in/stereo out which adds the two input channels and places that on both output channels.

I think the logical step forward would be to allow each mono input channel to have a stereo output channel, in essence giving each client two stereo streams. This would allow separate volume and pan controls for the two mono inputs, for example microphone and guitar. Ideally this could be done with no change to the server (the server would see two different clients) so it would work with older servers too. I would think that the two stereo channels could have names that would be appended to the client name, being defined in the setup window, so the server can manage them separately and they can be shown separately on the client mixerboard.

Does that make sense to you? Do you think that could be a first step toward your desired solution and would you be interested in implementing it?

[ngocdh]

Are you interested by testing Jack on Windows Jamulus version ?
With that you can connect what you want where you want.

@jujudusud how do I setup channels?

[ngocdh]

wow delay...

[ann0see]

@ngocdh could you please head over to e.g. #1712 to keep this thread focused. Hint: check Qjackctl's Graph button ;-)

[jujudusud]

Select your ASIO driver instead of portaudio.
After that stay at 128 frames and disable small buffers. Your delay should reduce drastically.
To set the connections, you have the graph or the patch au buttons.
For the delay, trust your ears and only your ears.

[jujudusud]

No news here ?

[emlynmac]

So after some research....

I can open up two jamulus clients on my multichannel car connected Mac and route the inputs through. It is not intuitive though, having to start the client from the command line. This does bode well though for being able to make my particular use case work in the short term....

I'll see about how this can be worked into the code more for a more user-friendly method.

[dakhubgit]

@ann0see only conceptually. The idea was that could call the server with something like --stereosplitchar \& and if then anybody named his identity Cindy&Bert, the server would split the stereo track it got into two mono mixer tracks. The idea was that this interface would not require an update to the clients (because getting everybody to update would be a real pain). There would be one issue with an unmodified client since the server can only declare one of the mixer strips it announces to the user as being the user themselves: if you click "Mute Myself", the Jamulus client is supposed to route your own input locally to your headphones, and obviously it would only locally route the one strip it considers your own, with its pan button reverting to a balance control. Apart from "Mute Myself" providing buggy local monitoring, that scheme should work by a server-side change only.

I currently have very little time working on Jamulus myself since my ensemble reverted to live rehearsals and I am working on hardware problems right now since a few of the virtual rehearsal advantages are now considered desirable live: "crank up the bass accordion" and "let's inject a drum part via your arranger thing" were easy with Jamulus and now require me to bring a second arranger keyboard in better working state and work on an old keyboard amp (the compression driver I used in there broke into pieces while rolling the washing machine sized amp across cobblestones).

[jujudusud]

We should provide a guide to explain:
How to use two different sound sources and mix it with panoramic pot.
This guide will have to explain that you have to set your inputs levels with hardware ajustement and that no Jamulus level fader is present. And also that it is just because you never wanted to clip your signal in the hardware preamp.
This guide should be in the KB.
@insolace : would you want to create this guide ?

I’m not sure I understand what you’re asking.

It is already possible to adjust the level of each channel at the Jamulus input using the pan control.
I believe that this setting, coupled with the level controls on the hardware, is sufficient to balance the two channels that are sent to the server.
I therefore propose to make a user guide and as you are the one requesting this feature I propose to make this guide.

[dakhubgit]

It is already possible to adjust the level of each channel at the Jamulus input using the pan control.
For Mono-in/Stereo-Out you cannot pan the channels of a "split-personality" source separately (left is fully-left, right is fully-right, pan control only affects their relative volumes), and even adjusting volumes separately is a pain if you want to raise just the level of one of them.

I believe that this setting, coupled with the level controls on the hardware, is sufficient to balance the two channels that are sent to the server.

Sure, the server gets sufficient information. The problem is that a stereo mixer strip does not provide sufficient user control over what to do with that information for a user with Stereo-out. In that case, you have a 2×2 matrix of gains from two signals to two outputs, and a stereo mixer strip only provides 2 controls (volume and balance) for filling that matrix. You'd need four (two mono strips with volume and pan, or volume, balance, pan, stereo width, but that paradigm stops working well when we want to add delay pan).

I therefore propose to make a user guide and as you are the one requesting this feature I propose to make this guide.

[insolace]

It is already possible to adjust the level of each channel at the Jamulus input using the pan control.
I believe that this setting, coupled with the level controls on the hardware, is sufficient to balance the two channels that are sent to the server.

But in this instance the signals are panned hard left and hard right. This is not the correct approach.

I therefore propose to make a user guide and as you are the one requesting this feature I propose to make this guide.

Now that I've had a chance to fork/review the codebase, I think my time would be better sent implementing the correct fix.

Sure, the server gets sufficient information. The problem is that a stereo mixer strip does not provide sufficient user control over what to do with that information for a user with Stereo-out. In that case, you have a 2×2 matrix of gains from two signals to two outputs, and a stereo mixer strip only provides 2 controls (volume and balance) for filling that matrix. You'd need four (two mono strips with volume and pan, or volume, balance, pan, stereo width, but that paradigm stops working well when we want to add delay pan).

A 2x2 matrix for a single stereo channel is what I am proposing, this is a common concept in dual mono mixing. It sounds more complicated than it is, and the code is already 75% of the way there. And more importantly, **this solution is much simpler and lower overhead than encouraging users to run dual clients, or having a single client pretend to be multiple clients **

Here is the relevant section of server.cpp:

There are already two separate data buffers for the left and right input channels (See lines 1215 and 1219). If a splitDualMono flag is set then we would mix vecsData[i] and vecsData2[i], and use two new floats (ie leftInputPanFloat and rightInputPanFloat) to pan those buffers as we add them to vecfIntermProcBuf[i].

Here is some example code to demonstrate the concept:

The updated user interface would look like this:

In this example "mTang-MS" and "benson" are both transmitting in stereo, while the other users are transmitting in mono. The stereo channels have a "L|R" button that when engaged enables the dual mono function, which creates two faders each with a pan. If the button is not engaged (default behavior), the channel behaves normally, and mono channels appear like they do now.

I am wrapping up some product launch deadlines at my company (we make hardware digital audio mixers and software editors that use Qt, so this is all very familiar territory), so I won't have time to dig into this and get it building for a few weeks. If someone who is already able to build the repo (@emlynmac perhaps?) wants to try this approach, I'd be happy to assist.

[emlynmac]

This looks like a great first step and I’d be happy to look.

[dcorson-ticino-com]

I have looked a little closer at this.

The ideal case would be to make a client that can open two client classes using the same hardware input. In that case it would not be necessary to change anything on the server and the multi-channel would work with all existing servers.

That would mean, however, disentangling the soundcard from the client so that two clients can use the same soundcard. That will be, IMHO, a major rewrite of the client and sound classes. We would need someone who is at ease with programming C++ and knows how to use its (supposed) advantages to do this. I couldn't, I feel like I get stuff done in spite of C++.

The solution to make the major changes on the server to interpret a stereo stream on the server as two mono streams and creating a supplementary user to support left and right channels separately, with only supporting changes on the client seems to me to be much simpler, but will need both the client and server to be the "new" version to work.

If no-one presents themselves as a candidate to make a multi-client version I will attack the second possibility with the major changes on the server in the next weeks.

[insolace]

Creating phantom clients and users is needlessly complicated. I've outlined the changes needed in the server, and while my 22 lines of code above are just a starting point, this is not a major change, and certainly much simpler than having the server create a second user/client/session.

If the goal is to get more than 2 inputs streaming then maybe another approach is warranted, but dual mono should be a very easy way to address musicians who have a mic and an instrument.

[dcorson-ticino-com]

Creating phantom clients and users is needlessly complicated.

The problem with your approach, which I like otherwise, is that it forces the server and all the clients on that server to update, even the clients that are not sourcing dual mono signals as they will have to be able to interpret them and display them.
Creating phantom clients means that only the server and the client sourcing the dual mono signal must be of the new version. Any old client can still interpret and display without a problem. I think this is a critical difference as I see a great hesitancy to update among the users I have contact with. Most are musicians and not very computer literate, but still don't want to be without Jamulus.

[insolace]

It sounds like this hesitation to do anything that would force someone to update is creating a situation where they don’t have to update, therefore they don’t.

Jamkazam pushes out updates almost weekly, and they have a large userbase of musicians that are not technical yet manage to update. What makes Jamulus different?

[ngocdh]

I think while KISS is certainly keeping code simple and clean, its mindset is quite conservative. That’s why people don’t want big changes, let alone breakthrough.
I might be wrong though, ‘cause big changes also means more work. After all we’re all volunteers not getting paid for our work here.

[ngocdh]

If we're only looking at the case where user already has two separate mono inputs, then one solution is source volume faders, which should be very easy to implement. No changes required for server or other users client.

In fact, I consider this solution as a variant of the input gain feature, only now it also allows to lower the volume of left and right channels separately.

It is not the best solution, but I'm sure it requires minimal code change, and still compatible with any other version.

However, if I'm not mistaken, this only applies to Windows users only :(

[insolace]

Source volume faders on the client side as they transmit to the server does not solve the problem of other users needing to have control of those sources in their own mix. A user should be able to control all of the source instruments discretely, and those user mixes are baked in server.cpp (see my screenshots above). A dual mono approach is the simplest solution that addresses the full problem.

[ngocdh]

I agree, and I think your approach is very doable and would help a lot of users.

The only problem imo is compatibility: client must first find out if the server supports dual mono faders. I still see servers running versions 3.4.x. But I'm sure it's a minor issue: users just have to choose a compatible server if they want the feature.

[insolace]

Newer clients will have to query the server version number, and if its incompatible then not show the new UI.

[dakhubgit]

DonC ***@***.***> writes:

I have looked a little closer at this. The ideal case would be to make a client that can open two client classes using the same hardware input.

I don't see the point in two clients running synchronised to the same sound card and the same server to compete for network and CPU resources in a random manner.

In that case it would not be necessary to change anything on the server and the multi-channel would work with all existing servers.

For splitting a stereo channel into two logical channels, this seems wasteful. Having separate packets for multiple channels also seems wasteful when the soundcard delivers all of them synchronously, and the possibility to end up with different delays and jitter buffer sizes for stuff running into the same soundcard on the same site also seems off-putting. I think it would be saner to invest the C++ programming into disentangling the "one mixer strip per client" requirement.

The solution to make the major changes on the server to interpret a stereo stream on the server as two mono streams and creating a supplementary user to support left and right channels separately, with only supporting changes on the client seems to me to be much simpler, but will need both the client and server to be the "new" version to work.

As I said: apart from correct "Mute myself" semantics (and the necessity to split a strip only audible to yourself into two seems not to be all that urgent), the client does not even need to know about this.

If no-one presents themselves as a candidate to make a multi-client version I will attack the second possibility with the major changes on the server in the next weeks.

That would be great. So far I've just invested time in trying to understand the setup of the server. So if you end up refactoring and commenting some code to make this task easier to you, don't hesitate to commit such changes for fear of being thought of as a wuss.

…

-- David Kastrup

[insolace]

I don’t see this as an advantage at all. The client already alerts the user that they are out of date, they should be updating and if anything the update process should be streamlined so that the end user doesn’t need to manually do anything other than click “ok”, and then the client does the download/install/update/restart process.

But in any case, newer servers and clients running the newer dual mono code should be backwards compatible with older servers/clients.

If an older client connects to a newer server, the server knows not to expect the dual mono control commands from that client.

If a newer client connects to an older server, it knows not to offer the dual mono controls to the user.

[dakhubgit]

I don't see the point in two clients running synchronised to the same sound card and the same server to compete for network and CPU resources in a random manner.

The point is that in that case only the client sourcing the dual mono signal needs to be updated. There is no change for either the server or the other clients. I think this is a major advantage. But it is well out of my competence (and patience) as a programmer.

Well, I think it turns out that I did not read carefully before replying: there was no talk about multiple clients, making that part of my response and your retort moot. I was replying to:

The ideal case would be to make a client that can open two client classes using the same hardware input.

"two client classes" would imply two instances of a data structure I think, not two processes. And if structured right, this might correspond naturally with the GUI functionality and the underlying internals.

[dcorson-ticino-com]

I don’t see this as an advantage at all. The client already alerts the user that they are out of date, they should be updating

This is what Microsoft says too and I don't know a single person that thinks it is good.
Outside of the software development bubble the average user equates updates with problems, unnecessary changes and time lost. None of the people I play with have ever updated their Jamulus, and many probably never will as they are running old Macs.
I don't think we have any visibility of the number of people running old clients, but we do have that info about the servers.
I have not done a statistic, but there are an awful lot of people running very old versions.

[insolace]

This is what Microsoft says too and I don't know a single person that thinks it is good.

I don’t understand, has development on Jamulus ceased?

This is circular logic. Users aren’t updating software, therefore they must not like updates, so we won’t write any, and look they aren’t updating so this must have been the right decision.

In any case this argument is straying from the issue at hand. I’ve already replied in this thread with a method where dual mono could be implemented in a way that maintains backwards compatibility, regardless if coddling end users at the cost of useability is a good strategy or not.

[dcorson-ticino-com]

The discussion is about what should be implemented.
That there are many ways of doing this is clear.
Personally I do not see disadvantages to implementing something that will also work for old clients, other than making the GUI having a dial mono channel may look better.

[ann0see]

DonC opened a PR: #1913

feel free to test ;-).

[kendoori]

Did anything come of this? I was able to get around this limitation in Linux by using Carla's patch bay, but I don't know how I would do the same in Windows...

[ann0see]

On windows use JACK. It should work quite similar https://jamulus.io/kb/2022/01/26/Routing-JACK-Windows.html

[pljones]

I use Reaper - it can handle a multi-input ASIO driver quite happily and has ReaRoute, which happily allows multiple instances of Jamulus for ASIO to run alongside each other.

(It also hosts my samplers - I only have a single audio in for a voice mic, plus MIDI input for my various drum kits.)

[dakhubgit]

ann0see ***@***.***> writes:

On windows use JACK. It should work quite similar https://jamulus.io/kb/2022/01/26/Routing-JACK-Windows.html

Of course the problem here is that you'll not end up with multiple channel strips for multiple instruments: it requires creating a submix entirely under control of one participant and outside of Jamulus.

…

-- David Kastrup