[Feature Dev] Waveform progress bar

[FrenchGithubUser]

Following this feature request : https://features.jellyfin.org/posts/432/waveform-for-audio#

I'd like to develop this feature, here is how I thought I'd do it :

The goal would be to have an API endpoint to retrieve music data, and the client will draw the waveform bar.

Create a function in AudioController (entry), use the ffmpeg command to generate music data.

Let me know if there are specific things I should take into account, specific parameters, or things that might go wrong.

[gnattu]

So this is basically a "trickplay but for music" to me. My main concern is that the output of ffmpeg is relatively hard to use. Can you be more specific on how you want to implement this, like where you want to store those data and how you want it it be passed back to the clients?

[FrenchGithubUser]

Yea that sounds like a better idea indeed

[Chaphasilor]

Finamp dev here. I'd be in favor of using some sort of standard if available, like audiowaveform, instead of cooking up something new. The data should be fetchable similar to trickplay or lyrics, the extraction should also be similar to trickplay or LUFS (volume gain normalization).
There are existing rendering libraries for standard formats which makes them much easier to use compared to a custom format.

[gnattu]

Is it really the case that audiowaveform already has exisiting rendering libraries(other than the original provided by BBC) for integration? I have seen none.

[Chaphasilor]

Well there's https://github.com/bbc/waveform-data.js and https://github.com/bbc/peaks.js for JavaScript, and https://pub.dev/packages/just_waveform for Flutter. I didn't find anything for Android or iOS so far, but all the other solutions don't seem to share a standard either. At least in the case of audiowaveform, the data format is clearly documented.

[gnattu]

That is what I meant for "hard for integration" even for web. I'm not even sure if their reference implementation would ever work on smart TVs which is also a supported target of our official client, and for a huge possibility it won't as those TVs have very ancient chromium fork as web engine.

And the tools for generating those waveforms provided by BBC also does not fit our needs. This means even if we choose to stick to the BBC provided spec, we are still going to write our own implementations for both the server and the client.

[ferferga]

I recall that there were already some clients doing this without server-side support?

[FrenchGithubUser]

I'm not aware of any client that does it, but I might have missed one. Moreover, to display a waveform bar without server-side support, the music has to be downloaded first. So for local files that's fine, but when streaming, only the already-downloaded part could have its waveform bar displayed. And being able to seek to a part of the music that has not yet been downloaded, thanks to the waveform bar is a nice feature

[FrenchGithubUser]

It seems like there is a pcm file format : https://wiki.multimedia.cx/index.php/PCM

PCM stands for pulse code modulation. In the context of audio coding PCM encodes an audio waveform in the time domain as a series of amplitudes.

Which is exactly what is needed to draw a waveform (a list of amplitudes)

Would this be a good data format to return then ?

[gnattu]

This would simply mean you decompress all audio files into raw wav which is significantly larger than the original file and IMO is an overkill for this use case

[FrenchGithubUser]

no, there is no information about frequencies. I just tried with a 12.6MB mp3 file, which resulted in a 647KB pcm file. And adjusting the sample rate can make it even smaller.

Otherwise, what would be a better approach ?

[gnattu]

Can you elaborate more on your intended approach like what the config you did exactly to get that "647kb" file? You will still need a quite high sample rate to get a correct waveform or the waveform will look very off compared with the original, and from your file size if my assumptions are correct, you are just forcing a very low sample rate with the pcm audio encoder which already changed the waveform graph too much to look like the original.

[gnattu]

And 647K PCM file is still too big and too overkill, even if the pcm encoder really makes sense. It is not better than just generate a mono color PNG of fixed size and let the client to resize it. A mono color PNG for a waveform could only take around 3KB each file and 647KB is more than 200 tracks.

[FrenchGithubUser]

here is the command I used : ffmpeg -y -i music.mp3 -acodec pcm_s16le -f s16le -ac 1 -ar 1000 test.pcm

and yes using ffmpeg -i music.mp3 -filter_complex "aformat=channel_layouts=mono,showwavespic=s=640x120" -frames:v 1 output.png generates a 2.3KB png file, which is quite lighter. I will investigate if a png file is reusable well enough

Thanks for the feedback !

[FrenchGithubUser]

PR opened here : jellyfin/jellyfin#13486