[API Request] Add pagination (size, offset) parameters to additional API methods

[gravelld]

Type of change

API tweak

Proposal description

I propose that size and offset request parameters be added to more API methods which are capable of returning arbitrarily-sized collections.

Music libraries may be very large. Some methods currently have no paging support and may return thousands of large records, causing a performance impact and memory demand on servers and clients.

If size and offset are available, the returned data structures may be smaller, more manageable and delivered at lower latency.

Backward compatibility impact

No impact - the new parameters would be optional.

Backward compatibility

• No backward compatibility impact.

API details

Each method gets two new optional parameters:

• size - The number of elements to return. Defaults to no limit (for backwards compatibility).
• offset - The zero-based offset to start from in the list. Defaults to zero.

My suggestion is that these parameters should be added to (in alphabetical order):

• getArtists
• getBookmarks
• getCaptions
• getChatMessages
• getInternetRadioStations
• getGenres
• getPlaylists
• getPodcasts
• getShares
• getUsers
• getVideos

Optional

There are some methods which perform the same feature but with the size parameter named count, for example getTopSongs. For these, we could add size as a synonym for count and look to remove count long term, deprecating it for now.

Security impacts

No response

Potential issues

No response

Alternative solutions

No response

[Tolriq]

As discussed before we first need to think about a proper pagination API and new proper endpoints to support that and proper filtering and sorts.

The main issue with raw size and offset values is that they are very hard to handle server side for sorts like random. You need a seed value and somehow keep it for the next queries, but how long and all the side effects if wrongly handled.

[Tolriq]

I posted below, but usually this is done via a continuation token, this also solve performance issues with the counts as server can cache the data.

But pagination is not the hardest, the main issue is the sorting/filtering on the new endpoints, there's requests parsts everywhere that we need to consolidate, to see what is needed to build (Mainly how deep we go in filtering / sorting to build the corresponding API)
Once this is done selecting the pagination details is a small final step.

[jeffvli]

Can we re-gauge the interest from server devs to implement a full new pagination and sort/filtering scheme versus implementing this proposal, #102, and #18?

From your comment in #18

From a few discussion with them it's also mostly to keep backward compatibility and not having to directly support a huge new API, but being able to add (or skip) new endpoints on a time / interest basis.

From my perspective, it seems that most server devs are on board with "incremental updates" as @paulijar put it. But collaborating on a new sorting/filtering spec and having it be adopted by all the servers/clients is giving me doubts since we're still stalled on lower hanging fruit like the auth key proposal #99.

[Tolriq]

Auth is a the most sensible subject due to it's nature, complexity and different understanding by all parties.

Filtering and sorting are basic things that clients wants to offer there's no security impacts.

The #18 is a new endpoint, why create a new endpoint that keeps all the current limitations of bad sorting and filtering and pagination, specially with a supported random sort.

Listing all the songs without all the new values filterable, limited sorting and no filtering, and not bringing those to search3 vastly limits what a client can do with that endpoint.

A proper endpoint that support filtering (and so searching), sorting and pagination offers everything a client needs. And is not complicated to do, the only hard part is to commonly decide where to stop on what is supported. Once it's decided designing the API and implementing it server side and client side is easy.

In the end it could just be a getAllXXX with well defined filtering and params, sorting (and field selection as a bonus) and limits send as a json post. The api returns a continuation token to request the rest of the data and end of story. We do that for the different data types and returns something consistent (so no indexed artists ...) and we have something usable for every use cases.

For the record Symfonium have a copy of the data and can do everything it wants, and it's nice but no other client can do half of what I do because of the ultra limited API. And personally I think it would be cool that desktop or iOS clients could do what Symfonium does without having to resort to what Symfonium does.

[jeffvli]

Don't get me wrong, I fully support the addition of improved filter/sorts and pagination (I was pushing hard for the v2 API). It's less about the complexity but more-so the buy-in from everyone else and the timeline needed to accomplish it. I rewrote my client app Feishin with the expectation that there were certain OpenSubsonic features that would be implemented, but 18 months later, my desired proposals have stalled indefinitely.

This is why I asked if you could be the one to begin the proposal process for this new functionality, since you have a good idea for implementation. If you can list your thoughts on the proposal, everyone else can have a good starting point for discussion for the new endpoints. Otherwise I fear it may stall and be buried in just discussions again.

[Tolriq]

Adding a new getsongs endpoint is the same amount of work nearly as adding something else. (Specially with servers like Navidrome that already embed a smart playlist engine)

About proposal, I'm as most of you limited by the number of hours in the days, and do everything client side so the list of requirements for the sorts / filtering should come from other clients that know up to what point they can handle those. (How many sorts, how many filters at the same time, multiple values for the same filter (like songs for genre X and Y), that sort of things).

Building the proposal and API once the needs are fully explained I can help as I currently deal with all major music media providers + cloud + my own internal engine so know most of the pitfalls.

So if you have time open a new post for the sorting / filtering, there's a few amount of data in a few post that should be consolidated as a starting point, then add all you can support, I can then add a few more suggestion if needed from what my users asked me to support client side so that it can be done by others, and we have a good bases.
Maybe @deluan could jump later on that post to describe what he can handle with his smartplaylists to see if there's idea that should be possible without his specific implementation.

I also still need to find time to move on the transcoding API part, I hate that I can't cast a large part of my music to some Sonos devices that does not support the bitDepth and bitRate and just need a small resample.

[paulijar]

As discussed before we first need to think about a proper pagination API and new proper endpoints to support that and proper filtering and sorts.

The main issue with raw size and offset values is that they are very hard to handle server side for sorts like random. You need a seed value and somehow keep it for the next queries, but how long and all the side effects if wrongly handled.

But then again, the original Subsonic API already contains pagination on the endpoints getAlbumList and getAlbumList2 which may be requested with type=random. Hence, all compliant servers are already handling this difficult case one way or another (no doubt, some may do it poorly).

Of course it's a possible solution to completely redesign the thing but this would be a lot more work than just adding some parameters. And if the amount of work needed means, that it will actually never happen, then the "best" may be the enemy of "good enough".

[Tolriq]

The goal of OS is not to quickly add hacks that may or may not work depending on the server choices.
The fact that some server may work and some others don't is the reason of OS. So extending on something that is broken will never be the solution.

This is not about good enough, this is about repeatable, predictable and consistent for all the servers and the clients.
This is not the case with offset and size with some sorts.

Furthermore there's needs to add a lot more filtering and sorting to most of those endpoints as requested in many other posts.
So that would also just be lost time on something temporary.

[paulijar]

I wouldn't call what was suggested in the OP as "quick hack", more like an incremental update which solves some problems without solving all the problems of the world. As such everything suggested in the OP is "repeatable, predictable and consistent" but if the random mode would be added to some of these endpoints, then that would of course extend the existing difficulties to these endpoints, too.

Do you have some ideas, how the API could be designed so that the pagination of the random content would not be a problem? Or were you just thinking that it should be clearly documented, how this combination is supposed to work?

For reference, my server-side implementation for this combo in getAlbumList works like this:

• The first time the endpoint is called, a shuffled list of indices is created and stored
• When some page is requested, the server takes the relevant slice with offset and size from that list of indices, and picks the albums matching those indices.
• Whenever the page 0 is requested, the list is reshuffled and stored
• If the set of albums on the server has changed since the previous request, the list is reshuffled.

This may not be perfect but IMO works rather well on all those clients which I have tested.

[lachlan-00]

one option is supplying identification of the result list which a client can call back to.

Ampache stores the results of a browse in a table with the objects based on session which has an id that can be recalled, I'm also hashing the total results in the output to an md5 which i could also store with that table

Could do something with the hash / browse id is able to be retrieved for future calls.

I would think a new endpoint of something like getResults where you can call the id and get objects based on offset/limit/count

[Tolriq]

The fact that we extend on an API that does not work the same for all servers for some cases makes the API not correct from OS point of view.
While it may work for the new end points that currently do not have random sorts, this does not solve anything and just delay the creation of the proper endpoint that are waited by many.

There's way more people wanting proper sorts and filtering on the new data than the need for pagination in most of those endpoints.

Do you have some ideas, how the API could be designed so that the pagination of the random content would not be a problem?

This is not just about random this is also about server side enhancement of performance, typically saving the counts when filtering, see @epoupon comment about returning the count in the related PR.

What most cloud servers do is to work with a continuation token to be passed in next queries. (Working mostly like @lachlan-00 says) This ensure that the data is tied to a specific set of queries and so work properly multi user, multi devices, multi sessions.

The first time the endpoint is called, a shuffled list of indices is created and stored
When some page is requested, the server takes the relevant slice with offset and size from that list of indices, and picks the albums matching those indices.
Whenever the page 0 is requested, the list is reshuffled and stored
If the set of albums on the server has changed since the previous request, the list is reshuffled.

There's currently no session ID or cookies in the API so there's no way to ensure that an user is identified by device and session if connected multiple times. So by definition this is an hack, it works most of the time.