Proposed design for a paginated version of the get-group-members API

[seanh]

The current get-group-members API just returns a list of membership objects as a top-level JSON array:

[
  {...},
  {...},
  {...}
]

Following the advice about pagination in JSON:API (see Pagination in the JSON:API spec) I propose to change the response format to this:

{
  "meta": {
    "page": {
      "total": 151
    }
  },
  "data": [
    {...},
    {...},
    {...}
  ]
}

The js-config currently contains this for the get-group-members API:

{
  ...,
  "api": {
    ...,
    "readGroupMembers": {
      "headers": {
        "X-CSRF-Token": "***"
      },
      "method": "GET",
      "url": "https://hypothes.is/api/groups/xyz123/members"
    }
  }
}

I propose to add the new page[offset] and page[limit] query params to the js-config, like this:

"readGroupMembers": {
  "headers": {
    "X-CSRF-Token": "***"
  },
  "method": "GET",
  "url": "https://hypothes.is/api/groups/xyz123/members",
  "params": {
    "page[offset]": "{:offset}",
    "page[limit]": "{:limit}"
}

It'll be up to the frontend to replace {:offset} and {:limit} in the query param values with the desired offset and limit. The syntax comes from the URL Pattern API which we already use in some URLs in js-config. It's not necessarily clear that this format was intended for query param values but it should work.

Notes:

• This would be a breaking change to the API so we'll use the presence of the page[offset] query param to trigger the new paginated version of the API. Without this query param the endpoint will return the legacy, non-paginated response.

  The frontend will have to use page[offset]=0 to get the first page (and not the legacy non-paginated response). In future we intend to retire the legacy version of the API, when we do that page[offset] can become optional and will default to 0.

  The page[limit] param will be optional. We'll pick default and maximum values for it, perhaps based on some testing of production response times (which'll require creating a test group with a lot of members in production).

• meta.page.total is the total number of members in the group, which the frontend needs in order to be able to conveniently generate individual page links (see below) as well as to display this number in the UI. There's nothing in the JSON:API spec itself about providing the total length when returning paginated responses, this design is taken from the Collection Sizes section of the "Cursor Pagination" Profile to JSON:API.

  Note that this is the total number of items, not pages.

  There's also an example of pagination on the JSON:API site that uses meta.totalPages for the total number of pages (as opposed to items). The example also has this to say:

  Note: Putting a property like "totalPages" in "meta" can be a convenient way to indicate to clients the total number of pages in a collection (as opposed to the "last" link, which simply gives the URI of the last page). However, all "meta" values are implementation-specific, so you can call this member whatever you like ("total", "count", etc.) or not use it at all.

• Our UI designs for the paginated version of the page have a pagination control that looks like this:

  < [1] 2 3 4 5 6 7 8 >
  

  It's unclear in this design whether < and > are prev and next page links or first and last.

  It's not shown in the designs but there'll presumably have to be some eliding of pages if there are too many of them.

  The designs also contain a control for changing the page size.

  The backend could also return links for each of the separate pages 1..8 and for prev/next/first/last pages. JSON:API allows this. But I think the frontend has all the information it needs (including the js-config and response meta object) to generate these URLs itself and maybe it's better if the frontend does this.

• There are different ways to do pagination query params. For example you could do page[n]=n and page[size]=50. Or page[after]. I've gone with an offset and limit design that will translate directly to OFFSET and LIMIT in an SQL query. But I haven't given much thought to the pros and cons of different approaches here.

[robertknight]

It's unclear in this design whether < and > are prev and next page links or first and last.

I assumed they are prev/next links, since that is how the links in these positions function on https://hypothes.is/search, google.com and a bunch of other sites.

It's not shown in the designs but there'll presumably have to be some eliding of pages if there are too many of them.

Yes.

The backend could also return links for each of the separate pages 1..8 and for prev/next/first/last pages. JSON:API allows this. But I think the frontend has all the information it needs (including the js-config and response meta object) to generate these URLs itself and maybe it's better if the frontend does this.

To summarize my comments from a call just now, I am wondering whether we should use offset/limit vs cursor-based pagination, or a hybrid (eg. caller specifies a cursor as a start point, then gets back a page of results plus links for the N pages after that). The current UI mockups all show groups with up to ~20 pages maximum. For this specific use case most groups do conform to that, but we do have a very small number of groups with thousands of members.

For this use case we likely could get away with using an offset even with groups that have thousands of members, but there will be some overhead in a large group. When we come to re-implement activity pages however and support navigating through annotations or documents in groups that have millions of entries, we would have to either use a cursor or limit the number of pages the user can navigate through.

For the use case of API users specifically iterating over all pages in a response, I think it would be a good idea to include an explicit next link which they can follow. Even if they could easily increment a page number parameter, fetching a URL provided in the response is still easier. We also have the freedom to make that URL efficient to fetch.

[seanh]

I am wondering whether we should use offset/limit vs cursor-based pagination, or a hybrid (eg. caller specifies a cursor as a start point, then gets back a page of results plus links for the N pages after that). The current UI mockups all show groups with up to ~20 pages maximum.

I'm not sure how this would work. To return a page of results plus offset-based links for the next few pages I think the backend would have to fetch the next few pages worth of results from the DB in order to know what the cursor values for those next page links should be, but if it has already fetched them from the DB the backend may as well return those next few pages worth of results to the frontend now, there seems no point in throwing them away and just returning page links (although I guess it could do that).

I'm also not sure how the backend would calculate the previous few pages? Since I think the UI would want links to those as well.

I'm also not sure how the backend would know the number of the current page and therefore the numbers of the next and previous pages.

I think with cursor-based pagination we may be stuck with a UI without the current page number and without direct links to any pages by number, just next, prev, first and last links.

I don't think you could do this sort of thing with cursor-based pagination:

First ... 15 16 17 [18] 19 20 21 ... Last

I think you'd be limited to a control more like this:

First Prev Next Last

[seanh]

Slack thread