[FEATURE] Implement Pagination and Sorting for APIs

[Guhapriya01]

Is this feature already requested?

• I have checked "open" and "closed" issues, and this is not a duplicate.

Problem or Missing Functionality

When retrieving large datasets through the API, there is currently no pagination or sorting functionality. This results in inefficiencies when handling large amounts of data and makes it difficult to filter and organize results effectively. Adding pagination and sorting would improve the performance and user experience by allowing more control over how data is fetched and presented.

Feature Description

I propose adding pagination and sorting functionality to the existing APIs. By leveraging Spring Data’s Pageable interface, we can allow users to limit the number of results returned per request and sort the results based on different fields. This enhancement would streamline the process of accessing large datasets and improve overall efficiency.

Screenshots

No response

Would you like to work on this feature?

Yes

Implementation Plan

1. Adding pagination and sorting parameters to the existing API endpoints.
2. Modifying the service and repository layers to support pagination and sorting using Spring Data’s Pageable interface.

[ajaynegi45]

1. Adding pagination and sorting parameters to the existing

Hi @Guhapriya01,

Thank you for expressing your interest in working on the "Implement Pagination and Sorting for APIs" issue. I'm delighted to inform you that I have assigned this issue to you. Your willingness to contribute to our project is much appreciated. Follow SOLID Design Principles

Feel free to start working, and if you have any questions or need assistance during the process, please don't hesitate to reach out.

[Shetu003]

Hello Sir, Can you please assign this issue to me?

[ajaynegi45]

Hello Sir, Can you please assign this issue to me?

Hi @Shetu003,

Thank you for expressing your interest in working on the "Implement Pagination and Sorting for APIs" issue. However, this issue is currently being worked on by another contributor, Guhapriya01 . You can explore other issues in our repository or check out our other projects under the GirlScript Summer of Code and Hacktoberfest.

Here are the links to explore:
https://bento.me/ajaynegi

Feel free to reach out if you have any questions or need guidance on where to start. We're excited to have your contributions!

[Guhapriya01]

Hi @ajaynegi45,

As I work on implementing pagination and sorting for the APIs, I wanted to bring to your attention that we may need to update the existing API documentation to reflect these changes. Would you like me to draft an update for the documentation, or should we discuss how to best proceed?

Additionally, I have just started implementing this feature for the Books API (endpoint: GET /api/books) and wanted to share the current response structure I’m getting. The response is sorted by title by default, and it includes 5 elements per page:

{
    "content": [
        {
            "bookId": 11,
            "title": "Circe",
            "author": "Madeline Miller",
            "isbn": "978-0316388681",
            "publisher": "Little, Brown and Company",
            "publishedYear": 2018,
            "genre": "Fantasy",
            "copiesAvailable": 7
        },
        {
            "bookId": 10,
            "title": "Educated",
            "author": "Tara Westover",
            "isbn": "978-0399590504",
            "publisher": "Random House",
            "publishedYear": 2018,
            "genre": "Memoir",
            "copiesAvailable": 5
        },
        {
            "bookId": 13,
            "title": "The Alchemist",
            "author": "Paulo Coelho",
            "isbn": "978-0062315007",
            "publisher": "HarperOne",
            "publishedYear": 1993,
            "genre": "Adventure",
            "copiesAvailable": 10
        },
        {
            "bookId": 12,
            "title": "The Night Circus",
            "author": "Erin Morgenstern",
            "isbn": "978-0385534635",
            "publisher": "Doubleday",
            "publishedYear": 2011,
            "genre": "Fantasy",
            "copiesAvailable": 2
        },
        {
            "bookId": 7,
            "title": "The Silent Patient",
            "author": "Alex Michaelides",
            "isbn": "978-1250219653",
            "publisher": "Celadon Books",
            "publishedYear": 2019,
            "genre": "Psychological Thriller",
            "copiesAvailable": 4
        }
    ],
    "pageable": {
        "pageNumber": 0,
        "pageSize": 5,
        "sort": {
            "sorted": true,
            "unsorted": false,
            "empty": false
        },
        "offset": 0,
        "paged": true,
        "unpaged": false
    },
    "last": false,
    "totalPages": 2,
    "totalElements": 7,
    "first": true,
    "numberOfElements": 5,
    "size": 5,
    "number": 0,
    "sort": {
        "sorted": true,
        "unsorted": false,
        "empty": false
    },
    "empty": false
}

Please let me know if this response format aligns with your expectations or if there are any changes I should make.

Thank you!

[ajaynegi45]

Hi @ajaynegi45,

As I work on implementing pagination and sorting for the APIs, I wanted to bring to your attention that we may need to update the existing API documentation to reflect these changes. Would you like me to draft an update for the documentation, or should we discuss how to best proceed?

Additionally, I have just started implementing this feature for the Books API (endpoint: GET /api/books) and wanted to share the current response structure I’m getting. The response is sorted by title by default, and it includes 5 elements per page:

{
    "content": [
        {
            "bookId": 11,
            "title": "Circe",
            "author": "Madeline Miller",
            "isbn": "978-0316388681",
            "publisher": "Little, Brown and Company",
            "publishedYear": 2018,
            "genre": "Fantasy",
            "copiesAvailable": 7
        },
        {
            "bookId": 10,
            "title": "Educated",
            "author": "Tara Westover",
            "isbn": "978-0399590504",
            "publisher": "Random House",
            "publishedYear": 2018,
            "genre": "Memoir",
            "copiesAvailable": 5
        },
        {
            "bookId": 13,
            "title": "The Alchemist",
            "author": "Paulo Coelho",
            "isbn": "978-0062315007",
            "publisher": "HarperOne",
            "publishedYear": 1993,
            "genre": "Adventure",
            "copiesAvailable": 10
        },
        {
            "bookId": 12,
            "title": "The Night Circus",
            "author": "Erin Morgenstern",
            "isbn": "978-0385534635",
            "publisher": "Doubleday",
            "publishedYear": 2011,
            "genre": "Fantasy",
            "copiesAvailable": 2
        },
        {
            "bookId": 7,
            "title": "The Silent Patient",
            "author": "Alex Michaelides",
            "isbn": "978-1250219653",
            "publisher": "Celadon Books",
            "publishedYear": 2019,
            "genre": "Psychological Thriller",
            "copiesAvailable": 4
        }
    ],
    "pageable": {
        "pageNumber": 0,
        "pageSize": 5,
        "sort": {
            "sorted": true,
            "unsorted": false,
            "empty": false
        },
        "offset": 0,
        "paged": true,
        "unpaged": false
    },
    "last": false,
    "totalPages": 2,
    "totalElements": 7,
    "first": true,
    "numberOfElements": 5,
    "size": 5,
    "number": 0,
    "sort": {
        "sorted": true,
        "unsorted": false,
        "empty": false
    },
    "empty": false
}

Please let me know if this response format aligns with your expectations or if there are any changes I should make.

Thank you!

Hi @Guhapriya01,

Thank you for the update and for bringing up the need to revise the API documentation. It’s a good call—I think updating the documentation to include the new pagination and sorting behavior is essential. Feel free to go ahead and draft the changes, and we can review them together.

As for the response structure, it looks great! Sorting by title and limiting to 5 elements per page works as expected. The structure is clean, and the inclusion of pagination details like totalPages, totalElements, and sort is very helpful. No changes are needed at this point, so please continue with the implementation.

Let me know if you encounter any issues or need further input!

[Guhapriya01]

Thanks for the feedback! I’ll continue with the implementation of pagination and sorting and reach out if I need further input.

[Guhapriya01]

Hi @ajaynegi45 ,

I wanted to let you know that I have created a pull request to implement the pagination and sorting functionality for the APIs. Next, I will start drafting the documentation to outline the new pagination and sorting parameters.

If you have any feedback or suggestions, please feel free to share!

Thank you!

[ajaynegi45]

Hi @ajaynegi45 ,

I wanted to let you know that I have created a pull request to implement the pagination and sorting functionality for the APIs. Next, I will start drafting the documentation to outline the new pagination and sorting parameters.

If you have any feedback or suggestions, please feel free to share!

Thank you!

Hi @Guhapriya01,

Your changes are fantastic and work seamlessly with the project. I've merged them into the project. However, I have a question: When I hit the URL for the first time, I receive the first sorted set of 5 data items. But when I hit the URL again, I still get the same 5 data items. How can I fetch the next set of data after those 5?

[Guhapriya01]

Thank you for the feedback!

To fetch the next set of data, you can use the pagination parameters I added. Here’s how:

• page: This is the page number you want to retrieve, starting from 0. For example, page=0 gives you the first 5 items.
• size: This indicates how many items you want to fetch per page, which is set to 5 by default.

So, if you want to get the next 5 items after the first request, you would change the page number:

• First request: GET /api/books?page=0 (default is 0, so you don't need to specify it)
• Next request: GET /api/books?page=1

Let me know if you have any questions!

[ajaynegi45]

documentatio

Thank you for clearing up my doubt! Take your time to create a very detailed API documentation. Please include all possible parameters such as page, size, sort, title, sortBy, sortDir, and provide examples for each one individually as well as in combinations (e.g., /api/books?page=0&size=7&...). Also, make sure to highlight any relevant points that developers might need to know when working with the API.

[Guhapriya01]

Sure, @ajaynegi45! I’ll start drafting a comprehensive API documentation, as per your suggestions.

I’ll keep you updated on the progress!

[Guhapriya01]

Hi @ajaynegi45,

As part of adding pagination and sorting to the API endpoints, I've noticed that using an invalid field for sorting results in a PropertyReferenceException, leading to a 500 Internal Server Error. To enhance the user experience, I plan to handle this exception by catching it and rethrowing a custom exception (InvalidSortFieldException). This will allow us to return a 400 Bad Request response for invalid sort fields instead.

I’m currently drafting the documentation for this feature, and I believe this change will help users understand the error more clearly and guide them to provide valid sorting parameters.

I'll proceed with this implementation and update the documentation accordingly. Please let me know if you have any suggestions or concerns!

[ajaynegi45]

Hi @ajaynegi45,

As part of adding pagination and sorting to the API endpoints, I've noticed that using an invalid field for sorting results in a PropertyReferenceException, leading to a 500 Internal Server Error. To enhance the user experience, I plan to handle this exception by catching it and rethrowing a custom exception (InvalidSortFieldException). This will allow us to return a 400 Bad Request response for invalid sort fields instead.

I’m currently drafting the documentation for this feature, and I believe this change will help users understand the error more clearly and guide them to provide valid sorting parameters.

I'll proceed with this implementation and update the documentation accordingly. Please let me know if you have any suggestions or concerns!

Hey, thanks for pointing that out and for the plan to handle invalid sort fields! That sounds like a great improvement to the user experience. While you're at it, maybe we could also refactor other exceptions where possible to make sure they all throw meaningful exceptions and error codes. This way, we can maintain consistency across the API and make troubleshooting easier for users.

Let me know what you think!

[Guhapriya01]

Hey, thanks for pointing that out and for the plan to handle invalid sort fields! That sounds like a great improvement to the user experience. While you're at it, maybe we could also refactor other exceptions where possible to make sure they all throw meaningful exceptions and error codes. This way, we can maintain consistency across the API and make troubleshooting easier for users.

Let me know what you think!

Thanks for your feedback Ajay! I completely agree that refactoring other exceptions to provide meaningful messages and error codes will enhance the consistency of the API and improve the overall user experience. I've already handled the PropertyReferenceException to throw a meaningful error response (ready for PR), as I focus on drafting the documentation.

Here’s a sample error response:

Status Code: 400 BAD REQUEST

{
    "timestamp": "2024-10-03T07:37:08.364+00:00",
    "message": "The specified 'sortBy' value is invalid.",
    "details": "uri=/api/borrowings"
}

I wanted to check in on your thoughts regarding our approach. Should we focus on completing the documentation first, especially since it's crucial for new contributors to understand the endpoints with pagination and sorting? Once that's done, we can look into handling other exceptions for consistency.

Let me know if you have any other thoughts!

[Guhapriya01]

Hi @ajaynegi45 ,

I’ve completed the API documentation and submitted a PR (#59). Please take a look and let me know if it needs any changes to meet the project’s needs. Thank you!

[Guhapriya01]

Hey, thanks for pointing that out and for the plan to handle invalid sort fields! That sounds like a great improvement to the user experience. While you're at it, maybe we could also refactor other exceptions where possible to make sure they all throw meaningful exceptions and error codes. This way, we can maintain consistency across the API and make troubleshooting easier for users.

Let me know what you think!

Hi @ajaynegi45,

Thank you for merging the PR (#59)! I wanted to follow up on your previous comment regarding refactoring other exceptions to maintain consistency across the API.

Did you have any specific thoughts on that, or should I take the initiative to refactor those exceptions?

[ajaynegi45]

Hey, thanks for pointing that out and for the plan to handle invalid sort fields! That sounds like a great improvement to the user experience. While you're at it, maybe we could also refactor other exceptions where possible to make sure they all throw meaningful exceptions and error codes. This way, we can maintain consistency across the API and make troubleshooting easier for users.
Let me know what you think!

Hi @ajaynegi45,

Thank you for merging the PR (#59)! I wanted to follow up on your previous comment regarding refactoring other exceptions to maintain consistency across the API.

Did you have any specific thoughts on that, or should I take the initiative to refactor those exceptions?

Regarding the refactoring, feel free to take the initiative. You can also refactor the code structure to follow design principles like SOLID or any other to enhance consistency and maintainability across the API.

[Guhapriya01]

Thanks! I’ll refactor the exceptions and code structure where possible to improve consistency and maintainability. I’ll keep you updated on my progress!