Assistance with pagination limit

[hyryuki]

Hello, I am currently working with the API (GET/{branch}/concepts endpoint) for data analysis, and I have encountered a limitation regarding pagination. Specially, I am unable to retrieve all the data due to the pagination limits enforced on the endpoint. When using the unlimited pagination (searchAfter) I get the following error:
"Unlimited pagination of the full concept representation is disabled in the deployment"
Given that I need to analyze all available data, this limitation poses a significant challenge.

Questions:

1. is there any possibility to enable unlimited pagination for my deployment or is this a global restriction for all users?
2. If unlimited pagination cannot be enabled, do you have any alternative suggestions or best practices for retrieving all the data efficiently? For example, are there specific parameters or methods I could use to maximize the amount of data retrieved per request?

Thank you for your help, and I look forward to your response!

[kaicode]

Unlimited concept pagination can be enabled for your deployment using this configuration option:

--snowstorm.rest-api.allowUnlimitedConceptPagination=true

It's set to false in the default configuration: https://github.com/IHTSDO/snowstorm/blob/10.4.2/src/main/resources/application.properties#L24