Lambda response error with large requests

[zacdezgeo]

Describe the bug

When making a request to the summary endpoint with 8 fields or more, the API returns an internal server error.

To Reproduce

num_fields = 8
fields_to_test = available_fields[:num_fields]

try:
    benchmark(fields_to_test, delay_before_request=0)  # Warm start
except Exception as e:
    print(f"Error with {num_fields} fields: {fields_to_test}")
    print(f"Error message: {str(e)}")

Returns:

Error with 8 fields: ['sum_pop_2020', 'ogc_fid', 'sum_pop_f_0_2020', 'sum_pop_f_10_2020', 'sum_pop_f_15_2020', 'sum_pop_f_1_2020', 'sum_pop_f_20_2020', 'sum_pop_f_25_2020']
Error message: Failed to get summary: {"message":"Internal Server Error"}

After digging into the Lambda logs, we are hitting hard Lambda limits around the response size of 6 MB.

Expected behavior

I'm unsure how often actual users will requests more than 8 fields. That being said, we should have a clear path to support these use cases either by communicating these limits when they occur and proposing an alternative approach, or we adapt something in the API to return the expected response directly.

[zacdezgeo]

@bpstewar @andresfchamorro; opened a lot of related issues here, but here's a summary of the solution I am proposing:

• Defining user constraints on the acceptable upper bounds of data that can be requested (Define user constraints for number of fields and AOI size #37)
• Using compression middleware to return more data through the API (Use compression middleware #36)
• Provide an example using geoparquet directly for larger data volumes (Provide example of accessing raw data directly for large volumes #38)

We could also dig into some other options afterward such as:

• pagination of the response: might not be feasible given a user will likely want the entire data requested
• aggregating data to lower resolution hexagon levels

[zacdezgeo]

In #32, @vincentsarago added compression, and the difference is notable. We deployed a new API version, using the Kenya AOI, and requesting a point geometry, a user can retrieve 28 fields before hitting the size limitation. Without the point geometry, a user can retrieve up to 32 fields.

The same considerations above still apply, but we might already be within the acceptable constraints for user requests. cc: @andresfchamorro @bpstewar

[alukach]

Closed by #45