Bug in error handling when feature layer query causes 500 error on server

[apulverizer]

Describe the bug
There was a change between 2.2.0.X and 2.4.X where the querying behavior no longer works if the initial query to get features returns a 500 error. There used to be logic which would set a the returnRecordCount and do pagination. This logic is broken in version 2.4.X.

To Reproduce
Steps to reproduce the behavior:

import arcgis

# This service must have a very high Max Record Count (e.g. 1 million)
# We have a valid use case for this super high Max Record count because some other applications do not paginate.
# This is just one way to trigger this issue
layer = "https://<some-layer-with-400k-features>/FeatureServer/0"

feature_layer = arcgis.features.FeatureLayer(url=layer)
count = feature_layer.query(return_count_only=True)
print(f"Expected Count: {count}")
results = feature_layer.query(out_sr=4326, as_df=True)
print(len(results['features']))

output:

Expected Count: 336731
1000

Only the first 1000 records are returned from the layer. It should be comparing the max_rec with the total count of the features not with the max_records which is 1000 by default.

The bug is in: arcgis\features\layer.py

def _retry_query_with_fewer_records(layer, url, params, raw):
    """Retries the query with a reduced result record count."""
    max_record = params.get("resultRecordCount", 1000)
    offset = params.get("resultOffset", 0)

    if max_record < 250:
        raise Exception("Max record count too low; query still failing.")

    result = None
    max_rec = (max_record + 1) // 2  # Halve the record count
    i = 0

    while max_rec * i < max_record:
        params["resultRecordCount"] = min(max_rec, max_record - max_rec * i)
        params["resultOffset"] = offset + max_rec * i

        try:
            records = _query(layer, url, params, raw=True)
            if result:
                result["features"].extend(records["features"])
            else:
                result = records
            i += 1
        except Exception as retry_exception:
            raise retry_exception

Screenshots
If applicable, add screenshots to help explain your problem.

Expected behavior
All features should be returned. This works in 2.2.0.4.

Platform (please complete the following information):

• OS: Windows 11
• Browser N/A
• Python API Version 2.4.0

Additional context
Add any other context about the problem here, attachments etc.

[nanaeaubry]

@apulverizer Thanks for catching that, I will make an update. In the meantime you could set the result record count equal to total number expected for it to not default to 1000