API endpoint /api/action/organization_list frequently times out

[simon-20]

Brief Description
The API endpoint /api/action/organization_list can be called with various parameters, or none.

As parameters are specified to include more information, the request time takes longer and longer.

With all_fields=true it takes around ~20 seconds.

With all_fields=true&include_extras=true it routinely takes 30+ seconds.

If you try to fetch all the available data, e.g., with:

http "https://iatiregistry.org/api/action/organization_list?all_fields=true&include_extras=true&include_tags=true&include_groups=true&include_users=true"

the request very often times out:

504 Gateway Time-out
nginx/1.24.0

Severity
Medium

Issue Location

https://iatiregistry.org/api/action/organization_list?all_fields=true&include_extras=true&include_tags=true&include_groups=true&include_users=true

Steps to Reproduce

Try to fetch the list of organisations with all the data possible.

[siwhitehouse]

Thanks @simon-20 - I was able to reproduce the 504 error immediately.

@robredpath I think this is another issue for the future, unless you think it should have a high priority.

[robredpath]

@simon-20 if I use paging (using limit and offset) then I get responses quite quickly - is that a suitable workaround for your current work?

[jarofgreen]

With all_fields=true it takes around ~20 seconds.
With all_fields=true&include_extras=true it routinely takes 30+ seconds.

Now they both regularly time out with a 504 error, both after 1 minute. Might this be related to today's go live?

This makes this issue more important.

[siwhitehouse]

@cormachallinanderilinx please see @jarofgreen 's comment above. I could recreate this by entering

https://iatiregistry.org/api/action/organization_list?all_fields=true

in the address bar of my browser. After a wait, I received a 504 Bad Gateway error. I've moved this to the top of "In Progress" on the project board. Please could you look into it as a matter of prioirty.

[simon-20]

@jarofgreen @siwhitehouse @cormachallinanderilinx

I think something has definitely changed with the release to the live Registry site today. The Bulk Data Service has been using the same Registry API URL for two weeks now, and up until today we hadn't had one 504, but since 2025-01-29T14:00:00Z every request made to that URL return 504.

I can get a response if I remove all the extra parameter flags, but in that case all that is returned is a bare list of publishers.

Even with just the single parameter, all_fields=true (which doesn't actually return all fields, of course, since it doesn't include the extras), the request times out.

[siwhitehouse]

@cormachallinanderilinx Do you have an update, please? This is the second time that an update to the Admin UI has - seemingly - had a negative impact on the API.

[cormachallinanderilinx]

The reason for this is because there are over 2 thousand publishers being returned and with extra fields, this makes it a huge returning payload.
We would not recommend using the organization_list with include fields, without using paging as Rob mentioned above
Typically we would limit the number of organizations being returned.

The reason it was timing out from last week is because with the changes to the publisher search meant we were returning new/additional fields on each publisher/org which caused the returning payload to increase in size.
Therefore, in the next release I will ensure that the API calls will call the current code and on the UI will call the new code.
However this may cause issues in the future as the number of publishers grow and I would strong suggest using paging.

Example the current response is 4.3MB which is a relatively response and will continue to grow as publishers are added