Run downloads in parallel #32
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
requested a review
from 
ghost
self-assigned this
|
Can we set this up on a dev server, and do a check whether more downloads are blocked compared to live? It would also be good to know whether anything's missing from https://iati-data-dump.codeforiati.org/ for probably this reason. |
I feel like I checked this when switching to parallel downloads on iati-data-dump, and it wasn’t the case (though I’d be interested to hear the results of your experiments!) Also, this parallel downloading was copied from d-portal (as mentioned in the acknowledgements), so I guess they’d also have similar problems. |
|
Thanks both. I've added --retry-connrefused and put this on dev servers. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.