Intermittent 500 errors from Flickr API during initial fetch

[garrettc]

I've been getting intermittent 500 errors from the Flickr API when doing the initial fetch. There doesn't seem to be any pattern: I've had it happen after 30 seconds, and after around an hour, which makes me think it's unrelated to #148 (this is on an M1 Macbook Pro with 32GB RAM).

After a few attempts I've managed to get 3500 out of 6662 photos, but the issue is every time it bails out I have to start again from the beginning (if I'm understanding how fetching works correctly? It starts from most recent and works its way back?).

An example error:

garrettc: Failed to fetch Photos: Error when fetching extra data: 
Error when fetching photo info (photo 23217341195): 
do_request: Status code 500 received

Photo 23217341195 is this one, and if I query it using the Flickr API explorer I get a valid result, so it feels like this is at Flickr's end.

I'm wondering if in addition to the --days argument, we could introduce --start and --end arguments which took a date that could be mapped to the min and max_taken_date arguments of the flickr.photos.search method?

Something like:

$ ./manage.py fetch_flickr_photos --start=2001-01-17 --end=2003-11-10

That way, if the API does barf, you could check in the admin area how far back the fetch got, and then start it again but restrict the window. I'm happy to take a run at producing a PR for this if it feels like a valid approach.

You also mentioned another approach on #229 where the fetch tracks progress and continues from that point. Would that be simpler than introducing two new arguments?

[philgyford]

I think the simplest would be to set a start/end as you suggest. Which would hopefully work around an import breaking either at the API end or on my/your end due to memory error etc.

[garrettc]

Okay, cool. I think I should have some time this weekend to take a stab at this. Stay tuned.

[garrettc]

I posted a draft PR (#232) a couple of days ago, and I've just realised that drafts don't notifiy anyone. It's definitely not ready because bits of it don't work, but it would be good to get some feedback.