Reduce memory usage when importing all Flickr photos #148
Labels
Comments
philgyford
added a commit
that referenced
this issue
Jul 13, 2016
Rather than fetching all the data and then saving the Photos, save them after fetching a single page (500 photos). An attempt to reduce the memory usage, but I don't think it achieved anything. Ref #148
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Running
./manage.py fetch_flickr_photos --days=allon a 512MB Digital Ocean box ended up with the process being killed after importing about 1500 photos. Works fine on my MacBook.Running
ps aux | grep 'python'while it runs shows the RSS usage continuing to grow as the script runs (from about 46,000 to 87,000 on my MacBook).I rejigged
ditto/flickr/fetch/fetchers.pyto save photo data after fetching a page of photos (500 photos), rather than storing all the data up and saving it all at the end, but that doesn't seem to change much.I tried poking around in
ditto/flickr/fetch/fetchers.pywith a couple of memory profiling things, but couldn't see anything that was growing, that didn't reset after starting a subsequent page of photos.I'm stumped, so leaving this for now.
The text was updated successfully, but these errors were encountered: