Save and resume offset into massive gallery, or specify it

[tcmbackwards]

I'm trying to download a deviantart gallery with over 80,000 items in it, but the API is aggressively rate limited and allows only 24 results per call with a delay of about 5 to 10 seconds necessary to avoid tripping the limiter. That's over 3300 calls just to get the URLs.

In addition, the limiter seems to detect if you're skipping downloads and making nothing but API calls and it puts you on 429 status more frequently, even if your API sleep value is more than your download sleep value.

That means that if I download with skip enabled for a day and get halfway through the gallery but then have to stop the download for some reason, I have to wait MORE than another day to get back to where I was in the offset value due to the increased chance I get put in 429 jail.

It would be helpful to be able to specify the offset into the gallery or to have the extractor save the offset value of the last API call it made for the gallery.

[tcmbackwards]

FYI image-range does not seem to help here, if I specify -o image-range="500-" the extractor still grinds through the gallery API counting files until it reaches number 500.

[mikf]

The first request starts at an offset of 499:

$ gallery-dl -v -o image-range="500-" https://www.deviantart.com/shimoda7
...
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=shimoda7&offset=499&limit=24&mature_content=true HTTP/1.1" 200 70

[tcmbackwards]

Here's what I was doing and you can see offset=0, is it not possible to give two arguments to -o?

PS G:\gallery-dl> .\gallery-dl.exe -v -o skip=true,image-range="590-" https://www.deviantart.com/shimoda7
[gallery-dl][debug] Version 1.27.3 - Executable (stable/windows)
[gallery-dl][debug] Python 3.8.10 - Windows-10-10.0.19045
[gallery-dl][debug] requests 2.31.0 - urllib3 2.1.0
[gallery-dl][debug] Configuration Files ['G:\\gallery-dl\\gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/shimoda7'
[deviantart][debug] Using DeviantartUserExtractor for 'https://www.deviantart.com/shimoda7'
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/shimoda7/gallery'
[deviantart][debug] Using custom API credentials (client-id 99999)
[deviantart][debug] Sleeping 4.00 seconds (api)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/shimoda7 HTTP/1.1" 200 401
[deviantart][debug] Sleeping 4.00 seconds (api)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=shimoda7&offset=0&limit=24&mature_content=true HTTP/1.1" 200 None
[deviantart][debug] Sleeping 4.00 seconds (api)

[mikf]

is it not possible to give two arguments to -o?

You set multiple options with -o by using it multiple times, once for each option:

gallery-dl.exe -v -o skip=true -o image-range="590-" https://www.deviantart.com/shimoda7

-o skip=true,image-range="590-"

This sets skip to "true,image-range=590-" and nothing else.

---

skip=true is the default. You can use --range instead of -o image-range=

[mikf]

but the API is aggressively rate limited

https://gdl-org.github.io/docs/configuration.html#extractor-deviantart-client-id-client-secret

[tcmbackwards]

Yes I'm using API credentials, if I typo something in the config file and it uses a public access token I get 429 pretty much instantly instead of being able to sleep 4 seconds per call.