major revision of SSL4EO downloading script

[cmalbrec]

according to #2345

[cmalbrec]

@microsoft-github-policy-service agree

[adamjstewart]

This is more than just a refactor, this is a complete rewrite. Is this new script compatible with the command-line args used in experiments/ssl4eo/landsat/download_*.sh? I noticed you also removed the rate throttle, does this not suffer from exceeding the GEE API rate limit?

To properly review this I would probably need:

• A list of changes made from the current version
• Why these changes were made
• Performance metrics before and after this change

It may be quicker and easier to simply fix the geolocation bug, I can't imagine it requires more than a few lines changed.

[cmalbrec]

Hi @adamjstewart :

Yes, with @AABNassim we took the liberty to rewrite the download script to better organize for future development compared to the previous one. https://github.com/cmalbrec/torchgeo/blob/issue-2345/experiments/ssl4eo/download_ssl4eo.sh serves as an example on how to adapt experiments/ssl4eo/landsat/download_*.sh.

We successfully utilized the new downloading script to swiftly curate https://huggingface.co/datasets/embed2scale/SSL4EO-S12-v1.1 . I vote to move forward with the new version and add mods like rate throttling (@wangyi111 may help here), etc.

[adamjstewart]

Is your version based on the SSL4EO-S12 or SSL4EO-L download script? The latter underwent months of work to make it more generic to support both Sentinel and Landsat, with better overlap checking, nodata checking, and optimized downloading. I wouldn't want to see those months of work undone.

If you want to move forward with this version, I'll still need to see:

• A list of changes made from the current version
• Why these changes were made
• Performance metrics before and after this change

P.S. I've had about a dozen people ask me when SSL4EO-S12 v1.1 will be added to TorchGeo so it is easier to download.