Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download ECCO files using Downloads and .netrc files #281

Merged
merged 33 commits into from
Dec 10, 2024
Merged
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
81d93cc
this should work
simone-silvestri Nov 28, 2024
867da9c
better naming
simone-silvestri Nov 28, 2024
71f2b19
only one download
simone-silvestri Nov 28, 2024
b36f60f
add download test
simone-silvestri Nov 28, 2024
31fb896
joinpath does not work on windows
simone-silvestri Nov 28, 2024
5b229db
test also downloading the bathymetry
simone-silvestri Nov 28, 2024
3ee4eaa
test dowloading bathymetry
simone-silvestri Nov 28, 2024
cba1991
restore tests
simone-silvestri Nov 28, 2024
51aff06
gracefull downloading
simone-silvestri Nov 28, 2024
d22edc9
try it now
simone-silvestri Nov 28, 2024
ebaa07d
fix typo
simone-silvestri Nov 28, 2024
879d611
make sure we delete the previous data before testing the download
simone-silvestri Nov 28, 2024
cd54cdb
Merge branch 'main' into ss/download-everywhere
simone-silvestri Nov 28, 2024
0c38ce1
should work
simone-silvestri Dec 2, 2024
25216bc
Merge branch 'ss/download-everywhere' of github.com:CliMA/ClimaOcean.…
simone-silvestri Dec 2, 2024
c0892b9
test distributed downloading
simone-silvestri Dec 2, 2024
6d73e98
Update test_distributed_utils.jl
simone-silvestri Dec 2, 2024
3fc3bd5
fix the download
simone-silvestri Dec 2, 2024
f3dec5c
generalize the downloader
simone-silvestri Dec 2, 2024
3525b76
generalize more
simone-silvestri Dec 2, 2024
6a3fa7d
generalize filename
simone-silvestri Dec 2, 2024
91be188
download_progress is part of the downloading utilities
simone-silvestri Dec 2, 2024
ab2563d
better docstring
simone-silvestri Dec 2, 2024
25b42cd
better docstring
simone-silvestri Dec 2, 2024
a77cebb
change docstring
simone-silvestri Dec 2, 2024
28f597c
Merge branch 'main' into ss/download-everywhere
simone-silvestri Dec 3, 2024
c8b6e97
fix tests
simone-silvestri Dec 4, 2024
42da589
Merge branch 'ss/download-everywhere' of github.com:CliMA/ClimaOcean.…
simone-silvestri Dec 4, 2024
ad64272
distribute among tasks
simone-silvestri Dec 5, 2024
5c38029
whoops added wrong file
simone-silvestri Dec 5, 2024
926394f
correct looping
simone-silvestri Dec 9, 2024
a3254de
bugfix
simone-silvestri Dec 10, 2024
5b65403
Merge branch 'main' into ss/download-everywhere
simone-silvestri Dec 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 18 additions & 9 deletions src/DataWrangling/DataWrangling.jl
Original file line number Diff line number Diff line change
Expand Up @@ -53,14 +53,24 @@ end
##### Downloading utilities
#####

# netrc-based downloader
# This downlader writes down a netrc file with the username and password for the given machine.
# To avoid storing passwords in plain text, it is recommended to use a temporary directory.
# For example:
# mktempdir(parent) do dir
# netrc_downloader(username, password, machine, dir)
# ... download files ...
# end
"""
netrc_downloader(username, password, machine, dir)

Create a downloader that uses a netrc file to authenticate with the given machine.
This downlader writes down a netrc file with the username and password for the given machine in the directory `dir`.
To avoid storing passwords in plain text, it is recommended to initialize the downloader in a temporary directory.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this avoid storing passwords in plain text?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmm, well it does but then the temporary directory with the netrc file is deleted when the do tmp; ...; end block is closed. So in theory for a finite amount of time the password is indeed stored in the filesystem. I can probably rewrite this comment

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory? I don't understand. The password is definitely stored in plain text right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not merely "theoretical", the design explicitly stores it in text.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, it is definitely stored in plain text

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also is it a problem that .netrc is written? Isn't this the recommended way to use .netrc anyways?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the environment variables route is much cleaner, but it looks like with .netrc we are forced to write down a file. I probably wouldn't like to have a stored file with a password automatically written so I think it is nice to raise this point in the docstring.


For example:

```
mktempdir(dir) do tmp

dowloader = netrc_downloader(username, password, machine, tmp)
... download files ...

end
```
"""
function netrc_downloader(username, password, machine, dir)
netrc_file = netrc_permission_file(username, password, machine, dir)
downloader = Downloads.Downloader()
Expand All @@ -79,7 +89,6 @@ function netrc_permission_file(username, password, machine, dir)
end

open(filepath, "a") do f
write(f, "\n")
write(f, "machine $machine login $username password $password\n")
end

Expand Down
Loading