Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Distributed DataWrangling #207

Closed
wants to merge 820 commits into from
Closed

Distributed DataWrangling #207

wants to merge 820 commits into from

Conversation

simone-silvestri
Copy link
Collaborator

This PR is a proposal to solve #206

It introduces a couple of distributed utilities:

blocking_download(url, filepath; kw...) # only rank 0 downloads
blocking_run(cmd) # only rank 0 executes the command
distributed_run(cmd) # if `cmd isa Union{AbstractVector, Tuple}` split the commands throughout the ranks

We would have to find a way to test this since distributed tests are not in place yet

simone-silvestri and others added 30 commits June 18, 2024 09:04
@simone-silvestri
remove useless file
6a53bf5
@simone-silvestri
bugfix
50fe313
@simone-silvestri
functions for time quantities
5862a0a
@simone-silvestri
bottom drag coefficient
ebf14fe
@simone-silvestri
back to reference height
103b504
@simone-silvestri
some improvements
7d4eb0e
@simone-silvestri
bugfix
8c165b9
@simone-silvestri
revert to chloride
34b6265
@simone-silvestri
changed pipeline
c5babb7
@simone-silvestri
use github documenter for the moment
215b692
@simone-silvestri
bugfix
a1ce58b
@simone-silvestri
change measurement to reference
31add81
@simone-silvestri
conversion depending on k
df0456e
@simone-silvestri
change to the
beaeacf
@simone-silvestri
change name
f235888
@simone-silvestri
bugfix
f3487d9
@simone-silvestri
namechange
c695266
@simone-silvestri
add surface flux tests
8473926
@simone-silvestri
change name to "gusty" velocity difference
2a6b1a4
@simone-silvestri
typo
36a3e17
@simone-silvestri
bugfix
29993f6
@simone-silvestri
correctly include runoff in forcing
24a551c
@simone-silvestri
bugfix
f1099d9
@simone-silvestri
speed up generate fluxes
a0682f0
@simone-silvestri
Merge remote-tracking branch 'origin/ss/omip-simulation' into ss/omip…
4f1f75e 
…-with-restoring
@simone-silvestri
error in project
fc53863
@simone-silvestri
ok found the issue!
26b8dee
@simone-silvestri
Merge branch 'ss/omip-with-restoring' of github.com:CliMA/ClimaOcean.…
8de82f2 
…jl into ss/omip-with-restoring
@simone-silvestri
correct the functions
e1f386d
@simone-silvestri
change to z_data
0309ddd
@simone-silvestri
Merge branch 'ss/mpi-simulations-for-conferenve' of github.com:CliMA/…
83ec082 
…ClimaOcean.jl into ss/mpi-simulations-for-conferenve
@simone-silvestri
Merge branch 'main' into ss/immersed-boundary-conditions-take-2
14c9e94
@simone-silvestri
try it out?
9c19dda
@simone-silvestri
bah not nice but it works
e1ce0b7
@simone-silvestri
add climatology
9bafffa
@simone-silvestri
Merge branch 'main' into ss/immersed-boundary-conditions-take-2
50f2046
@simone-silvestri
update project
18b1ed8
@simone-silvestri
Merge branch 'main' into ss/immersed-boundary-conditions-take-2
cca28de
@simone-silvestri
update Oceananigans
fe7faf7
@simone-silvestri
Merge branch 'main' into ss/immersed-boundary-conditions-take-2
43379eb
@simone-silvestri
Merge remote-tracking branch 'origin/main' into ss/mpi-simulations-fo…
f62ba36 
…r-conferenve
@simone-silvestri
test it out on 8 gpus
d449469
@simone-silvestri
Merge remote-tracking branch 'origin/ss/immersed-boundary-conditions-…
d0da037 
…take-2' into ss/mpi-simulations-for-conferenve
@simone-silvestri
add new prototype
03b9859
@simone-silvestri
only surface restoring
407f5fb
@simone-silvestri
correct defaults
5cac620
@simone-silvestri
Merge branch 'main' into ss/mpi-simulations-for-conferenve
e459a16
@simone-silvestri
remove diagnostics
c8d97e4
@simone-silvestri
remove manifest
185c2a4
@simone-silvestri
remove stuff
208f011
@simone-silvestri
back in OceanSimulations
2757ff5
@simone-silvestri
back on prototype
f2b5bd6
@simone-silvestri
back also here
4741ec1
@simone-silvestri
distributed data wrangling
7033fab
@simone-silvestri
back to previous code
f52a0d8
@simone-silvestri
better
26b96e0
@simone-silvestri simone-silvestri mentioned this pull request Oct 31, 2024
Merged
@simone-silvestri
Copy link
Collaborator Author

Closed in favor of #208

@simone-silvestri simone-silvestri deleted the ss/mpi-simulations-for-conferenve branch October 31, 2024 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@simone-silvestri @navidcy