Caching inpainting data + some updates to the one degree simulation #230

glwagner · 2024-11-10T06:27:23Z

The main thing this PR does is refactor ECCO_field so that inpainted data is cached and reloaded. The inpainted data is saved to a jld2 file. This really speeds up prototyping especially with small models (such as the three degree model in one_degree_simulation.jl) where inpainting dominates the cost of initializing a simulation.

Project.toml

glwagner · 2024-11-10T06:40:39Z

Just to clarify, the bump to 0.1.8 is also required to use the latest Oceananigans

navidcy · 2024-11-10T06:41:32Z

What do you mean "latest": latest tagged or dev version?

codecov · 2024-11-10T07:45:30Z

Codecov Report

Attention: Patch coverage is 0% with 53 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (7619d40) to head (5715bf4).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/DataWrangling/ECCO/ECCO.jl	0.00%	37 Missing ⚠️
src/DataWrangling/ECCO/ECCO_restoring.jl	0.00%	15 Missing ⚠️
src/OceanSimulations/OceanSimulations.jl	0.00%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #230   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files         33      33           
  Lines       1846    1869   +23     
=====================================
- Misses      1846    1869   +23

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

navidcy · 2024-11-10T08:07:21Z

when on GPU, I get the following:

julia> maximum(simulation.model.ocean.model.velocities.u)
ERROR: Number of threads per block exceeds kernel limit (896 > 768).
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] diagnose_launch_failure(f::CUDA.CuFunction, err::CUDA.CuError; blockdim::CUDA.CuDim3, threaddim::CUDA.CuDim3, shmem::Int64)
    @ CUDA ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:120
  [3] launch(::CUDA.CuFunction, ::CUDA.KernelState, ::CartesianIndices{…}, ::CartesianIndices{…}, ::CUDA.CuDeviceArray{…}, ::Oceananigans.AbstractOperations.ConditionalOperation{…}; blocks::Int64, threads::Int64, cooperative::Bool, shmem::Int64, stream::CUDA.CuStream)
    @ CUDA ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:73
  [4] launch
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:52 [inlined]
  [5] #972
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:189 [inlined]
  [6] macro expansion
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:149 [inlined]
  [7] macro expansion
    @ ./none:0 [inlined]
  [8] convert_arguments
    @ ./none:0 [inlined]
  [9] #cudacall#971
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:191 [inlined]
 [10] cudacall
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:187 [inlined]
 [11] macro expansion
    @ ~/.julia/packages/CUDA/2kjXI/src/compiler/execution.jl:279 [inlined]
 [12] macro expansion
    @ ./none:0 [inlined]
 [13] AbstractKernel
    @ ./none:0 [inlined]
 [14] macro expansion
    @ ~/.julia/packages/CUDA/2kjXI/src/compiler/execution.jl:114 [inlined]
 [15] mapreducedim!(f::typeof(identity), op::typeof(max), R::SubArray{…}, A::Oceananigans.AbstractOperations.ConditionalOperation{…}; init::Nothing)
    @ CUDA ~/.julia/packages/CUDA/2kjXI/src/mapreduce.jl:271
 [16] mapreducedim!(f::typeof(identity), op::typeof(max), R::SubArray{…}, A::Oceananigans.AbstractOperations.ConditionalOperation{…})
    @ CUDA ~/.julia/packages/CUDA/2kjXI/src/mapreduce.jl:169
 [17] mapreducedim!(f::Function, op::Function, R::SubArray{…}, A::Oceananigans.AbstractOperations.ConditionalOperation{…})
    @ GPUArrays ~/.julia/packages/GPUArrays/qt4ax/src/host/mapreduce.jl:10
 [18] maximum!(f::Function, r::SubArray{…}, A::Oceananigans.AbstractOperations.ConditionalOperation{…}; init::Bool)
    @ Base ./reducedim.jl:1036
 [19] maximum!(f::Function, r::Field{…}, a::Oceananigans.AbstractOperations.ConditionalOperation{…}; condition::Nothing, mask::Float64, kwargs::@Kwargs{…})
    @ Oceananigans.Fields ~/.julia/packages/Oceananigans/p4IUv/src/Fields/field.jl:670
 [20] maximum(f::Function, c::Field{…}; condition::Nothing, mask::Float64, dims::Function)
    @ Oceananigans.Fields ~/.julia/packages/Oceananigans/p4IUv/src/Fields/field.jl:700
 [21] maximum
    @ ~/.julia/packages/Oceananigans/p4IUv/src/Fields/field.jl:689 [inlined]
 [22] maximum(c::Field{…})
    @ Oceananigans.Fields ~/.julia/packages/Oceananigans/p4IUv/src/Fields/field.jl:709
 [23] top-level scope
    @ REPL[62]:1

caused by: CUDA error: too many resources requested for launch (code 701, ERROR_LAUNCH_OUT_OF_RESOURCES)
Stacktrace:
  [1] throw_api_error(res::CUDA.cudaError_enum)
    @ CUDA ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/libcuda.jl:30
  [2] check
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/libcuda.jl:37 [inlined]
  [3] cuLaunchKernel
    @ ~/.julia/packages/CUDA/2kjXI/lib/utils/call.jl:34 [inlined]
  [4] (::CUDA.var"#966#967"{Bool, Int64, CUDA.CuStream, CUDA.CuFunction, CUDA.CuDim3, CUDA.CuDim3})(kernelParams::Vector{Ptr{Nothing}})
    @ CUDA ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:66
  [5] macro expansion
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:33 [inlined]
  [6] macro expansion
    @ ./none:0 [inlined]
  [7] pack_arguments(::CUDA.var"#966#967"{…}, ::CUDA.KernelState, ::CartesianIndices{…}, ::CartesianIndices{…}, ::CUDA.CuDeviceArray{…}, ::Oceananigans.AbstractOperations.ConditionalOperation{…})
    @ CUDA ./none:0
  [8] launch(::CUDA.CuFunction, ::CUDA.KernelState, ::CartesianIndices{…}, ::CartesianIndices{…}, ::CUDA.CuDeviceArray{…}, ::Oceananigans.AbstractOperations.ConditionalOperation{…}; blocks::Int64, threads::Int64, cooperative::Bool, shmem::Int64, stream::CUDA.CuStream)
    @ CUDA ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:59
  [9] launch
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:52 [inlined]
 [10] #972
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:189 [inlined]
 [11] macro expansion
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:149 [inlined]
 [12] macro expansion
    @ ./none:0 [inlined]
 [13] convert_arguments
    @ ./none:0 [inlined]
 [14] #cudacall#971
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:191 [inlined]
 [15] cudacall
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/execution.jl:187 [inlined]
 [16] macro expansion
    @ ~/.julia/packages/CUDA/2kjXI/src/compiler/execution.jl:279 [inlined]
 [17] macro expansion
    @ ./none:0 [inlined]
 [18] AbstractKernel
    @ ./none:0 [inlined]
 [19] macro expansion
    @ ~/.julia/packages/CUDA/2kjXI/src/compiler/execution.jl:114 [inlined]
 [20] mapreducedim!(f::typeof(identity), op::typeof(max), R::SubArray{…}, A::Oceananigans.AbstractOperations.ConditionalOperation{…}; init::Nothing)
    @ CUDA ~/.julia/packages/CUDA/2kjXI/src/mapreduce.jl:271
 [21] mapreducedim!(f::typeof(identity), op::typeof(max), R::SubArray{…}, A::Oceananigans.AbstractOperations.ConditionalOperation{…})
    @ CUDA ~/.julia/packages/CUDA/2kjXI/src/mapreduce.jl:169
 [22] mapreducedim!(f::Function, op::Function, R::SubArray{…}, A::Oceananigans.AbstractOperations.ConditionalOperation{…})
    @ GPUArrays ~/.julia/packages/GPUArrays/qt4ax/src/host/mapreduce.jl:10
 [23] maximum!(f::Function, r::SubArray{…}, A::Oceananigans.AbstractOperations.ConditionalOperation{…}; init::Bool)
    @ Base ./reducedim.jl:1036
 [24] maximum!(f::Function, r::Field{…}, a::Oceananigans.AbstractOperations.ConditionalOperation{…}; condition::Nothing, mask::Float64, kwargs::@Kwargs{…})
    @ Oceananigans.Fields ~/.julia/packages/Oceananigans/p4IUv/src/Fields/field.jl:670
 [25] maximum(f::Function, c::Field{…}; condition::Nothing, mask::Float64, dims::Function)
    @ Oceananigans.Fields ~/.julia/packages/Oceananigans/p4IUv/src/Fields/field.jl:700
 [26] maximum
    @ ~/.julia/packages/Oceananigans/p4IUv/src/Fields/field.jl:689 [inlined]
 [27] maximum(c::Field{…})
    @ Oceananigans.Fields ~/.julia/packages/Oceananigans/p4IUv/src/Fields/field.jl:709
 [28] top-level scope
    @ REPL[62]:1
Some type information was truncated. Use `show(err)` to see complete types.

…into glw/more-one-degree

glwagner · 2024-11-10T17:29:42Z

Ok, this is a problem with reductions on immersed boundary grids unfortunately. For the moment, we need to use maximum(interior(u)).

navidcy · 2024-11-10T22:06:59Z

experiments/one_degree_simulation/one_degree_simulation.jl

it worths renaming the file if we are gonna leave it as 3 degree horizontal resolution

navidcy · 2024-11-10T22:07:34Z

Ok, this is a problem with reductions on immersed boundary grids unfortunately. For the moment, we need to use maximum(interior(u)).

I changed that in the progress function

glwagner · 2024-11-11T01:38:11Z

What do you mean "latest": latest tagged or dev version?

The latest tagged

navidcy · 2024-11-11T05:34:36Z

Is this ready?

navidcy · 2024-11-11T05:35:05Z

(if so, can we not merge until the docs built on main finishes to see if docs get deployed?)

simone-silvestri · 2024-11-13T08:16:30Z

also, the ECCO_field function should use @root begin for writing down the unpainted data to make sure there are no conflicts between different processes

simone-silvestri · 2024-11-13T17:04:15Z

Actually, I think it is necessary to give an option to toggle the writing / reading because we might want to use a different inpainting procedure than the one of the saved file.

Another option is to add some metadata with the inpainting steps used to the file and read it in only if it matches with the requested ECCO_field

glwagner · 2024-11-13T22:42:16Z

Actually, I think it is necessary to give an option to toggle the writing / reading because we might want to use a different inpainting procedure than the one of the saved file.

Another option is to add some metadata with the inpainting steps used to the file and read it in only if it matches with the requested ECCO_field

That would be nice, though it can be achieved currently by deleting the inpainting files...

simone-silvestri · 2024-11-15T12:56:20Z

On tartarus the one-degree simulation currently fails because of parameter space (I am using Oceananigans branch #3924)

julia> include("experiments/one_degree_simulation/one_degree_simulation.jl")
[ Info: Inpainting ECCO4Monthly temperature data from 1993-01-01T00:00:00...
[ Info:  ... (8.486 seconds)
[ Info: Inpainting ECCO4Monthly temperature data from 1993-02-01T00:00:00...
[ Info:  ... (39.686 ms)
[ Info: Inpainting ECCO4Monthly salinity data from 1993-01-01T00:00:00...
[ Info:  ... (40.863 ms)
[ Info: Inpainting ECCO4Monthly salinity data from 1993-02-01T00:00:00...
[ Info:  ... (39.685 ms)
┌ Warning: `ImmersedBoundaryCondition` is experimental.
└ @ Oceananigans.ImmersedBoundaries ~/.julia/packages/Oceananigans/5b4AH/src/ImmersedBoundaries/immersed_boundary_condition.jl:57
┌ Warning: `ImmersedBoundaryCondition` is experimental.
└ @ Oceananigans.ImmersedBoundaries ~/.julia/packages/Oceananigans/5b4AH/src/ImmersedBoundaries/immersed_boundary_condition.jl:57
┌ Warning: `ImmersedBoundaryCondition` is experimental.
└ @ Oceananigans.ImmersedBoundaries ~/.julia/packages/Oceananigans/5b4AH/src/ImmersedBoundaries/immersed_boundary_condition.jl:57
┌ Warning: `ImmersedBoundaryCondition` is experimental.
└ @ Oceananigans.ImmersedBoundaries ~/.julia/packages/Oceananigans/5b4AH/src/ImmersedBoundaries/immersed_boundary_condition.jl:57
┌ Warning: This simulation will run forever as stop iteration = stop time = wall time limit = Inf.
└ @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/5b4AH/src/Simulations/simulation.jl:55
[ Info: Initializing simulation...
[ Info: Time: 0 seconds, n: 0, Δt: 15 minutes, max|u|: (0.00e+00, 0.00e+00, 0.00e+00) m s⁻¹, extrema(T): (31.11, -1.96) ᵒC, wall time: 10.721 seconds
[ Info:     ... simulation initialization complete (7.444 seconds)
[ Info: Executing initial time step...
ERROR: LoadError: Kernel invocation uses too much parameter memory.
4.055 KiB exceeds the 4.000 KiB limit imposed by sm_70 / PTX v7.8.

Relevant parameters:
  [2] Gu::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}} uses 72 bytes
  [3] grid::ImmersedBoundaryGrid{Float64, Periodic, Oceananigans.Grids.RightConnected, Bounded, OrthogonalSphericalShellGrid{Float64, Periodic, Oceananigans.Grids.RightConnected, Bounded, OffsetArrays.OffsetMatrix{Float64, CUDA.CuDeviceMatrix{Float64, 1}}, OffsetArrays.OffsetVector{Float64, CUDA.CuDeviceVector{Float64, 1}}, OffsetArrays.OffsetVector{Float64, CUDA.CuDeviceVector{Float64, 1}}, OrthogonalSphericalShellGrids.Tripolar{Int64, Int64, Int64}, Nothing}, GridFittedBottom{Field{Center, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, Oceananigans.ImmersedBoundaries.CenterImmersedCondition}, Nothing, Nothing, Nothing} uses 1.406 KiB
  [5] args::Tuple{VectorInvariant{1, Float64, false, Oceananigans.Advection.EnstrophyConserving{Float64}, Oceananigans.Advection.VelocityStencil, Oceananigans.Advection.EnergyConserving{Float64}, Oceananigans.Advection.EnergyConserving{Float64}, Oceananigans.Advection.EnergyConserving{Float64}, Oceananigans.Advection.OnlySelfUpwinding{Oceananigans.Advection.EnergyConserving{Float64}, Oceananigans.Advection.FunctionStencil{typeof(Oceananigans.Advection.divergence_smoothness)}, Oceananigans.Advection.FunctionStencil{typeof(Oceananigans.Advection.divergence_smoothness)}, Oceananigans.Advection.FunctionStencil{typeof(Oceananigans.Advection.u_smoothness)}, Oceananigans.Advection.FunctionStencil{typeof(Oceananigans.Advection.v_smoothness)}}}, HydrostaticSphericalCoriolis{Oceananigans.Coriolis.ActiveCellEnstrophyConserving, Float64}, Tuple{Oceananigans.TurbulenceClosures.TKEBasedVerticalDiffusivities.CATKEVerticalDiffusivity{VerticallyImplicitTimeDiscretization, Oceananigans.TurbulenceClosures.TKEBasedVerticalDiffusivities.CATKEMixingLength{Float64}, Float64, Nothing, Oceananigans.TurbulenceClosures.TKEBasedVerticalDiffusivities.CATKEEquation{Float64}}, Oceananigans.TurbulenceClosures.IsopycnalSkewSymmetricDiffusivity{VerticallyImplicitTimeDiscretization, Float64, Float64, Oceananigans.TurbulenceClosures.SmallSlopeIsopycnalTensor{Float64}, Oceananigans.TurbulenceClosures.FluxTapering{Float64}, Float64}, HorizontalScalarDiffusivity{Oceananigans.TurbulenceClosures.ExplicitTimeDiscretization, @NamedTuple{T::Float64, S::Float64, e::Float64}, Float64, @NamedTuple{T::Float64, S::Float64, e::Float64}}}, Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, @NamedTuple{u::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, v::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, w::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}, SplitExplicitFreeSurface{OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, @NamedTuple{U::Field{Face, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, V::Field{Center, Face, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}}, @NamedTuple{η::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, U::Field{Face, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, V::Field{Center, Face, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}}, Float64, Nothing, Oceananigans.Models.HydrostaticFreeSurfaceModels.SplitExplicitFreeSurfaces.FixedSubstepNumber{Float64, NTuple{50, Float64}}, Oceananigans.Models.HydrostaticFreeSurfaceModels.SplitExplicitFreeSurfaces.ForwardBackwardScheme}, @NamedTuple{T::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, S::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, e::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}, Buoyancy{SeawaterBuoyancy{Float64, SeawaterPolynomials.BoussinesqEquationOfState{SeawaterPolynomials.TEOS10.TEOS10SeawaterPolynomial{Float64}, Float64}, Nothing, Nothing}, Oceananigans.Grids.NegativeZDirection}, Tuple{@NamedTuple{κu::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, κc::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, κe::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Le::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Jᵇ::Field{Center, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, previous_compute_time::CUDA.CuRefValue{Float64}, previous_velocities::@NamedTuple{u::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, v::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}, _tupled_tracer_diffusivities::@NamedTuple{T::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, S::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, e::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}, _tupled_implicit_linear_coefficients::@NamedTuple{T::Oceananigans.Fields.ZeroField{Int64, 3}, S::Oceananigans.Fields.ZeroField{Int64, 3}, e::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}}, @NamedTuple{ϵ_R₃₃::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}, Nothing}, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, @NamedTuple{}, @NamedTuple{time::Float64, last_Δt::Float64, last_stage_Δt::Float64, iteration::Int64, stage::Int64}} uses 2.562 KiB

simone-silvestri · 2024-11-15T13:18:12Z

I have split the arguments, this is how they look like

Advection:
VectorInvariant{1, Float64, false, Oceananigans.Advection.EnstrophyConserving{Float64}, Oceananigans.Advection.VelocityStencil, Oceananigans.Advection.EnergyConserving{Float64}, Oceananigans.Advection.EnergyConserving{Float64}, Oceananigans.Advection.EnergyConserving{Float64}, Oceananigans.Advection.OnlySelfUpwinding{Oceananigans.Advection.EnergyConserving{Float64}, Oceananigans.Advection.FunctionStencil{typeof(Oceananigans.Advection.divergence_smoothness)}, Oceananigans.Advection.FunctionStencil{typeof(Oceananigans.Advection.divergence_smoothness)}, Oceananigans.Advection.FunctionStencil{typeof(Oceananigans.Advection.u_smoothness)}, Oceananigans.Advection.FunctionStencil{typeof(Oceananigans.Advection.v_smoothness)}}}, 

Coriolis:
HydrostaticSphericalCoriolis{Oceananigans.Coriolis.ActiveCellEnstrophyConserving, Float64}, 

Closures:

(1) Catke:
Oceananigans.TurbulenceClosures.TKEBasedVerticalDiffusivities.CATKEVerticalDiffusivity{VerticallyImplicitTimeDiscretization, Oceananigans.TurbulenceClosures.TKEBasedVerticalDiffusivities.CATKEMixingLength{Float64}, Float64, Nothing, Oceananigans.TurbulenceClosures.TKEBasedVerticalDiffusivities.CATKEEquation{Float64}}

(2) GM + Redi:
Oceananigans.TurbulenceClosures.IsopycnalSkewSymmetricDiffusivity{VerticallyImplicitTimeDiscretization, Float64, Float64, Oceananigans.TurbulenceClosures.SmallSlopeIsopycnalTensor{Float64}, Oceananigans.TurbulenceClosures.FluxTapering{Float64}, Float64}, HorizontalScalarDiffusivity{Oceananigans.TurbulenceClosures.ExplicitTimeDiscretization, @NamedTuple{T::Float64, S::Float64, e::Float64}, Float64, @NamedTuple{T::Float64, S::Float64, e::Float64}}},

Immersed Boundary Conditions u:
Oceananigans.BoundaryConditions.DefaultBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}

Velocities:
@NamedTuple{u::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, v::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, w::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}, 

Free Surface:
SplitExplicitFreeSurface{OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, @NamedTuple{U::Field{Face, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, V::Field{Center, Face, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}}, @NamedTuple{η::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, U::Field{Face, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, V::Field{Center, Face, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}}, Float64, Nothing, Oceananigans.Models.HydrostaticFreeSurfaceModels.SplitExplicitFreeSurfaces.FixedSubstepNumber{Float64, NTuple{50, Float64}}, Oceananigans.Models.HydrostaticFreeSurfaceModels.SplitExplicitFreeSurfaces.ForwardBackwardScheme}, 

Tracers:
@NamedTuple{T::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, S::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, e::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}, 

Buoyancy:
Buoyancy{SeawaterBuoyancy{Float64, SeawaterPolynomials.BoussinesqEquationOfState{SeawaterPolynomials.TEOS10.TEOS10SeawaterPolynomial{Float64}, Float64}, Nothing, Nothing}, Oceananigans.Grids.NegativeZDirection}, 

Diffusivities:

(1) Catke:
@NamedTuple{κu::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, κc::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, κe::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Le::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Jᵇ::Field{Center, Center, Nothing, Nothing, Nothing, Nothing, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, Float64, Nothing, Nothing, Nothing}, previous_compute_time::CUDA.CuRefValue{Float64}, previous_velocities::@NamedTuple{u::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, v::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}, _tupled_tracer_diffusivities::@NamedTuple{T::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, S::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, e::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}, _tupled_implicit_linear_coefficients::@NamedTuple{T::Oceananigans.Fields.ZeroField{Int64, 3}, S::Oceananigans.Fields.ZeroField{Int64, 3}, e::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}}, 

(2) GM + Redi
@NamedTuple{ϵ_R₃₃::OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}, Nothing}, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, @NamedTuple{}, @NamedTuple{time::Float64, last_Δt::Float64, last_stage_Δt::Float64, iteration::Int64, stage::Int64}}

I think on simple change might be changing the weights in the splitexplicit free surface from an NTuple{50, Float64} to an array, which will save us 8 * 50 = 400 bytes. Not a lot but enough to make the simulation run.

Actually, now that I think about it, we do not need the free surface in the velocities kernel if it is not an ExplicitFreeSurface, so we can remove completely this argument and pass a nothing in the kernel

glwagner · 2024-11-18T15:26:53Z

It doesn't matter if this works on GPU for this PR. We can fix that later.

glwagner · 2024-11-20T06:16:03Z

Could this PR be the issue? @navidcy @simone-silvestri

glwagner and others added 2 commits November 9, 2024 23:26

Update compat

6adbbbf

don't relax compass

746b599

navidcy reviewed Nov 10, 2024

View reviewed changes

Project.toml Outdated Show resolved Hide resolved

Update Project.toml

4ac5f21

glwagner changed the title ~~Keeping working on one degree simulation~~ Bump OrthogonalSphericalShellGrids to 0.1.8 and keeping working on one degree simulation Nov 10, 2024

Update Project.toml

2736766

glwagner changed the title ~~Bump OrthogonalSphericalShellGrids to 0.1.8 and keeping working on one degree simulation~~ Bump OrthogonalSphericalShellGrids to 0.1.7 and keeping working on one degree simulation Nov 10, 2024

glwagner added 2 commits November 9, 2024 23:45

Update Project.toml

8545662

Generalize default free surface for IBG

e148194

navidcy added 4 commits November 10, 2024 19:44

Update Project.toml

9c3aa7e

use interior for max of field

ce7c116

Merge branch 'glw/more-one-degree' of github.com:CliMA/ClimaOcean.jl …

f21a42b

…into glw/more-one-degree

smaller Δt

7509819

glwagner mentioned this pull request Nov 10, 2024

free_surface default for a DistributedGrid #232

Closed

Merge branch 'main' into glw/more-one-degree

b3913e9

navidcy self-requested a review November 10, 2024 22:06

navidcy approved these changes Nov 10, 2024

View reviewed changes

navidcy reviewed Nov 10, 2024

View reviewed changes

navidcy approved these changes Nov 10, 2024

View reviewed changes

Merge branch 'main' into glw/more-one-degree

d58fd5c

Implement inpainting saving

870e57a

Modify one degree simulation

7008439

simone-silvestri added 2 commits November 13, 2024 18:06

some more updates to the one degree simulation

13091c9

bugfix

60b8a25

simone-silvestri added 5 commits November 15, 2024 10:08

Merge remote-tracking branch 'origin/main' into glw/more-one-degree

680f4b5

cache inpainted data?

eef5a1b

ebugfix

c653299

advancing

d1dd2f3

small bug

b1ae2ae

simone-silvestri and others added 9 commits November 15, 2024 08:56

some bugfixes

ccbe516

add a @root to caching

bccd3a8

open with read permission

f1d3ced

interpolate on grid beforehand

517782c

fix minimum-seaice in the meantime

3539660

comment

235a67c

trhow an error

5c23b6e

correct arch

bcfc307

Merge remote-tracking branch 'origin/main' into glw/more-one-degree

4c26acd

glwagner added 2 commits November 18, 2024 08:28

rm fake sea ice again

88cc7e0

Make default true for caching

5715bf4

glwagner changed the title ~~Saving inpainting data + OrthogonalSphericalShellGrids 0.1.7~~ Saving inpainting data + some updates to the one degree simulation Nov 18, 2024

glwagner changed the title ~~Saving inpainting data + some updates to the one degree simulation~~ Caching inpainting data + some updates to the one degree simulation Nov 18, 2024

glwagner merged commit 5a32b35 into main Nov 18, 2024
19 checks passed

glwagner deleted the glw/more-one-degree branch November 18, 2024 16:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Caching inpainting data + some updates to the one degree simulation #230

Caching inpainting data + some updates to the one degree simulation #230

glwagner commented Nov 10, 2024 •

edited

Loading

glwagner commented Nov 10, 2024

navidcy commented Nov 10, 2024

codecov bot commented Nov 10, 2024 •

edited

Loading

navidcy commented Nov 10, 2024

glwagner commented Nov 10, 2024

navidcy Nov 10, 2024

navidcy commented Nov 10, 2024

glwagner commented Nov 11, 2024

navidcy commented Nov 11, 2024

navidcy commented Nov 11, 2024

simone-silvestri commented Nov 13, 2024

simone-silvestri commented Nov 13, 2024

glwagner commented Nov 13, 2024

simone-silvestri commented Nov 15, 2024

simone-silvestri commented Nov 15, 2024 •

edited

Loading

glwagner commented Nov 18, 2024 •

edited

Loading

glwagner commented Nov 20, 2024

Caching inpainting data + some updates to the one degree simulation #230

Caching inpainting data + some updates to the one degree simulation #230

Conversation

glwagner commented Nov 10, 2024 • edited Loading

glwagner commented Nov 10, 2024

navidcy commented Nov 10, 2024

codecov bot commented Nov 10, 2024 • edited Loading

Codecov Report

navidcy commented Nov 10, 2024

glwagner commented Nov 10, 2024

navidcy Nov 10, 2024

Choose a reason for hiding this comment

navidcy commented Nov 10, 2024

glwagner commented Nov 11, 2024

navidcy commented Nov 11, 2024

navidcy commented Nov 11, 2024

simone-silvestri commented Nov 13, 2024

simone-silvestri commented Nov 13, 2024

glwagner commented Nov 13, 2024

simone-silvestri commented Nov 15, 2024

simone-silvestri commented Nov 15, 2024 • edited Loading

glwagner commented Nov 18, 2024 • edited Loading

glwagner commented Nov 20, 2024

glwagner commented Nov 10, 2024 •

edited

Loading

codecov bot commented Nov 10, 2024 •

edited

Loading

simone-silvestri commented Nov 15, 2024 •

edited

Loading

glwagner commented Nov 18, 2024 •

edited

Loading