Parallel Ribasim core tests fail non-deterministically on MacOS

[SouthEndMusic]

Only this failure remains: #825 (comment)

Specifically @testitem "Allocation solve" and @testitem "Expand logic_mapping". They pass when running the tests sequentially (runtests(Ribasim) in runtests.jl) but sometimes fail when running the tests in parallel
(runtests(Ribasim; nworkers = min(4, Sys.CPU_THREADS ÷ 2), nworker_threads = 2) in runtests.jl).

https://docs.juliahub.com/General/ReTestItems/stable/autodocs/#ReTestItems.runtests

[visr]

Perhaps it is unrelated to parallel after all, since there are also random failures now that we switched to sequential. Or perhaps both issues occur.

Seen locally (now updated to 4.0, see #827):

Test Failed at D:\Ribasim\core\test\allocation_test.jl:22
  Expression: F[(NodeID(1), NodeID(2))] ≈ 4.5
   Evaluated: 4.0 ≈ 4.5

But after changing this, on CI we get the opposite https://github.com/Deltares/Ribasim/actions/runs/6982110719/job/19000606007?pr=827#step:9:444

Test Failed at /home/runner/work/Ribasim/Ribasim/core/test/allocation_test.jl:22
  Expression: F[(NodeID(1), NodeID(2))] ≈ 4.0
   Evaluated: 4.5 ≈ 4.0

A different issue, on main CI, also non-deterministic: https://github.com/Deltares/Ribasim/actions/runs/6981404082/job/18998462984?pr=823#step:9:540

Test Failed at /home/runner/work/Ribasim/Ribasim/core/test/utils_test.jl:173
  Expression: Ribasim.expand_logic_mapping(logic_mapping)
    Expected: Multiple control states found for DiscreteControl node #1 for truth state `TTF`: foo, bar.
  No exception thrown

[visr]

In 8ef0361 I temporarily workaround / disable these two tests, and re-enable parallel tests, to get CI functional.

Proper fixes in #828 (comment)

[visr]

Here is one non-deterministic issue that still happens sometimes, that is related to the switch to ReTestItems: https://github.com/Deltares/Ribasim/actions/runs/6983243135/job/19004024801#step:9:443

ERROR: LoadError: IOError: mkdir("/var/folders/3s"; mode=0o777): permission denied (EACCES)
Stacktrace:
  [1] uv_error
    @ Base ./libuv.jl:100 [inlined]
  [2] mkdir(path::String; mode::UInt16)
    @ Base.Filesystem ./file.jl:185
  [3] mkdir
    @ Base.Filesystem ./file.jl:177 [inlined]
  [4] mkpath(path::String; mode::UInt16)
    @ Base.Filesystem ./file.jl:241
  [5] mkpath(path::String; mode::UInt16) (repeats 3 times)
    @ Base.Filesystem ./file.jl:239
  [6] mkpath
    @ ./file.jl:235 [inlined]
  [7] runtests(shouldrun::typeof(ReTestItems.default_shouldrun), paths::String; nworkers::Int64, nworker_threads::Int64, worker_init_expr::Expr, testitem_timeout::Float64, retries::Int64, memory_threshold::Float64, debug::Int64, name::Nothing, tags::Nothing, report::Bool, logs::Symbol, verbose_results::Bool, test_end_expr::Expr)
    @ ReTestItems ~/.julia/packages/ReTestItems/HZCMZ/src/ReTestItems.jl:234
  [8] runtests(shouldrun::Function, pkg::Module; kw::@Kwargs{nworkers::Int64, nworker_threads::Int64})
    @ ReTestItems ~/.julia/packages/ReTestItems/HZCMZ/src/ReTestItems.jl:196
  [9] runtests
    @ ReTestItems ~/.julia/packages/ReTestItems/HZCMZ/src/ReTestItems.jl:193 [inlined]
 [10] #runtests#38
    @ ReTestItems ~/.julia/packages/ReTestItems/HZCMZ/src/ReTestItems.jl:192 [inlined]

This looks like it could be a ReTestItems issue, if it does parallel mkdir and the folder name is "/var/folders/3s", that looks like a recipe for race conditions.

[Hofer-Julian]

@visr Now that #872 has been merged, should we close this issue?

[Hofer-Julian]

As discussed with @visr we keep it open for now, I will put it back into "To do"

[SnippenE]

Can not fix because we couldn't find a solution