Incompatability of SCS with Julia 1.7.2 on Mac "M1"

[odunbar]

coauthored with @anagha548

On fresh install with Mac M1, we found an issue with precompile of "BinaryProviders" within SCS

Seems like this issue is down to them choosing not to support Julia 1.7, it however is fixed with Julia 1.6
jump-dev/SCS.jl#235

[ilopezgp]

Which version of Julia 1.7? 1.7.3 does not work with M1. A certain 1.7.0 binary should work according to julialang:

Note that unlike with v1.7.2, binaries for macOS M1, a tier 3 platform, are not available for this release due to infrastructure issues and the general instability of native M1 support in 1.7. (That has improved in 1.8.) Users with M1 Macs can still run the Intel 1.7 binary with Rosetta.

[odunbar]

It was with v1.7.2 which according to that quote should work. But did not. At the moment we have reverted to 1.6.6 which worked fine - I thought I would flag it up

[ilopezgp]

Thanks! Hopefully 1.8 (wich will come out any moment now) has resolved this issue.

[tsj5]

Revisiting this issue now that 1.8.0 has been released. All results here are for a fresh install of EKP on an M1 mac, using the macarch64 (M1-specific) build of 1.8.0.

1. Building on 1.8 instead of 1.7 doesn't resolve the precompilation error with BinaryProviders reported above; here's the traceback:

Traceback

julia> import Pkg; Pkg.precompile()
Precompiling project...
  ✗ BinaryProvider
  ✗ SCS
  ✗ EnsembleKalmanProcesses
  0 dependencies successfully precompiled in 7 seconds. 71 already precompiled.

ERROR: The following 2 direct dependencies failed to precompile:

EnsembleKalmanProcesses [aa8a2aa5-91d8-4396-bcef-d4f2ec43552d]

Failed to precompile EnsembleKalmanProcesses [aa8a2aa5-91d8-4396-bcef-d4f2ec43552d] to /Users/tsj/.julia/compiled/v1.8/EnsembleKalmanProcesses/jl_TJZfxP.
ERROR: LoadError: UndefVarError: libscsindir not defined
Stacktrace:
 [1] getproperty(x::Module, f::Symbol)
   @ Base ./Base.jl:31
 [2] top-level scope
   @ ~/.julia/packages/SCS/kEMNd/src/SCS.jl:27
 [3] include
   @ ./Base.jl:419 [inlined]
 [4] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
   @ Base ./loading.jl:1554
 [5] top-level scope
   @ stdin:1
in expression starting at /Users/tsj/.julia/packages/SCS/kEMNd/src/SCS.jl:1
in expression starting at stdin:1
ERROR: LoadError: Failed to precompile SCS [c946c3f1-0d1f-5ce8-9dea-7daa1f7e2d13] to /Users/tsj/.julia/compiled/v1.8/SCS/jl_2MaWBv.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:1705
  [3] compilecache
    @ ./loading.jl:1649 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1337
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
  [6] macro expansion
    @ ./loading.jl:1180 [inlined]
  [7] macro expansion
    @ ./lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
  [9] include(mod::Module, _path::String)
    @ Base ./Base.jl:419
 [10] include(x::String)
    @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanProcesses.jl:1
 [11] top-level scope
    @ ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanProcess.jl:353
 [12] include(mod::Module, _path::String)
    @ Base ./Base.jl:419
 [13] include(x::String)
    @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanProcesses.jl:1
 [14] top-level scope
    @ ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanProcesses.jl:13
 [15] include
    @ ./Base.jl:419 [inlined]
 [16] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1554
 [17] top-level scope
    @ stdin:1
in expression starting at /Users/tsj/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/SparseEnsembleKalmanInversion.jl:2
in expression starting at /Users/tsj/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanProcess.jl:353
in expression starting at /Users/tsj/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanProcesses.jl:1
in expression starting at stdin:1

SCS [c946c3f1-0d1f-5ce8-9dea-7daa1f7e2d13]

Failed to precompile SCS [c946c3f1-0d1f-5ce8-9dea-7daa1f7e2d13] to /Users/tsj/.julia/compiled/v1.8/SCS/jl_zrE4Pd.
ERROR: LoadError: UndefVarError: libscsindir not defined
Stacktrace:
 [1] getproperty(x::Module, f::Symbol)
   @ Base ./Base.jl:31
 [2] top-level scope
   @ ~/.julia/packages/SCS/kEMNd/src/SCS.jl:27
 [3] include
   @ ./Base.jl:419 [inlined]
 [4] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
   @ Base ./loading.jl:1554
 [5] top-level scope
   @ stdin:1
in expression starting at /Users/tsj/.julia/packages/SCS/kEMNd/src/SCS.jl:1
in expression starting at stdin:1

Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types /Applications/Julia-1.8-arm.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Types.jl:67
 [2] precompile(ctx::Pkg.Types.Context, pkgs::Vector{String}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API /Applications/Julia-1.8-arm.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/API.jl:1427
 [3] precompile
   @ /Applications/Julia-1.8-arm.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/API.jl:1058 [inlined]
 [4] #precompile#225
   @ /Applications/Julia-1.8-arm.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/API.jl:1057 [inlined]
 [5] precompile (repeats 2 times)
   @ /Applications/Julia-1.8-arm.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/API.jl:1057 [inlined]
 [6] top-level scope
   @ REPL[3]:1

2. Upgrades were blocked by the version constraints manually specified in Project.toml. There's been a lot of recent release activity for Convex.jl and SCS.jl, so we might hope updating would fix the build error.

Pkg compat report

(EnsembleKalmanProcesses) pkg> status --outdated -m
Project EnsembleKalmanProcesses v0.9.1
Status `~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/Manifest.toml`
⌅ [14f7f29c] AMD v0.4.0 (<v0.5.0): LDLFactorizations
⌅ [1520ce14] AbstractTrees v0.3.4 (<v0.4.2): Convex
⌅ [f65535da] Convex v0.14.18 (<v0.15.2) [compat]
⌅ [40e66cde] LDLFactorizations v0.8.2 (<v0.9.0): Convex
⌅ [b8f27783] MathOptInterface v0.10.9 (<v1.7.0): Convex, SCS
⌅ [d8a4904e] MutableArithmetics v0.3.3 (<v1.0.4): MathOptInterface
⌅ [c946c3f1] SCS v0.8.2 (<v1.1.2) [compat]
⌅ [af6e375f] SCS_GPU_jll v2.1.4+0 (<v3.2.0+0): SCS
⌅ [f4f2fc5b] SCS_jll v2.1.2+1 (<v3.2.0+0): SCS

Results of Pkg.update()

julia> Pkg.update()
    Updating registry at `~/.julia/registries/General.toml`
   Installed AbstractTrees ───── v0.4.2
   Installed AMD ─────────────── v0.5.0
   Installed LDLFactorizations ─ v0.9.0
   Installed Convex ──────────── v0.15.2
   Installed MathOptInterface ── v1.7.0
  Downloaded artifact: OpenBLAS32
  Downloaded artifact: SCS
    Updating `~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/Project.toml`
  [f65535da] ↑ Convex v0.14.18 ⇒ v0.15.2
  [c946c3f1] ↑ SCS v0.8.2 ⇒ v1.1.2
    Updating `~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/Manifest.toml`
  [14f7f29c] ↑ AMD v0.4.0 ⇒ v0.5.0
  [1520ce14] ↑ AbstractTrees v0.3.4 ⇒ v0.4.2
  [b99e7846] - BinaryProvider v0.5.10
  [f65535da] ↑ Convex v0.14.18 ⇒ v0.15.2
  [40e66cde] ↑ LDLFactorizations v0.8.2 ⇒ v0.9.0
  [b8f27783] ↑ MathOptInterface v0.10.9 ⇒ v1.7.0
  [d8a4904e] ↑ MutableArithmetics v0.3.3 ⇒ v1.0.4
  [c946c3f1] ↑ SCS v0.8.2 ⇒ v1.1.2
⌅ [656ef2d0] + OpenBLAS32_jll v0.3.17+0
  [af6e375f] ↑ SCS_GPU_jll v2.1.4+0 ⇒ v3.2.0+0
  [f4f2fc5b] ↑ SCS_jll v2.1.2+1 ⇒ v3.2.0+0

3. This fixes the build error, but introduces a test suite error due to (apparently) a breaking change in SCS.Optimizer. I'm unsure if there's overlap with the other difficulties running tests on 1.8 reported in Test on latest Julia version (~1.8) #188.

Results of Pkg.test()

     Testing EnsembleKalmanProcesses
      Status `/private/var/folders/97/_ch5sjw12pqc7m99x35f6g9w0000gn/T/jl_s1d7Xj/Project.toml`
  [f65535da] Convex v0.15.2
  [31c24e10] Distributions v0.25.68
  [ffbed154] DocStringExtensions v0.9.1
  [aa8a2aa5] EnsembleKalmanProcesses v0.9.1 `~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl`
  [429524aa] Optim v1.7.2
  [1fd47b50] QuadGK v2.4.2
  [c946c3f1] SCS v1.1.2
  [860ef19b] StableRNGs v1.0.0
  [2913bbd2] StatsBase v0.33.21
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [9a3f8284] Random `@stdlib/Random`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics `@stdlib/Statistics`
  [fa267f1f] TOML v1.0.0 `@stdlib/TOML`
  [8dfed614] Test `@stdlib/Test`
      Status `/private/var/folders/97/_ch5sjw12pqc7m99x35f6g9w0000gn/T/jl_s1d7Xj/Manifest.toml`
  [14f7f29c] AMD v0.5.0
  [1520ce14] AbstractTrees v0.4.2
  [30b0a656] ArrayInterfaceCore v0.1.17
  [6e4b80f9] BenchmarkTools v1.3.1
  [49dc2e85] Calculus v0.5.1
  [d360d2e6] ChainRulesCore v1.15.3
  [9e997f8a] ChangesOfVariables v0.1.4
  [523fee87] CodecBzip2 v0.7.2
  [944b1d66] CodecZlib v0.7.0
  [bbf7d656] CommonSubexpressions v0.3.0
  [34da2185] Compat v4.2.0
  [187b0558] ConstructionBase v1.4.1
  [f65535da] Convex v0.15.2
  [9a962f9c] DataAPI v1.10.0
  [864edb3b] DataStructures v0.18.13
  [b429d917] DensityInterface v0.4.0
  [163ba53b] DiffResults v1.0.3
  [b552c78f] DiffRules v1.11.1
  [31c24e10] Distributions v0.25.68
  [ffbed154] DocStringExtensions v0.9.1
  [fa6b7ba4] DualNumbers v0.6.8
  [aa8a2aa5] EnsembleKalmanProcesses v0.9.1 `~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl`
  [1a297f60] FillArrays v0.13.2
  [6a86dc24] FiniteDiff v2.15.0
  [f6369f11] ForwardDiff v0.10.32
  [34004b35] HypergeometricFunctions v0.3.11
  [3587e190] InverseFunctions v0.1.7
  [92d709cd] IrrationalConstants v0.1.1
  [692b3bcd] JLLWrappers v1.4.1
  [682c06a0] JSON v0.21.3
  [40e66cde] LDLFactorizations v0.9.0
  [d3d80556] LineSearches v7.2.0
  [2ab3a3ac] LogExpFunctions v0.3.18
  [1914dd2f] MacroTools v0.5.9
  [b8f27783] MathOptInterface v1.7.0
  [e1d29d7a] Missings v1.0.2
  [d8a4904e] MutableArithmetics v1.0.4
  [d41bc354] NLSolversBase v7.8.2
  [77ba4419] NaNMath v1.0.1
  [429524aa] Optim v1.7.2
  [bac558e1] OrderedCollections v1.4.1
  [90014a1f] PDMats v0.11.16
  [d96e819e] Parameters v0.12.3
  [69de0a69] Parsers v2.4.0
  [85a6dd25] PositiveFactorizations v0.2.4
  [21216c6a] Preferences v1.3.0
  [1fd47b50] QuadGK v2.4.2
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [79098fc4] Rmath v0.7.0
  [c946c3f1] SCS v1.1.2
  [efcf1570] Setfield v1.1.1
  [a2af1166] SortingAlgorithms v1.0.1
  [276daf66] SpecialFunctions v2.1.7
  [860ef19b] StableRNGs v1.0.0
  [90137ffa] StaticArrays v1.5.6
  [1e83bf80] StaticArraysCore v1.3.0
  [82ae8749] StatsAPI v1.5.0
  [2913bbd2] StatsBase v0.33.21
  [4c63d2b9] StatsFuns v1.0.1
  [3bb67fe8] TranscodingStreams v0.9.8
  [3a884ed6] UnPack v1.0.2
  [6e34b625] Bzip2_jll v1.0.8+0
⌅ [656ef2d0] OpenBLAS32_jll v0.3.17+0
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [f50d1b31] Rmath_jll v0.3.0+0
  [af6e375f] SCS_GPU_jll v3.2.0+0
  [f4f2fc5b] SCS_jll v3.2.0+0
  [0dad84c5] ArgTools v1.1.1 `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [8ba89e20] Distributed `@stdlib/Distributed`
  [f43a241f] Downloads v1.6.0 `@stdlib/Downloads`
  [7b1f6079] FileWatching `@stdlib/FileWatching`
  [9fa8497b] Future `@stdlib/Future`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [b27032c2] LibCURL v0.6.3 `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [a63ad114] Mmap `@stdlib/Mmap`
  [ca575930] NetworkOptions v1.2.0 `@stdlib/NetworkOptions`
  [44cfe95a] Pkg v1.8.0 `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [9abbd945] Profile `@stdlib/Profile`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics `@stdlib/Statistics`
  [4607b0f0] SuiteSparse `@stdlib/SuiteSparse`
  [fa267f1f] TOML v1.0.0 `@stdlib/TOML`
  [a4e569a6] Tar v1.10.0 `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [e66e0078] CompilerSupportLibraries_jll v0.5.2+0 `@stdlib/CompilerSupportLibraries_jll`
  [deac9b47] LibCURL_jll v7.84.0+0 `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll v1.10.2+0 `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll v2.28.0+0 `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll v2022.2.1 `@stdlib/MozillaCACerts_jll`
  [4536629a] OpenBLAS_jll v0.3.20+0 `@stdlib/OpenBLAS_jll`
  [05823500] OpenLibm_jll v0.8.1+0 `@stdlib/OpenLibm_jll`
  [83775a58] Zlib_jll v1.2.12+3 `@stdlib/Zlib_jll`
  [8e850b90] libblastrampoline_jll v5.1.1+0 `@stdlib/libblastrampoline_jll`
  [8e850ede] nghttp2_jll v1.48.0+0 `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll v17.4.0+0 `@stdlib/p7zip_jll`
        Info Packages marked with ⌅ have new versions available but cannot be upgraded.
Precompiling project...
  1 dependency successfully precompiled in 0 seconds. 74 already precompiled.
     Testing Running tests...
Starting tests for DataContainers
┌ Warning: `MvNormal(d::Int, σ::Real)` is deprecated, use `MvNormal(LinearAlgebra.Diagonal(FillArrays.Fill(σ ^ 2, d)))` instead.
│   caller = ip:0x0
└ @ Core :-1
Completed tests for DataContainers, 5 seconds elapsed
Starting tests for ParameterDistributions
Completed tests for ParameterDistributions, 15 seconds elapsed
Starting tests for Observations
Completed tests for Observations, 2 seconds elapsed
Starting tests for EnsembleKalmanProcess
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: IgnoreFailures.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
┌ Warning: New ensemble covariance determinant is less than 0.01 times its previous value.
│ Consider reducing the EK time step.
└ @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanInversion.jl:183
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: IgnoreFailures.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
┌ Warning: New ensemble covariance determinant is less than 0.01 times its previous value.
│ Consider reducing the EK time step.
└ @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanInversion.jl:183
┌ Warning: New ensemble covariance determinant is less than 0.01 times its previous value.
│ Consider reducing the EK time step.
└ @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanInversion.jl:183
┌ Warning: New ensemble covariance determinant is less than 0.01 times its previous value.
│ Consider reducing the EK time step.
└ @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanInversion.jl:183
┌ Warning: New ensemble covariance determinant is less than 0.01 times its previous value.
│ Consider reducing the EK time step.
└ @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanInversion.jl:183
┌ Warning: New ensemble covariance determinant is less than 0.01 times its previous value.
│ Consider reducing the EK time step.
└ @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanInversion.jl:183
┌ Warning: New ensemble covariance determinant is less than 0.01 times its previous value.
│ Consider reducing the EK time step.
└ @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanInversion.jl:183
┌ Warning: New ensemble covariance determinant is less than 0.01 times its previous value.
│ Consider reducing the EK time step.
└ @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanInversion.jl:183
┌ Warning: New ensemble covariance determinant is less than 0.01 times its previous value.
│ Consider reducing the EK time step.
└ @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanInversion.jl:183
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: IgnoreFailures.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: IgnoreFailures.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
[ Info: 1 particle failure(s) detected. Handler used: SampleSuccGauss.
SparseInversion: Error During Test at /Users/tsj/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/test/EnsembleKalmanProcess/runtests.jl:432
  Got exception outside of a @test
  TaskFailedException
  Stacktrace:
    [1] wait
      @ ./task.jl:345 [inlined]
    [2] threading_run(fun::EnsembleKalmanProcesses.var"#183#threadsfor_fun#37"{EnsembleKalmanProcesses.var"#183#threadsfor_fun#36#38"{EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64}}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, UnitRange{Int64}}}, static::Bool)
      @ Base.Threads ./threadingconstructs.jl:38
    [3] macro expansion
      @ ./threadingconstructs.jl:89 [inlined]
    [4] sparse_eki_update(ekp::EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64}}, u::Matrix{Float64}, g::Matrix{Float64}, y::Matrix{Float64}, obs_noise_cov::Matrix{Float64})
      @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/SparseEnsembleKalmanInversion.jl:154
    [5] (::EnsembleKalmanProcesses.var"#failsafe_update#31")(ekp::EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64}}, u::Matrix{Float64}, g::Matrix{Float64}, y::Matrix{Float64}, obs_noise_cov::Matrix{Float64}, failed_ens::Vector{Int64})
      @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/SparseEnsembleKalmanInversion.jl:54
    [6] update_ensemble!(ekp::EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64}}, g::Matrix{Float64}; cov_threshold::Float64, Δt_new::Nothing, deterministic_forward_map::Bool, failed_ens::Nothing)
      @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/SparseEnsembleKalmanInversion.jl:233
    [7] update_ensemble!(ekp::EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64}}, g::Matrix{Float64})
      @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/SparseEnsembleKalmanInversion.jl:192
    [8] macro expansion
      @ ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/test/EnsembleKalmanProcess/runtests.jl:493 [inlined]
    [9] macro expansion
      @ /Applications/Julia-1.8-arm.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [10] macro expansion
      @ ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/test/EnsembleKalmanProcess/runtests.jl:434 [inlined]
   [11] macro expansion
      @ /Applications/Julia-1.8-arm.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [12] top-level scope
      @ ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/test/EnsembleKalmanProcess/runtests.jl:15
   [13] include
      @ ./client.jl:476 [inlined]
   [14] macro expansion
      @ ./timing.jl:383 [inlined]
   [15] include_test(_module::String)
      @ Main ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/test/runtests.jl:11
   [16] macro expansion
      @ ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/test/runtests.jl:36 [inlined]
   [17] macro expansion
      @ /Applications/Julia-1.8-arm.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Test/src/Test.jl:1357 [inlined]
   [18] top-level scope
      @ ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/test/runtests.jl:17
   [19] include(fname::String)
      @ Base.MainInclude ./client.jl:476
   [20] top-level scope
      @ none:6
   [21] eval
      @ ./boot.jl:368 [inlined]
   [22] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:276
   [23] _start()
      @ Base ./client.jl:522

      nested task error: MethodError: no method matching SCS.Optimizer(; verbose=false)
      Closest candidates are:
        SCS.Optimizer() at ~/.julia/packages/SCS/JD7s9/src/MOI_wrapper/MOI_wrapper.jl:133 got unsupported keyword argument "verbose"
      Stacktrace:
        [1] (::EnsembleKalmanProcesses.var"#34#35")()
          @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/SparseEnsembleKalmanInversion.jl:103
        [2] _instantiate_and_check(optimizer_constructor::EnsembleKalmanProcesses.var"#34#35")
          @ MathOptInterface ~/.julia/packages/MathOptInterface/LqT2k/src/instantiate.jl:70
        [3] instantiate(optimizer_constructor::Function; with_bridge_type::Nothing)
          @ MathOptInterface ~/.julia/packages/MathOptInterface/LqT2k/src/instantiate.jl:125
        [4] instantiate
          @ ~/.julia/packages/MathOptInterface/LqT2k/src/instantiate.jl:121 [inlined]
        [5] solve!(problem::Convex.Problem{Float64}, optimizer_factory::Function; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
          @ Convex ~/.julia/packages/Convex/ukggP/src/solution.jl:269
        [6] solve!(problem::Convex.Problem{Float64}, optimizer_factory::Function)
          @ Convex ~/.julia/packages/Convex/ukggP/src/solution.jl:268
        [7] sparse_qp(ekp::EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64}}, v_j::Vector{Float64}, cov_vv_inv::Matrix{Float64}, H_u::Matrix{Float64}, H_g::Matrix{Float64}, y_j::Vector{Float64}; H_uc::Matrix{Float64})
          @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/SparseEnsembleKalmanInversion.jl:103
        [8] macro expansion
          @ ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/SparseEnsembleKalmanInversion.jl:156 [inlined]
        [9] (::EnsembleKalmanProcesses.var"#183#threadsfor_fun#37"{EnsembleKalmanProcesses.var"#183#threadsfor_fun#36#38"{EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64}}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
          @ EnsembleKalmanProcesses ./threadingconstructs.jl:84
       [10] #183#threadsfor_fun
          @ ./threadingconstructs.jl:51 [inlined]
       [11] (::Base.Threads.var"#1#2"{EnsembleKalmanProcesses.var"#183#threadsfor_fun#37"{EnsembleKalmanProcesses.var"#183#threadsfor_fun#36#38"{EnsembleKalmanProcess{Float64, Int64, SparseInversion{Float64}}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, Matrix{Float64}, UnitRange{Int64}}}, Int64})()
          @ Base.Threads ./threadingconstructs.jl:30
┌ Warning: Sample covariance matrix over ensemble is singular.
│  Appplying variance inflation.
└ @ EnsembleKalmanProcesses ~/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/src/EnsembleKalmanProcess.jl:301
Completed tests for EnsembleKalmanProcess, 12 seconds elapsed
Starting tests for Localizers
Completed tests for Localizers, 1 seconds elapsed
Starting tests for UQParameters
Completed tests for UQParameters, 3 seconds elapsed
Test Summary:                   | Pass  Error  Total   Time
EnsembleKalmanProcesses         |  830      1    831  35.9s
  DataContainers                |   12            12   1.0s
  ParameterDistributions        |  378           378  14.3s
  Observations                  |   26            26   1.8s
  EnsembleKalmanProcess         |  159      1    160  11.5s
    EnsembleKalmanSampler       |   13            13   2.5s
    EnsembleKalmanInversion     |   99            99   1.6s
    UnscentedKalmanInversion    |   36            36   1.5s
    SparseInversion             |    2      1      3   5.0s
    EnsembleKalmanProcess utils |    5             5   0.4s
  Localization                  |   21            21   0.6s
  UQParameters                  |  234           234   2.7s
ERROR: LoadError: Some tests did not pass: 830 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /Users/tsj/Documents/climate/clima/temp/EnsembleKalmanProcesses.jl/test/runtests.jl:16
ERROR: Package EnsembleKalmanProcesses errored during testing

[ilopezgp]

Hopefully some of these issues are resolved with #146, @jinlong83 is going to work on that.

[jinlong83]

Hopefully some of these issues are resolved with #146, @jinlong83 is going to work on that.

@tsj5 @ilopezgp The fix for #146 should be ready by the end of next week.

[ilopezgp]

Closing since we have updated to Julia 1.8