From 4685f8b2fad7a5bfaa7d373722e8046625197015 Mon Sep 17 00:00:00 2001 From: "Documenter.jl" Date: Mon, 30 Oct 2023 07:33:36 +0000 Subject: [PATCH] build based on d58560b8e --- dev/.documenter-siteinfo.json | 2 +- dev/comparisons/cppfortran/index.html | 2 +- dev/comparisons/matlab/index.html | 2 +- dev/comparisons/python/index.html | 2 +- dev/comparisons/r/index.html | 2 +- dev/getting_started/find_root/index.html | 2 +- .../first_optimization/index.html | 2 +- .../{15c441e5.svg => 561ccb23.svg} | 80 +- .../{4c1adcc8.svg => 732601a1.svg} | 136 +- .../{2bd5e1c9.svg => 880868d4.svg} | 64 +- .../{b01a8575.svg => ab225d80.svg} | 136 +- .../first_simulation/index.html | 6 +- dev/getting_started/fit_simulation/index.html | 2 +- .../getting_started/index.html | 2 +- dev/getting_started/installation/index.html | 2 +- dev/highlevels/array_libraries/index.html | 2 +- .../developer_documentation/index.html | 2 +- dev/highlevels/equation_solvers/index.html | 2 +- .../function_approximation/index.html | 2 +- dev/highlevels/implicit_layers/index.html | 2 +- dev/highlevels/interfaces/index.html | 2 +- dev/highlevels/inverse_problems/index.html | 2 +- dev/highlevels/learning_resources/index.html | 2 +- .../model_libraries_and_importers/index.html | 2 +- dev/highlevels/modeling_languages/index.html | 2 +- dev/highlevels/numerical_utilities/index.html | 2 +- dev/highlevels/parameter_analysis/index.html | 2 +- .../index.html | 2 +- dev/highlevels/plots_visualization/index.html | 2 +- dev/highlevels/symbolic_learning/index.html | 2 +- dev/highlevels/symbolic_tools/index.html | 2 +- .../uncertainty_quantification/index.html | 2 +- dev/index.html | 6 +- dev/overview/index.html | 2 +- dev/showcase/bayesian_neural_ode/3171e8ff.svg | 316 ---- dev/showcase/bayesian_neural_ode/45b3565a.svg | 389 ---- dev/showcase/bayesian_neural_ode/5447e78c.svg | 328 ++++ dev/showcase/bayesian_neural_ode/5fc0d793.svg | 731 -------- dev/showcase/bayesian_neural_ode/d5f26c8c.svg | 389 ++++ dev/showcase/bayesian_neural_ode/da41ed8f.svg | 169 -- dev/showcase/bayesian_neural_ode/de435bca.svg | 169 ++ dev/showcase/bayesian_neural_ode/e38ccbe5.svg | 731 ++++++++ dev/showcase/bayesian_neural_ode/index.html | 42 +- .../blackhole/{6e6831b7.svg => 141a6e2a.svg} | 72 +- .../blackhole/{92ef0486.svg => 1efc1e48.svg} | 68 +- .../blackhole/{104f826e.svg => 6b039505.svg} | 574 +++--- .../blackhole/{7bdbe226.svg => 84cb8445.svg} | 72 +- .../blackhole/{8ef4842d.svg => f90430a5.svg} | 80 +- dev/showcase/blackhole/index.html | 10 +- dev/showcase/brusselator/index.html | 2 +- .../gpu_spde/{761e69dc.svg => 11214966.svg} | 240 +-- .../gpu_spde/{5a053849.svg => 35cbe00e.svg} | 252 +-- dev/showcase/gpu_spde/5b152400.svg | 1565 +++++++++++++++++ dev/showcase/gpu_spde/d5f23e83.svg | 1534 ---------------- dev/showcase/gpu_spde/index.html | 62 +- .../massively_parallel_gpu/index.html | 2 +- .../{534623c5.svg => 2c4c4ce0.svg} | 76 +- .../{2b5bbf8a.svg => 489ac7be.svg} | 44 +- .../{5fe4f215.svg => 86c321bc.svg} | 320 ++-- .../{8ebb7bff.svg => a2c2395e.svg} | 68 +- .../{c97f6698.svg => a583b620.svg} | 156 +- .../{604b577a.svg => b189172d.svg} | 156 +- .../{3d3d4a16.svg => bd202645.svg} | 80 +- .../{f774f765.svg => ca401a68.svg} | 264 +-- .../{8b443a7a.svg => d1fa5e21.svg} | 76 +- dev/showcase/missing_physics/index.html | 28 +- .../ode_types/{5a7e9cc0.svg => 36f8da94.svg} | 244 +-- .../ode_types/{174c148e.svg => 4ccd3ad4.svg} | 472 ++--- .../ode_types/{dea91176.svg => 7a5651df.svg} | 720 ++++---- .../ode_types/{234de620.svg => 93f3d2e1.svg} | 72 +- .../ode_types/{9149ebbe.svg => cd59a14e.svg} | 892 +++++----- dev/showcase/ode_types/index.html | 8 +- .../{c47d2005.svg => 0a47808e.svg} | 70 +- .../{3a730523.svg => 10a893d2.svg} | 1072 +++++------ .../200aeb10.svg | 150 -- .../39eff794.svg | 150 -- .../{847d49d1.svg => 3b5a696d.svg} | 64 +- .../{24fe6a59.svg => 4b301c1c.svg} | 70 +- .../{251a8272.svg => 9136ceba.svg} | 60 +- .../ac1faa55.svg | 150 ++ .../c16032af.svg | 150 ++ .../{3f13945b.svg => d739c0f0.svg} | 1072 +++++------ .../optimization_under_uncertainty/index.html | 16 +- dev/showcase/pinngpu/index.html | 10 +- dev/showcase/showcase/index.html | 2 +- .../{5e671764.svg => 1c89b5cd.svg} | 80 +- .../{257fb233.svg => 244e1c76.svg} | 80 +- .../{c175b5fa.svg => c6d9db61.svg} | 80 +- dev/showcase/symbolic_analysis/index.html | 8 +- 89 files changed, 7642 insertions(+), 7599 deletions(-) rename dev/getting_started/first_simulation/{15c441e5.svg => 561ccb23.svg} (91%) rename dev/getting_started/first_simulation/{4c1adcc8.svg => 732601a1.svg} (91%) rename dev/getting_started/first_simulation/{2bd5e1c9.svg => 880868d4.svg} (90%) rename dev/getting_started/first_simulation/{b01a8575.svg => ab225d80.svg} (91%) delete mode 100644 dev/showcase/bayesian_neural_ode/3171e8ff.svg delete mode 100644 dev/showcase/bayesian_neural_ode/45b3565a.svg create mode 100644 dev/showcase/bayesian_neural_ode/5447e78c.svg delete mode 100644 dev/showcase/bayesian_neural_ode/5fc0d793.svg create mode 100644 dev/showcase/bayesian_neural_ode/d5f26c8c.svg delete mode 100644 dev/showcase/bayesian_neural_ode/da41ed8f.svg create mode 100644 dev/showcase/bayesian_neural_ode/de435bca.svg create mode 100644 dev/showcase/bayesian_neural_ode/e38ccbe5.svg rename dev/showcase/blackhole/{6e6831b7.svg => 141a6e2a.svg} (94%) rename dev/showcase/blackhole/{92ef0486.svg => 1efc1e48.svg} (93%) rename dev/showcase/blackhole/{104f826e.svg => 6b039505.svg} (76%) rename dev/showcase/blackhole/{7bdbe226.svg => 84cb8445.svg} (88%) rename dev/showcase/blackhole/{8ef4842d.svg => f90430a5.svg} (86%) rename dev/showcase/gpu_spde/{761e69dc.svg => 11214966.svg} (88%) rename dev/showcase/gpu_spde/{5a053849.svg => 35cbe00e.svg} (87%) create mode 100644 dev/showcase/gpu_spde/5b152400.svg delete mode 100644 dev/showcase/gpu_spde/d5f23e83.svg rename dev/showcase/missing_physics/{534623c5.svg => 2c4c4ce0.svg} (84%) rename dev/showcase/missing_physics/{2b5bbf8a.svg => 489ac7be.svg} (97%) rename dev/showcase/missing_physics/{5fe4f215.svg => 86c321bc.svg} (85%) rename dev/showcase/missing_physics/{8ebb7bff.svg => a2c2395e.svg} (88%) rename dev/showcase/missing_physics/{c97f6698.svg => a583b620.svg} (80%) rename dev/showcase/missing_physics/{604b577a.svg => b189172d.svg} (79%) rename dev/showcase/missing_physics/{3d3d4a16.svg => bd202645.svg} (83%) rename dev/showcase/missing_physics/{f774f765.svg => ca401a68.svg} (82%) rename dev/showcase/missing_physics/{8b443a7a.svg => d1fa5e21.svg} (90%) rename dev/showcase/ode_types/{5a7e9cc0.svg => 36f8da94.svg} (79%) rename dev/showcase/ode_types/{174c148e.svg => 4ccd3ad4.svg} (69%) rename dev/showcase/ode_types/{dea91176.svg => 7a5651df.svg} (67%) rename dev/showcase/ode_types/{234de620.svg => 93f3d2e1.svg} (88%) rename dev/showcase/ode_types/{9149ebbe.svg => cd59a14e.svg} (69%) rename dev/showcase/optimization_under_uncertainty/{c47d2005.svg => 0a47808e.svg} (89%) rename dev/showcase/optimization_under_uncertainty/{3a730523.svg => 10a893d2.svg} (58%) delete mode 100644 dev/showcase/optimization_under_uncertainty/200aeb10.svg delete mode 100644 dev/showcase/optimization_under_uncertainty/39eff794.svg rename dev/showcase/optimization_under_uncertainty/{847d49d1.svg => 3b5a696d.svg} (90%) rename dev/showcase/optimization_under_uncertainty/{24fe6a59.svg => 4b301c1c.svg} (82%) rename dev/showcase/optimization_under_uncertainty/{251a8272.svg => 9136ceba.svg} (90%) create mode 100644 dev/showcase/optimization_under_uncertainty/ac1faa55.svg create mode 100644 dev/showcase/optimization_under_uncertainty/c16032af.svg rename dev/showcase/optimization_under_uncertainty/{3f13945b.svg => d739c0f0.svg} (62%) rename dev/showcase/symbolic_analysis/{5e671764.svg => 1c89b5cd.svg} (98%) rename dev/showcase/symbolic_analysis/{257fb233.svg => 244e1c76.svg} (96%) rename dev/showcase/symbolic_analysis/{c175b5fa.svg => c6d9db61.svg} (98%) diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json index 38c442c21aa..0234d8d2057 100644 --- a/dev/.documenter-siteinfo.json +++ b/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.9.3","generation_timestamp":"2023-10-30T04:38:04","documenter_version":"1.1.2"}} \ No newline at end of file +{"documenter":{"julia_version":"1.9.3","generation_timestamp":"2023-10-30T07:27:49","documenter_version":"1.1.2"}} \ No newline at end of file diff --git a/dev/comparisons/cppfortran/index.html b/dev/comparisons/cppfortran/index.html index 28e33daa383..ca37ba7f850 100644 --- a/dev/comparisons/cppfortran/index.html +++ b/dev/comparisons/cppfortran/index.html @@ -1,2 +1,2 @@ -Getting Started with Julia's SciML for the C++/Fortran User · Overview of Julia's SciML
GitHub

Getting Started with Julia's SciML for the C++/Fortran User

You don't need help if you're a Fortran guru. I'm just kidding, you're not a Lisp developer. If you're coming from C++ or Fortran, you may be familiar with high-performance computing environments similar to SciML, such as PETSc, Trilinos, or Sundials. The following are some points to help the transition.

Why SciML? High-Level Workflow Reasons

If you're coming from “hardcore” C++/Fortran computing environments, some things to check out with Julia's SciML are:

  • Interactivity - use the interactive REPL to easily investigate numerical details.
  • Metaprogramming performance tools - tools like LoopVectorization.jl can be used to generate faster code than even some of the most hand-optimized C++/Fortran code. Current benchmarks show this SIMD-optimized Julia code outperforming OpenBLAS and MKL BLAS implementations in many performance regimes.
  • Symbolic modeling languages - writing models by hand can leave a lot of performance on the table. Using high-level modeling tools like ModelingToolkit can automate symbolic simplifications, which improve the stability and performance of numerical solvers. On complex models, even the best handwritten C++/Fortran code is orders of magnitude behind the code that symbolic tearing algorithms can achieve!
  • Composable Library Components - In C++/Fortran environments, every package feels like a silo. Arrays made for PETSc cannot easily be used in Trilinos, and converting Sundials NVector outputs to DataFrames for post-simulation data processing is a process itself. The Julia SciML environment embraces interoperability. Don't wait for SciML to do it: by using generic coding with JIT compilation, these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • Wrappers to the Libraries You Know and Trust - Moving to SciML does not have to be a quick transition. SciML has extensive wrappers to many widely-used classical solver environments such as SUNDIALS and Hairer's classic Fortran ODE solvers (dopri5, dop853, etc.). Using these wrapped solvers is painless and can be swapped in for the Julia versions with one line of code. This gives you a way to incrementally adopt new features/methods while retaining the older pieces you know and trust.
  • Don't Start from Scratch - SciML builds on the extensive Base library of Julia, and thus grows and improves with every update to the language. With hundreds of monthly contributors to SciML and hundreds of monthly contributors to Julia, SciML is one of the most actively developed open-source scientific computing ecosystems out there!
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, Sundials/Hairer in purple/red represent C++/Fortrans most commonly used solvers:

Why SciML? Some Technical Details

Let's face the facts, in the open benchmarks the pure-Julia solvers tend to outperform the classic “best” C++ and Fortran solvers in almost every example (with a few notable exceptions). But why?

The answer is two-fold: Julia is as fast as C++/Fortran, and the algorithms are what matter.

Julia is as Fast as C++/Fortran

While Julia code looks high level like Python or MATLAB, its performance is on par with C++ and Fortran. At a technical level, when Julia code is type-stable, i.e. that the types that are returned from a function are deducible at compile-time from the types that go into a function, then Julia can optimize it as much as C++ or Fortran by automatically devirtualizing all dynamic behavior and compile-time optimizing the quasi-static code. This is not an empirical statement, it's a provable type-theoretic result. The resulting compiler used on the resulting quasi-static representation is LLVM, the same optimizing compiler used by clang and LFortran.

For more details on how Julia code is optimized and how to optimize your own Julia code, check out this chapter from the SciML Book.

SciML's Julia Algorithms Have Performance Advantages in Many Common Regimes

There are many ways which Julia's algorithms achieve performance advantages. Some facts to highlight include:

Let's Dig Deep Into One Case: Adjoints of ODEs for Solving Inverse Problems

To really highlight how JIT compilation and automatic differentiation integration can change algorithms, let's look at the problem of differentiating an ODE solver. As is derived and discussed in detail at a seminar with the American Statistical Association, there are many ways to implement well-known “adjoint” methods which are required for performance. Each has different stability and performance trade-offs, and Julia's SciML is the only system to systemically offer all of the trade-off options. In many cases, using analytical adjoints of a solver is not advised due to performance reasons, with the trade-off described in detail here. Likewise, even when analytical adjoints are used, it turns out that for general nonlinear equations there is a trick which uses automatic differentiation in the construction of the analytical adjoint to improve its performance. As demonstrated in this publication, this can lead to about 2-3 orders of magnitude performance improvements. These AD-enhanced adjoints are showcased as the seeding methods in this plot:

Unless one directly defines special “vjp” functions, this is how the Julia SciML methods achieve orders of magnitude performance advantages over CVODES's adjoints and PETSC's TS-adjoint.

Moral of the story, even there are many reasons to use automatic differentiation of a solver, and even if an analytical adjoint rule is used for some specific performance reason, that analytical expression can often times be accelerated by orders of magnitude itself by embedding some form of automatic differentiation into it. This is just one algorithm of many which are optimized in this fashion.

+Getting Started with Julia's SciML for the C++/Fortran User · Overview of Julia's SciML
GitHub

Getting Started with Julia's SciML for the C++/Fortran User

You don't need help if you're a Fortran guru. I'm just kidding, you're not a Lisp developer. If you're coming from C++ or Fortran, you may be familiar with high-performance computing environments similar to SciML, such as PETSc, Trilinos, or Sundials. The following are some points to help the transition.

Why SciML? High-Level Workflow Reasons

If you're coming from “hardcore” C++/Fortran computing environments, some things to check out with Julia's SciML are:

  • Interactivity - use the interactive REPL to easily investigate numerical details.
  • Metaprogramming performance tools - tools like LoopVectorization.jl can be used to generate faster code than even some of the most hand-optimized C++/Fortran code. Current benchmarks show this SIMD-optimized Julia code outperforming OpenBLAS and MKL BLAS implementations in many performance regimes.
  • Symbolic modeling languages - writing models by hand can leave a lot of performance on the table. Using high-level modeling tools like ModelingToolkit can automate symbolic simplifications, which improve the stability and performance of numerical solvers. On complex models, even the best handwritten C++/Fortran code is orders of magnitude behind the code that symbolic tearing algorithms can achieve!
  • Composable Library Components - In C++/Fortran environments, every package feels like a silo. Arrays made for PETSc cannot easily be used in Trilinos, and converting Sundials NVector outputs to DataFrames for post-simulation data processing is a process itself. The Julia SciML environment embraces interoperability. Don't wait for SciML to do it: by using generic coding with JIT compilation, these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • Wrappers to the Libraries You Know and Trust - Moving to SciML does not have to be a quick transition. SciML has extensive wrappers to many widely-used classical solver environments such as SUNDIALS and Hairer's classic Fortran ODE solvers (dopri5, dop853, etc.). Using these wrapped solvers is painless and can be swapped in for the Julia versions with one line of code. This gives you a way to incrementally adopt new features/methods while retaining the older pieces you know and trust.
  • Don't Start from Scratch - SciML builds on the extensive Base library of Julia, and thus grows and improves with every update to the language. With hundreds of monthly contributors to SciML and hundreds of monthly contributors to Julia, SciML is one of the most actively developed open-source scientific computing ecosystems out there!
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, Sundials/Hairer in purple/red represent C++/Fortrans most commonly used solvers:

Why SciML? Some Technical Details

Let's face the facts, in the open benchmarks the pure-Julia solvers tend to outperform the classic “best” C++ and Fortran solvers in almost every example (with a few notable exceptions). But why?

The answer is two-fold: Julia is as fast as C++/Fortran, and the algorithms are what matter.

Julia is as Fast as C++/Fortran

While Julia code looks high level like Python or MATLAB, its performance is on par with C++ and Fortran. At a technical level, when Julia code is type-stable, i.e. that the types that are returned from a function are deducible at compile-time from the types that go into a function, then Julia can optimize it as much as C++ or Fortran by automatically devirtualizing all dynamic behavior and compile-time optimizing the quasi-static code. This is not an empirical statement, it's a provable type-theoretic result. The resulting compiler used on the resulting quasi-static representation is LLVM, the same optimizing compiler used by clang and LFortran.

For more details on how Julia code is optimized and how to optimize your own Julia code, check out this chapter from the SciML Book.

SciML's Julia Algorithms Have Performance Advantages in Many Common Regimes

There are many ways which Julia's algorithms achieve performance advantages. Some facts to highlight include:

Let's Dig Deep Into One Case: Adjoints of ODEs for Solving Inverse Problems

To really highlight how JIT compilation and automatic differentiation integration can change algorithms, let's look at the problem of differentiating an ODE solver. As is derived and discussed in detail at a seminar with the American Statistical Association, there are many ways to implement well-known “adjoint” methods which are required for performance. Each has different stability and performance trade-offs, and Julia's SciML is the only system to systemically offer all of the trade-off options. In many cases, using analytical adjoints of a solver is not advised due to performance reasons, with the trade-off described in detail here. Likewise, even when analytical adjoints are used, it turns out that for general nonlinear equations there is a trick which uses automatic differentiation in the construction of the analytical adjoint to improve its performance. As demonstrated in this publication, this can lead to about 2-3 orders of magnitude performance improvements. These AD-enhanced adjoints are showcased as the seeding methods in this plot:

Unless one directly defines special “vjp” functions, this is how the Julia SciML methods achieve orders of magnitude performance advantages over CVODES's adjoints and PETSC's TS-adjoint.

Moral of the story, even there are many reasons to use automatic differentiation of a solver, and even if an analytical adjoint rule is used for some specific performance reason, that analytical expression can often times be accelerated by orders of magnitude itself by embedding some form of automatic differentiation into it. This is just one algorithm of many which are optimized in this fashion.

diff --git a/dev/comparisons/matlab/index.html b/dev/comparisons/matlab/index.html index afff31a94fe..6847e3f43ac 100644 --- a/dev/comparisons/matlab/index.html +++ b/dev/comparisons/matlab/index.html @@ -1,2 +1,2 @@ -Getting Started with Julia's SciML for the MATLAB User · Overview of Julia's SciML
GitHub

Getting Started with Julia's SciML for the MATLAB User

If you're a MATLAB user who has looked into Julia for some performance improvements, you may have noticed that the standard library does not have all of the “batteries” included with a base MATLAB installation. Where's the ODE solver? Where's fmincon and fsolve? Those scientific computing functionalities are the pieces provided by the Julia SciML ecosystem!

Why SciML? High-Level Workflow Reasons

  • Performance - The key reason people are moving from MATLAB to Julia's SciML in droves is performance. Even simple ODE solvers are much faster!, demonstrating orders of magnitude performance improvements for differential equations, nonlinear solving, optimization, and more. And the performance advantages continue to grow as more complex algorithms are required.
  • Julia is quick to learn from MATLAB - Most ODE codes can be translated in a few minutes. If you need help, check out the QuantEcon MATLAB-Python-Julia Cheat Sheet.
  • Package Management and Versioning - Julia's package manager takes care of dependency management, testing, and continuous delivery in order to make the installation and maintenance process smoother. For package users, this means it's easier to get packages with complex functionality in your hands.
  • Free and Open Source - If you want to know how things are being computed, just look at our GitHub organization. Lots of individuals use Julia's SciML to research how the algorithms actually work because of how accessible and tweakable the ecosystem is!
  • Composable Library Components - In MATLAB environments, every package feels like a silo. Functions made for one file exchange library cannot easily compose with another. SciML's generic coding with JIT compilation these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, MATLAB in orange represents MATLAB's most commonly used solvers:

Need a case study?

Check out this talk from NASA Scientists getting a 15,000x acceleration by switching from Simulink to Julia's ModelingToolkit!

Need Help Translating from MATLAB to Julia?

The following resources can be particularly helpful when adopting Julia for SciML for the first time:

MATLAB to Julia SciML Functionality Translations

The following chart will help you get quickly acquainted with Julia's SciML Tools:

MATLAB FunctionSciML-Supported Julia packages
plotPlots, Makie
sparseSparseArrays
interp1DataInterpolations
\, gmres, cgLinearSolve
fsolveNonlinearSolve
quadIntegrals
fminconOptimization
odeXXDifferentialEquations
ode45Tsit5
ode113VCABM
ode23sRosenbrock23
ode15sQNDF or FBDF
ode15iIDA
bvp4c and bvp5cDifferentialEquations
Simulink, SimscapeModelingToolkit
fftFFTW
chebfunApproxFun
+Getting Started with Julia's SciML for the MATLAB User · Overview of Julia's SciML
GitHub

Getting Started with Julia's SciML for the MATLAB User

If you're a MATLAB user who has looked into Julia for some performance improvements, you may have noticed that the standard library does not have all of the “batteries” included with a base MATLAB installation. Where's the ODE solver? Where's fmincon and fsolve? Those scientific computing functionalities are the pieces provided by the Julia SciML ecosystem!

Why SciML? High-Level Workflow Reasons

  • Performance - The key reason people are moving from MATLAB to Julia's SciML in droves is performance. Even simple ODE solvers are much faster!, demonstrating orders of magnitude performance improvements for differential equations, nonlinear solving, optimization, and more. And the performance advantages continue to grow as more complex algorithms are required.
  • Julia is quick to learn from MATLAB - Most ODE codes can be translated in a few minutes. If you need help, check out the QuantEcon MATLAB-Python-Julia Cheat Sheet.
  • Package Management and Versioning - Julia's package manager takes care of dependency management, testing, and continuous delivery in order to make the installation and maintenance process smoother. For package users, this means it's easier to get packages with complex functionality in your hands.
  • Free and Open Source - If you want to know how things are being computed, just look at our GitHub organization. Lots of individuals use Julia's SciML to research how the algorithms actually work because of how accessible and tweakable the ecosystem is!
  • Composable Library Components - In MATLAB environments, every package feels like a silo. Functions made for one file exchange library cannot easily compose with another. SciML's generic coding with JIT compilation these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, MATLAB in orange represents MATLAB's most commonly used solvers:

Need a case study?

Check out this talk from NASA Scientists getting a 15,000x acceleration by switching from Simulink to Julia's ModelingToolkit!

Need Help Translating from MATLAB to Julia?

The following resources can be particularly helpful when adopting Julia for SciML for the first time:

MATLAB to Julia SciML Functionality Translations

The following chart will help you get quickly acquainted with Julia's SciML Tools:

MATLAB FunctionSciML-Supported Julia packages
plotPlots, Makie
sparseSparseArrays
interp1DataInterpolations
\, gmres, cgLinearSolve
fsolveNonlinearSolve
quadIntegrals
fminconOptimization
odeXXDifferentialEquations
ode45Tsit5
ode113VCABM
ode23sRosenbrock23
ode15sQNDF or FBDF
ode15iIDA
bvp4c and bvp5cDifferentialEquations
Simulink, SimscapeModelingToolkit
fftFFTW
chebfunApproxFun
diff --git a/dev/comparisons/python/index.html b/dev/comparisons/python/index.html index 1deffd9d13d..1926c2dd9b2 100644 --- a/dev/comparisons/python/index.html +++ b/dev/comparisons/python/index.html @@ -1,2 +1,2 @@ -Getting Started with Julia's SciML for the Python User · Overview of Julia's SciML
GitHub

Getting Started with Julia's SciML for the Python User

If you're a Python user who has looked into Julia, you're probably wondering what is the equivalent to SciPy is. And you found it: it's the SciML ecosystem! To a Python developer, SciML is SciPy, but with the high-performance GPU, capabilities of PyTorch, and neural network capabilities, all baked right in. With SciML, there is no “separate world” of machine learning sublanguages: there is just one cohesive package ecosystem.

Why SciML? High-Level Workflow Reasons

  • Performance - The key reason people are moving from SciPy to Julia's SciML in droves is performance. Even simple ODE solvers are much faster!, demonstrating orders of magnitude performance improvements for differential equations, nonlinear solving, optimization, and more. And the performance advantages continue to grow as more complex algorithms are required.
  • Package Management and Versioning - Julia's package manager takes care of dependency management, testing, and continuous delivery in order to make the installation and maintenance process smoother. For package users, this means it's easier to get packages with complex functionality in your hands.
  • Composable Library Components - In Python environments, every package feels like a silo. Functions made for one file exchange library cannot easily compose with another. SciML's generic coding with JIT compilation these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, SciPy in yellow represents Python's most commonly used solvers:

Need Help Translating from Python to Julia?

The following resources can be particularly helpful when adopting Julia for SciML for the first time:

Python to Julia SciML Functionality Translations

The following chart will help you get quickly acquainted with Julia's SciML Tools:

Workflow ElementSciML-Supported Julia packages
MatplotlibPlots, Makie
scipy.specialSpecialFunctions
scipy.linalg.solveLinearSolve
scipy.integrateIntegrals
scipy.optimizeOptimization
scipy.optimize.fsolveNonlinearSolve
scipy.interpolateDataInterpolations
scipy.fftFFTW
scipy.linalgJulia's Built-In Linear Algebra
scipy.sparseSparseArrays, ARPACK
odeint/solve_ivpDifferentialEquations
scipy.integrate.solve_bvpBoundary-value problem
PyTorchFlux, Lux
gillespy2Catalyst, JumpProcesses
scipy.optimize.approx_fprimeFiniteDiff
autogradForwardDiff*, Enzyme*, DiffEqSensitivity
StanTuring
sympySymbolics

Why is Differentiable Programming Important for Scientific Computing?

Check out this blog post that goes into detail on how training neural networks in tandem with simulation improves performance by orders of magnitude. But can't you use analytical adjoint definitions? You can, but there are tricks to mix automatic differentiation into the adjoint definitions for a few orders of magnitude improvement too, as explained in this blog post.

These facts, along with many others, compose to algorithmic improvements with the implementation improvements, which leads to orders of magnitude improvements!

+Getting Started with Julia's SciML for the Python User · Overview of Julia's SciML
GitHub

Getting Started with Julia's SciML for the Python User

If you're a Python user who has looked into Julia, you're probably wondering what is the equivalent to SciPy is. And you found it: it's the SciML ecosystem! To a Python developer, SciML is SciPy, but with the high-performance GPU, capabilities of PyTorch, and neural network capabilities, all baked right in. With SciML, there is no “separate world” of machine learning sublanguages: there is just one cohesive package ecosystem.

Why SciML? High-Level Workflow Reasons

  • Performance - The key reason people are moving from SciPy to Julia's SciML in droves is performance. Even simple ODE solvers are much faster!, demonstrating orders of magnitude performance improvements for differential equations, nonlinear solving, optimization, and more. And the performance advantages continue to grow as more complex algorithms are required.
  • Package Management and Versioning - Julia's package manager takes care of dependency management, testing, and continuous delivery in order to make the installation and maintenance process smoother. For package users, this means it's easier to get packages with complex functionality in your hands.
  • Composable Library Components - In Python environments, every package feels like a silo. Functions made for one file exchange library cannot easily compose with another. SciML's generic coding with JIT compilation these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, SciPy in yellow represents Python's most commonly used solvers:

Need Help Translating from Python to Julia?

The following resources can be particularly helpful when adopting Julia for SciML for the first time:

Python to Julia SciML Functionality Translations

The following chart will help you get quickly acquainted with Julia's SciML Tools:

Workflow ElementSciML-Supported Julia packages
MatplotlibPlots, Makie
scipy.specialSpecialFunctions
scipy.linalg.solveLinearSolve
scipy.integrateIntegrals
scipy.optimizeOptimization
scipy.optimize.fsolveNonlinearSolve
scipy.interpolateDataInterpolations
scipy.fftFFTW
scipy.linalgJulia's Built-In Linear Algebra
scipy.sparseSparseArrays, ARPACK
odeint/solve_ivpDifferentialEquations
scipy.integrate.solve_bvpBoundary-value problem
PyTorchFlux, Lux
gillespy2Catalyst, JumpProcesses
scipy.optimize.approx_fprimeFiniteDiff
autogradForwardDiff*, Enzyme*, DiffEqSensitivity
StanTuring
sympySymbolics

Why is Differentiable Programming Important for Scientific Computing?

Check out this blog post that goes into detail on how training neural networks in tandem with simulation improves performance by orders of magnitude. But can't you use analytical adjoint definitions? You can, but there are tricks to mix automatic differentiation into the adjoint definitions for a few orders of magnitude improvement too, as explained in this blog post.

These facts, along with many others, compose to algorithmic improvements with the implementation improvements, which leads to orders of magnitude improvements!

diff --git a/dev/comparisons/r/index.html b/dev/comparisons/r/index.html index 537dea139a5..788febf30a7 100644 --- a/dev/comparisons/r/index.html +++ b/dev/comparisons/r/index.html @@ -1,2 +1,2 @@ -Getting Started with Julia's SciML for the R User · Overview of Julia's SciML
GitHub

Getting Started with Julia's SciML for the R User

If you're an R user who has looked into Julia, you're probably wondering where all of the scientific computing packages are. How do I solve ODEs? Solve f(x)=0 for x? Etc. SciML is the ecosystem for doing this with Julia.

Why SciML? High-Level Workflow Reasons

  • Performance - The key reason people are moving from R to Julia's SciML in droves is performance. Even simple ODE solvers are much faster!, demonstrating orders of magnitude performance improvements for differential equations, nonlinear solving, optimization, and more. And the performance advantages continue to grow as more complex algorithms are required.
  • Composable Library Components - In R environments, every package feels like a silo. Functions made for one file exchange library cannot easily compose with another. SciML's generic coding with JIT compilation these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • A Global Harmonious Documentation for Scientific Computing - R's documentation for scientific computing is scattered in a bunch of individual packages where the developers do not talk to each other! This not only leads to documentation differences, but also “style” differences: one package uses tol while the other uses atol. With Julia's SciML, the whole ecosystem is considered together, and inconsistencies are handled at the global level. The goal is to be working in one environment with one language.
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, deSolve in blue represents R's most commonly used solver:

Need Help Translating from R to Julia?

The following resources can be particularly helpful when adopting Julia for SciML for the first time:

R to Julia SciML Functionality Translations

The following chart will help you get quickly acquainted with Julia's SciML Tools:

R Function/PackageSciML-Supported Julia packages
data.frameDataFrames
plotPlots, Makie
ggplot2AlgebraOfGraphics
deSolveDifferentialEquations
StanTuring

Want to See the Power of Julia?

Check out this R-Bloggers blog post on diffeqr, a package which uses ModelingToolkit to translate R code to Julia, and achieves 350x acceleration over R's popular deSolve ODE solver package. But when the solve is done purely in Julia, it achieves 2777x acceleration over deSolve!

+Getting Started with Julia's SciML for the R User · Overview of Julia's SciML
GitHub

Getting Started with Julia's SciML for the R User

If you're an R user who has looked into Julia, you're probably wondering where all of the scientific computing packages are. How do I solve ODEs? Solve f(x)=0 for x? Etc. SciML is the ecosystem for doing this with Julia.

Why SciML? High-Level Workflow Reasons

  • Performance - The key reason people are moving from R to Julia's SciML in droves is performance. Even simple ODE solvers are much faster!, demonstrating orders of magnitude performance improvements for differential equations, nonlinear solving, optimization, and more. And the performance advantages continue to grow as more complex algorithms are required.
  • Composable Library Components - In R environments, every package feels like a silo. Functions made for one file exchange library cannot easily compose with another. SciML's generic coding with JIT compilation these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • A Global Harmonious Documentation for Scientific Computing - R's documentation for scientific computing is scattered in a bunch of individual packages where the developers do not talk to each other! This not only leads to documentation differences, but also “style” differences: one package uses tol while the other uses atol. With Julia's SciML, the whole ecosystem is considered together, and inconsistencies are handled at the global level. The goal is to be working in one environment with one language.
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, deSolve in blue represents R's most commonly used solver:

Need Help Translating from R to Julia?

The following resources can be particularly helpful when adopting Julia for SciML for the first time:

R to Julia SciML Functionality Translations

The following chart will help you get quickly acquainted with Julia's SciML Tools:

R Function/PackageSciML-Supported Julia packages
data.frameDataFrames
plotPlots, Makie
ggplot2AlgebraOfGraphics
deSolveDifferentialEquations
StanTuring

Want to See the Power of Julia?

Check out this R-Bloggers blog post on diffeqr, a package which uses ModelingToolkit to translate R code to Julia, and achieves 350x acceleration over R's popular deSolve ODE solver package. But when the solve is done purely in Julia, it achieves 2777x acceleration over deSolve!

diff --git a/dev/getting_started/find_root/index.html b/dev/getting_started/find_root/index.html index 7d93ea6ad7f..a3f2b1ae125 100644 --- a/dev/getting_started/find_root/index.html +++ b/dev/getting_started/find_root/index.html @@ -69,4 +69,4 @@ 0.0 0.0 0.0

Step 5: Analyze the Solution

Now let's check out the solution. First of all, what kind of thing is the sol? We can see that by asking for its type:

typeof(sol)
SciMLBase.NonlinearSolution{Float64, 1, Vector{Float64}, Vector{Float64}, SciMLBase.NonlinearProblem{Vector{Float64}, true, Vector{Float64}, SciMLBase.NonlinearFunction{true, SciMLBase.FullSpecialize, ModelingToolkit.var"#f#733"{RuntimeGeneratedFunctions.RuntimeGeneratedFunction{(:ˍ₋arg1, :ˍ₋arg2), ModelingToolkit.var"#_RGF_ModTag", ModelingToolkit.var"#_RGF_ModTag", (0x892348f8, 0xa22abdbe, 0xa285f9d8, 0x95831967, 0xeaff7cf7), Nothing}, RuntimeGeneratedFunctions.RuntimeGeneratedFunction{(:ˍ₋out, :ˍ₋arg1, :ˍ₋arg2), ModelingToolkit.var"#_RGF_ModTag", ModelingToolkit.var"#_RGF_ModTag", (0x2a5931a9, 0x19d9b662, 0x2773c77d, 0x8f1eafb9, 0xd485c91b), Nothing}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Vector{Symbol}, ModelingToolkit.var"#generated_observed#736"{ModelingToolkit.NonlinearSystem, Dict{Any, Any}}, Nothing, ModelingToolkit.NonlinearSystem, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardNonlinearProblem}, NonlinearSolve.NewtonRaphson{0, true, Val{:forward}, Nothing, typeof(NonlinearSolve.DEFAULT_PRECS), true, nothing}, Nothing, Nothing, SciMLBase.NLStats}

From this, we can see that it is an NonlinearSolution. We can see the documentation for how to use the NonlinearSolution by checking the NonlinearSolve.jl solution type page. For example, the solution is stored as .u. What is the solution to our nonlinear system, and what is the final residual value? We can check it as follows:

# Analyze the solution
-@show sol.u, sol.resid
([0.0, 0.0, 0.0], [0.0, 0.0, 0.0])
+@show sol.u, sol.resid
([0.0, 0.0, 0.0], [0.0, 0.0, 0.0])
diff --git a/dev/getting_started/first_optimization/index.html b/dev/getting_started/first_optimization/index.html index 685098e9a5b..d463c1dcdee 100644 --- a/dev/getting_started/first_optimization/index.html +++ b/dev/getting_started/first_optimization/index.html @@ -28,4 +28,4 @@ sol = solve(prob, NLopt.LD_LBFGS())
u: 2-element Vector{Float64}:
  1.0
  1.0

Step 4: Analyze the Solution

Now let's check out the solution. First of all, what kind of thing is the sol? We can see that by asking for its type:

typeof(sol)
SciMLBase.OptimizationSolution{Float64, 1, Vector{Float64}, Optimization.OptimizationCache{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), OptimizationForwardDiffExt.var"#38#56"{ForwardDiff.GradientConfig{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 2, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 2}}}, OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}}, OptimizationForwardDiffExt.var"#41#59"{ForwardDiff.HessianConfig{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 2, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Float64}, ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 2}, 2}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 2}}}, OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}}, OptimizationForwardDiffExt.var"#44#62", Nothing, OptimizationForwardDiffExt.var"#48#66"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, OptimizationForwardDiffExt.var"#53#71"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, NLopt.Algorithm, Base.Iterators.Cycle{Tuple{Optimization.NullData}}, Bool, Optimization.NullCallback}, NLopt.Algorithm, Float64, NLopt.Opt, Float64, Nothing}

From this, we can see that it is an OptimizationSolution. We can see the documentation for how to use the OptimizationSolution by checking the Optimization.jl solution type page. For example, the solution is stored as .u. What is the solution to our optimization, and what is the final loss value? We can check it as follows:

# Analyze the solution
-@show sol.u, L(sol.u, p)
([1.0, 1.0], 0.0)
+@show sol.u, L(sol.u, p)
([1.0, 1.0], 0.0)
diff --git a/dev/getting_started/first_simulation/15c441e5.svg b/dev/getting_started/first_simulation/561ccb23.svg similarity index 91% rename from dev/getting_started/first_simulation/15c441e5.svg rename to dev/getting_started/first_simulation/561ccb23.svg index dc69ecb3969..dfe9170ff3b 100644 --- a/dev/getting_started/first_simulation/15c441e5.svg +++ b/dev/getting_started/first_simulation/561ccb23.svg @@ -1,54 +1,54 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/getting_started/first_simulation/4c1adcc8.svg b/dev/getting_started/first_simulation/732601a1.svg similarity index 91% rename from dev/getting_started/first_simulation/4c1adcc8.svg rename to dev/getting_started/first_simulation/732601a1.svg index 0c1a107d86e..7c4ab038a8f 100644 --- a/dev/getting_started/first_simulation/4c1adcc8.svg +++ b/dev/getting_started/first_simulation/732601a1.svg @@ -1,86 +1,86 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/getting_started/first_simulation/2bd5e1c9.svg b/dev/getting_started/first_simulation/880868d4.svg similarity index 90% rename from dev/getting_started/first_simulation/2bd5e1c9.svg rename to dev/getting_started/first_simulation/880868d4.svg index 38858fc4520..f74526e583b 100644 --- a/dev/getting_started/first_simulation/2bd5e1c9.svg +++ b/dev/getting_started/first_simulation/880868d4.svg @@ -1,46 +1,46 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/getting_started/first_simulation/b01a8575.svg b/dev/getting_started/first_simulation/ab225d80.svg similarity index 91% rename from dev/getting_started/first_simulation/b01a8575.svg rename to dev/getting_started/first_simulation/ab225d80.svg index 2642c15b77e..1e020827dfd 100644 --- a/dev/getting_started/first_simulation/b01a8575.svg +++ b/dev/getting_started/first_simulation/ab225d80.svg @@ -1,86 +1,86 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/getting_started/first_simulation/index.html b/dev/getting_started/first_simulation/index.html index 2a20b86c8e3..e2ff34b827f 100644 --- a/dev/getting_started/first_simulation/index.html +++ b/dev/getting_started/first_simulation/index.html @@ -35,7 +35,7 @@ p1 = plot(sol, title = "Rabbits vs Wolves") p2 = plot(sol, idxs = z, title = "Total Animals") -plot(p1, p2, layout = (2, 1))Example block output

Step-by-Step Solution

Step 1: Install and Import the Required Packages

To do this tutorial, we will need a few components:

To start, let's add these packages as demonstrated in the installation tutorial:

using Pkg
+plot(p1, p2, layout = (2, 1))
Example block output

Step-by-Step Solution

Step 1: Install and Import the Required Packages

To do this tutorial, we will need a few components:

To start, let's add these packages as demonstrated in the installation tutorial:

using Pkg
 Pkg.add(["ModelingToolkit", "DifferentialEquations", "Plots"])

Now we're ready. Let's load in these packages:

using ModelingToolkit, DifferentialEquations, Plots

Step 2: Define our ODE Equations

Now let's define our ODEs. We use the ModelingToolkit.@variabes statement to declare our variables. We have the independent variable time t, and then define our 3 state variables:

# Define our state variables: state(t) = initial condition
 @variables t x(t)=1 y(t)=1 z(t)=2

\[ \begin{equation} \left[ @@ -152,7 +152,7 @@ 3.9376728023907703 2.57936090872541 1.940128509890674

returns the time series of the observable z at time points corresponding to sol.t. We can use this with the automated plotting functionality. First let's create a plot of x and y over time using plot(sol) which will plot all of the states. Then next, we will explicitly tell it to make a plot with the index being z, i.e. idxs=z.

Note

Note that one can pass an array of indices as well, so idxs=[x,y,z] would make a plot with all three lines together!

# Plot the solution
-p1 = plot(sol, title = "Rabbits vs Wolves")
Example block output
p2 = plot(sol, idxs = z, title = "Total Animals")
Example block output

Finally, let's make a plot where we merge these two plot elements. To do so, we can take our two plot objects, p1 and p2, and make a plot with both of them. Then we tell Plots to do a layout of (2,1), or 2 rows and 1 columns. Let's see what happens when we bring these together:

plot(p1, p2, layout = (2, 1))
Example block output

And tada, we have a full analysis of our ecosystem!

Bonus Step: Emoji Variables

If you made it this far, then congrats, you get to learn a fun fact! Since Julia code can use Unicode, emojis work for variable names. Here's the simulation using emojis of rabbits and wolves to define the system:

using ModelingToolkit, DifferentialEquations
+p1 = plot(sol, title = "Rabbits vs Wolves")
Example block output
p2 = plot(sol, idxs = z, title = "Total Animals")
Example block output

Finally, let's make a plot where we merge these two plot elements. To do so, we can take our two plot objects, p1 and p2, and make a plot with both of them. Then we tell Plots to do a layout of (2,1), or 2 rows and 1 columns. Let's see what happens when we bring these together:

plot(p1, p2, layout = (2, 1))
Example block output

And tada, we have a full analysis of our ecosystem!

Bonus Step: Emoji Variables

If you made it this far, then congrats, you get to learn a fun fact! Since Julia code can use Unicode, emojis work for variable names. Here's the simulation using emojis of rabbits and wolves to define the system:

using ModelingToolkit, DifferentialEquations
 @parameters α=1.5 β=1.0 γ=3.0 δ=1.0
 @variables t 🐰(t)=1.0 🐺(t)=1.0
 D = Differential(t)
@@ -205,4 +205,4 @@
  [1.816420140681761, 4.064056625315978]
  [1.1465021407690728, 2.7911706616216976]
  [0.9557986135403302, 1.6235622951850799]
- [1.0337581256020607, 0.9063703842886133]

Now go make your professor mad that they have to grade a fully emojified code. I'll vouch for you: the documentation told you to do this.

+ [1.0337581256020607, 0.9063703842886133]

Now go make your professor mad that they have to grade a fully emojified code. I'll vouch for you: the documentation told you to do this.

diff --git a/dev/getting_started/fit_simulation/index.html b/dev/getting_started/fit_simulation/index.html index 8b3b50972fb..c5f1e167a10 100644 --- a/dev/getting_started/fit_simulation/index.html +++ b/dev/getting_started/fit_simulation/index.html @@ -136,4 +136,4 @@ 1.5002841181763384 1.0009286254415546 3.0002381483082114 - 1.0004604305929508

and the answer from the optimization is our desired parameters.

+ 1.0004604305929508

and the answer from the optimization is our desired parameters.

diff --git a/dev/getting_started/getting_started/index.html b/dev/getting_started/getting_started/index.html index 14be1c50249..4fe08a5e40a 100644 --- a/dev/getting_started/getting_started/index.html +++ b/dev/getting_started/getting_started/index.html @@ -1,2 +1,2 @@ -Getting Started with Julia's SciML · Overview of Julia's SciML
GitHub

Getting Started with Julia's SciML

Quickly: What is Julia's SciML Ecosystem?

Julia's SciML is:

  • SciPy or MATLAB's standard library but in Julia, but
  • Runs orders of magnitude faster, even outperforms C and Fortran libraries, and
  • Is fully compatible with machine learning and automatic differentiation,
  • All while having an easy-to-use high level interactive development environment.

Interested?

Introductory Tutorials

Note

Each of the SciML packages starts with its own introductory tutorial as well! Once you have started to get the hang of a few things, start checking out the introductory tutorials of the different packages. For example, the DifferentialEquations.jl getting started tutorial is a fun one!

Coming from...

Are you familiar with other scientific computing tools? Take a look at the guided introductions below.

+Getting Started with Julia's SciML · Overview of Julia's SciML
GitHub

Getting Started with Julia's SciML

Quickly: What is Julia's SciML Ecosystem?

Julia's SciML is:

  • SciPy or MATLAB's standard library but in Julia, but
  • Runs orders of magnitude faster, even outperforms C and Fortran libraries, and
  • Is fully compatible with machine learning and automatic differentiation,
  • All while having an easy-to-use high level interactive development environment.

Interested?

Introductory Tutorials

Note

Each of the SciML packages starts with its own introductory tutorial as well! Once you have started to get the hang of a few things, start checking out the introductory tutorials of the different packages. For example, the DifferentialEquations.jl getting started tutorial is a fun one!

Coming from...

Are you familiar with other scientific computing tools? Take a look at the guided introductions below.

diff --git a/dev/getting_started/installation/index.html b/dev/getting_started/installation/index.html index 4743e0559de..e2746de4267 100644 --- a/dev/getting_started/installation/index.html +++ b/dev/getting_started/installation/index.html @@ -1,3 +1,3 @@ Installing SciML Software · Overview of Julia's SciML
GitHub

Installing SciML Software

Step 1: Install Julia

Download Julia using this website.

Note

Some Linux distributions do weird and incorrect things with Julia installations! Please install Julia using the binaries provided by the official JuliaLang website!

To ensure that you have installed Julia correctly, open it up and type versioninfo() in the REPL. It should look like the following:

(with the CPU/OS/etc. details matching your computer!)

If you got stuck in this installation process, ask for help on the Julia Discourse or in the Julia Zulip chatrooms

Optional Step 1.5: Get VS Code Setup with the Julia Extension

You can run SciML with Julia in any development environment you please, but our recommended environment is VS Code. For more information on using Julia with VS Code, check out the Julia VS Code Extension website. Let's install it!

First download VS Code from the official website.

Next, open Visual Studio Code and click Extensions.

Then, search for “Julia” in the search bar on the top of the extension tab, click on the “Julia” extension, and click the install button on the tab that opens up.

To make sure your installation is correct, try running some code. Open a new file by either going to the top left navigation bar File |> New Text File, or hitting Ctrl+n. Name your new file test.jl (important: the Julia VS Code functionality only turns on when using a .jl file!). Next, type 1+1 and hit Ctrl+Enter. A Julia REPL should pop up and the result 2 should be displayed. Your environment should look something like this:

For more help on using the VS Code editor with Julia, check out the VS Code in Julia documentation. Useful keyboard commands can be found here.

Once again, if you got stuck in this installation process, ask for help on the Julia Discourse or in the Julia Zulip chatrooms

Step 2: Install a SciML Package

SciML is over 130 Julia packages. That's too much stuff to give someone in a single download! Thus instead, the SciML organization divides its functionality into composable modules that can be mixed and matched as required. Installing SciML ecosystem functionality is equivalent to installation of such packages.

For example, do you need the differential equation solver? Then install DifferentialEquations via the command:

using Pkg;
-Pkg.add("DifferentialEquations");

in the Julia REPL. Or, for a more robust REPL experience, hit the ] command to make the blue pkg> REPL environment start, and type in add DifferentialEquations. The package REPL environment will have nice extras like auto-complete that will be useful in the future. This command should run an installation sequence and precompile all of the packages (precompile = "run a bunch of performance optimizations!"). Don't be surprised if this installation process takes ~10 minutes on older computers. During the installation, it should look like this:

And that's it!

How do I test that my installed correctly?

The best way is to build and run your first simulation!

+Pkg.add("DifferentialEquations");

in the Julia REPL. Or, for a more robust REPL experience, hit the ] command to make the blue pkg> REPL environment start, and type in add DifferentialEquations. The package REPL environment will have nice extras like auto-complete that will be useful in the future. This command should run an installation sequence and precompile all of the packages (precompile = "run a bunch of performance optimizations!"). Don't be surprised if this installation process takes ~10 minutes on older computers. During the installation, it should look like this:

And that's it!

How do I test that my installed correctly?

The best way is to build and run your first simulation!

diff --git a/dev/highlevels/array_libraries/index.html b/dev/highlevels/array_libraries/index.html index 383def4d5ec..9eb1a23c12a 100644 --- a/dev/highlevels/array_libraries/index.html +++ b/dev/highlevels/array_libraries/index.html @@ -22,4 +22,4 @@ lorenz_p = (σ = 10.0, ρ = 28.0, β = 8 / 3) lorenz_ic = ComponentArray(x = 0.0, y = 0.0, z = 0.0) -lorenz_prob = ODEProblem(lorenz!, lorenz_ic, tspan, lorenz_p)

Is that beautiful? Yes, it is.

StaticArrays.jl: Statically-Defined Arrays

StaticArrays.jl is a library for statically-defined arrays. Because these arrays have type-level information for size, they recompile the solvers for every new size. They can be dramatically faster for small sizes (up to approximately size 10), but for larger equations they increase compile time with little to no benefit.

CUDA.jl: NVIDIA CUDA-Based GPU Array Computations

CUDA.jl is the library for defining arrays which live on NVIDIA GPUs (CuArray). SciML's libraries will respect the GPU-ness of the inputs, i.e., if the input arrays live on the GPU then the operations will all take place on the GPU or else the libraries will error if it's unable to do so. Thus, using CUDA.jl's CuArray is how one GPU-accelerates any computation with the SciML organization's libraries. Simply use a CuArray as the initial condition to an ODE solve or as the initial guess for a nonlinear solve, and the whole solve will recompile to take place on the GPU.

AMDGPU.jl: AMD-Based GPU Array Computations

AMDGPU.jl is the library for defining arrays which live on AMD GPUs (ROCArray). SciML's libraries will respect the GPU-ness of the inputs, i.e., if the input arrays live on the GPU then the operations will all take place on the GPU or else the libraries will error if it's unable to do so. Thus using AMDGPU.jl's ROCArray is how one GPU-accelerates any computation with the SciML organization's libraries. Simply use a ROCArray as the initial condition to an ODE solve or as the initial guess for a nonlinear solve, and the whole solve will recompile to take place on the GPU.

FillArrays.jl: Lazy Arrays

FillArrays.jl is a library for defining arrays with lazy values. For example, an O(1) representation of the identity matrix is given by Eye{Int}(5). FillArrays.jl is used extensively throughout the ecosystem to improve runtime and memory performance.

BandedMatrices.jl: Fast Banded Matrices

Banded matrices show up in many equation solver contexts, such as the Jacobians of many partial differential equations. While the base SparseMatrixCSC sparse matrix type can represent such matrices, BandedMatrices.jl is a specialized format specifically for BandedMatrices which can be used to greatly improve performance of operations on a banded matrix.

BlockBandedMatrices.jl: Fast Block-Banded Matrices

Block banded matrices show up in many equation solver contexts, such as the Jacobians of many systems of partial differential equations. While the base SparseMatrixCSC sparse matrix type can represent such matrices, BlockBandedMatrices.jl is a specialized format specifically for BlockBandedMatrices which can be used to greatly improve performance of operations on a block-banded matrix.

+lorenz_prob = ODEProblem(lorenz!, lorenz_ic, tspan, lorenz_p)

Is that beautiful? Yes, it is.

StaticArrays.jl: Statically-Defined Arrays

StaticArrays.jl is a library for statically-defined arrays. Because these arrays have type-level information for size, they recompile the solvers for every new size. They can be dramatically faster for small sizes (up to approximately size 10), but for larger equations they increase compile time with little to no benefit.

CUDA.jl: NVIDIA CUDA-Based GPU Array Computations

CUDA.jl is the library for defining arrays which live on NVIDIA GPUs (CuArray). SciML's libraries will respect the GPU-ness of the inputs, i.e., if the input arrays live on the GPU then the operations will all take place on the GPU or else the libraries will error if it's unable to do so. Thus, using CUDA.jl's CuArray is how one GPU-accelerates any computation with the SciML organization's libraries. Simply use a CuArray as the initial condition to an ODE solve or as the initial guess for a nonlinear solve, and the whole solve will recompile to take place on the GPU.

AMDGPU.jl: AMD-Based GPU Array Computations

AMDGPU.jl is the library for defining arrays which live on AMD GPUs (ROCArray). SciML's libraries will respect the GPU-ness of the inputs, i.e., if the input arrays live on the GPU then the operations will all take place on the GPU or else the libraries will error if it's unable to do so. Thus using AMDGPU.jl's ROCArray is how one GPU-accelerates any computation with the SciML organization's libraries. Simply use a ROCArray as the initial condition to an ODE solve or as the initial guess for a nonlinear solve, and the whole solve will recompile to take place on the GPU.

FillArrays.jl: Lazy Arrays

FillArrays.jl is a library for defining arrays with lazy values. For example, an O(1) representation of the identity matrix is given by Eye{Int}(5). FillArrays.jl is used extensively throughout the ecosystem to improve runtime and memory performance.

BandedMatrices.jl: Fast Banded Matrices

Banded matrices show up in many equation solver contexts, such as the Jacobians of many partial differential equations. While the base SparseMatrixCSC sparse matrix type can represent such matrices, BandedMatrices.jl is a specialized format specifically for BandedMatrices which can be used to greatly improve performance of operations on a banded matrix.

BlockBandedMatrices.jl: Fast Block-Banded Matrices

Block banded matrices show up in many equation solver contexts, such as the Jacobians of many systems of partial differential equations. While the base SparseMatrixCSC sparse matrix type can represent such matrices, BlockBandedMatrices.jl is a specialized format specifically for BlockBandedMatrices which can be used to greatly improve performance of operations on a block-banded matrix.

diff --git a/dev/highlevels/developer_documentation/index.html b/dev/highlevels/developer_documentation/index.html index c6d9e247ea4..8d13411f2ea 100644 --- a/dev/highlevels/developer_documentation/index.html +++ b/dev/highlevels/developer_documentation/index.html @@ -1,3 +1,3 @@ Developer Documentation · Overview of Julia's SciML
GitHub

Developer Documentation

For uniformity and clarity, the SciML Open-Source Software Organization has many well-defined rules and practices for its development. However, we stress one important principle:

Do not be deterred from contributing if you think you do not know everything. No one knows everything. These rules and styles are designed for iterative contributions. Open pull requests and contribute what you can with what you know, and the maintainers will help you learn and do the rest!

If you need any help contributing, please feel welcome joining our community channels.

We welcome everybody.

Getting Started With Contributing to SciML

To get started contributing to SciML, check out the following resources:

SciMLStyle: The SciML Style Guide for Julia

SciML Code Style

This is a style guide for how to program in Julia for SciML contributions. It describes everything one needs to know, from preferred naming schemes of functions to fundamental dogmas for designing traits. We stress that this style guide is meant to be comprehensive for the sake of designing automatic formatters and teaching desired rules, but complete knowledge and adherence to the style guide is not required for contributions!

COLPRAC: Contributor's Guide on Collaborative Practices for Community Packages

ColPrac: Contributor's Guide on Collaborative Practices for Community Packages

What are the rules for when PRs should be merged? What are the rules for whether to tag a major, minor, or patch release? All of these development rules are defined in COLPRAC.

DiffEq Developer Documentation

There are many solver libraries which share similar internals, such as OrdinaryDiffEq.jl, StochasticDiffEq.jl, and DelayDiffEq.jl. This section of the documentation describes the internal systems of these packages and how they are used to quickly write efficient solvers.

Third-Party Libraries to Note

Documenter.jl

Documenter.jl is the documentation generation library that the SciML organization uses, and thus its documentation is the documentation of the documentation.

JuliaFormatter.jl

JuliaFormatter.jl is the formatter used by the SciML organization to enforce the SciML Style. Setting style = "sciml" in a .JuliaFormatter.toml file of a repo and using the standard FormatCheck.yml as part of continuous integration makes JuliaFormatter check for SciML Style compliance on pull requests.

To run JuliaFormatter in a SciML repository, do:

using JuliaFomatter, DevedPackage
-JuliaFormatter.format(pkgdir(DevedPackage))

which will reformat the code according to the SciML Style.

GitHub Actions Continuous Integrations

The SciML Organization uses continuous integration testing to always ensure tests are passing when merging pull requests. The organization uses the GitHub Actions supplied by Julia Actions to accomplish this. Common continuous integration scripts are:

  • CI.yml, the standard CI script
  • Downstream.yml, used to specify packages for downstream testing. This will make packages which depend on the current package also be tested to ensure that “non-breaking changes” do not actually break other packages.
  • Documentation.yml, used to run the documentation automatic generation with Documenter.jl
  • FormatCheck.yml, used to check JuliaFormatter SciML Style compliance

CompatHelper

CompatHelper is used to automatically create pull requests whenever a dependent package is upper bounded. The results of CompatHelper PRs should be checked to ensure that the latest version of the dependencies are grabbed for the test process. After successful CompatHelper PRs, i.e. if the increase of the upper bound did not cause a break to the tests, a new version tag should follow. It is set up by adding the CompatHelper.yml GitHub action.

TagBot

TagBot automatically creates tags in the GitHub repository whenever a package is registered to the Julia General repository. It is set up by adding the TagBot.yml GitHub action.

+JuliaFormatter.format(pkgdir(DevedPackage))

which will reformat the code according to the SciML Style.

GitHub Actions Continuous Integrations

The SciML Organization uses continuous integration testing to always ensure tests are passing when merging pull requests. The organization uses the GitHub Actions supplied by Julia Actions to accomplish this. Common continuous integration scripts are:

  • CI.yml, the standard CI script
  • Downstream.yml, used to specify packages for downstream testing. This will make packages which depend on the current package also be tested to ensure that “non-breaking changes” do not actually break other packages.
  • Documentation.yml, used to run the documentation automatic generation with Documenter.jl
  • FormatCheck.yml, used to check JuliaFormatter SciML Style compliance

CompatHelper

CompatHelper is used to automatically create pull requests whenever a dependent package is upper bounded. The results of CompatHelper PRs should be checked to ensure that the latest version of the dependencies are grabbed for the test process. After successful CompatHelper PRs, i.e. if the increase of the upper bound did not cause a break to the tests, a new version tag should follow. It is set up by adding the CompatHelper.yml GitHub action.

TagBot

TagBot automatically creates tags in the GitHub repository whenever a package is registered to the Julia General repository. It is set up by adding the TagBot.yml GitHub action.

diff --git a/dev/highlevels/equation_solvers/index.html b/dev/highlevels/equation_solvers/index.html index a52dea7a8c0..9cad3c9e067 100644 --- a/dev/highlevels/equation_solvers/index.html +++ b/dev/highlevels/equation_solvers/index.html @@ -1,2 +1,2 @@ -Equation Solvers · Overview of Julia's SciML
GitHub

Equation Solvers

The SciML Equation Solvers cover a large set of SciMLProblems with SciMLAlgorithms that are efficient, numerically stable, and flexible. These methods tie into libraries like SciMLSensitivity.jl to be fully differentiable and compatible with machine learning pipelines, and are designed for integration with applications like parameter estimation, global sensitivity analysis, and more.

LinearSolve.jl: Unified Interface for Linear Solvers

LinearSolve.jl is the canonical library for solving LinearProblems. It includes:

  • Fast pure Julia LU factorizations which outperform standard BLAS
  • KLU for faster sparse LU factorization on unstructured matrices
  • UMFPACK for faster sparse LU factorization on matrices with some repeated structure
  • MKLPardiso wrappers for handling many sparse matrices faster than SuiteSparse (KLU, UMFPACK) methods
  • GPU-offloading for large dense matrices
  • Wrappers to all of the Krylov implementations (Krylov.jl, IterativeSolvers.jl, KrylovKit.jl) for easy testing of all of them. LinearSolve.jl handles the API differences, especially with the preconditioner definitions
  • A polyalgorithm that smartly chooses between these methods
  • A caching interface which automates caching of symbolic factorizations and numerical factorizations as optimally as possible
  • Compatible with arbitrary AbstractArray and Number types, such as GPU-based arrays, uncertainty quantification number types, and more.

NonlinearSolve.jl: Unified Interface for Nonlinear Solvers

NonlinearSolve.jl is the canonical library for solving NonlinearProblems. It includes:

  • Fast non-allocating implementations on static arrays of common methods (Newton-Rhapson)
  • Bracketing methods (Bisection, Falsi) for methods with known upper and lower bounds (IntervalNonlinearProblem)
  • Wrappers to common other solvers (NLsolve.jl, MINPACK, KINSOL from Sundials) for trust region methods, line search-based approaches, etc.
  • Built over the LinearSolve.jl API for maximum flexibility and performance in the solving approach
  • Compatible with arbitrary AbstractArray and Number types, such as GPU-based arrays, uncertainty quantification number types, and more.

DifferentialEquations.jl: Unified Interface for Differential Equation Solvers

DifferentialEquations.jl is the canonical library for solving DEProblems. This includes:

  • Discrete equations (function maps, discrete stochastic (Gillespie/Markov) simulations) (DiscreteProblem)
  • Ordinary differential equations (ODEs) (ODEProblem)
  • Split and Partitioned ODEs (Symplectic integrators, IMEX Methods) (SplitODEProblem)
  • Stochastic ordinary differential equations (SODEs or SDEs) (SDEProblem)
  • Stochastic differential-algebraic equations (SDAEs) (SDEProblem with mass matrices)
  • Random differential equations (RODEs or RDEs) (RODEProblem)
  • Differential algebraic equations (DAEs) (DAEProblem and ODEProblem with mass matrices)
  • Delay differential equations (DDEs) (DDEProblem)
  • Neutral, retarded, and algebraic delay differential equations (NDDEs, RDDEs, and DDAEs)
  • Stochastic delay differential equations (SDDEs) (SDDEProblem)
  • Experimental support for stochastic neutral, retarded, and algebraic delay differential equations (SNDDEs, SRDDEs, and SDDAEs)
  • Mixed discrete and continuous equations (Hybrid Equations, Jump Diffusions) (DEProblems with callbacks and JumpProblem)

The well-optimized DifferentialEquations solvers benchmark as some of the fastest implementations of classic algorithms. It also includes algorithms from recent research which routinely outperform the “standard” C/Fortran methods, and algorithms optimized for high-precision and HPC applications. Simultaneously, it wraps the classic C/Fortran methods, making it easy to switch over to them whenever necessary. Solving differential equations with different methods from different languages and packages can be done by changing one line of code, allowing for easy benchmarking to ensure you are using the fastest method possible.

DifferentialEquations.jl integrates with the Julia package sphere. Examples are:

  • GPU acceleration through CUDAnative.jl and CuArrays.jl
  • Automated sparsity detection with Symbolics.jl
  • Automatic Jacobian coloring with SparseDiffTools.jl, allowing for fast solutions to problems with sparse or structured (Tridiagonal, Banded, BlockBanded, etc.) Jacobians
  • Allowing the specification of linear solvers for maximal efficiency
  • Progress meter integration with the Juno IDE for estimated time to solution
  • Automatic plotting of time series and phase plots
  • Built-in interpolations
  • Wraps for common C/Fortran methods, like Sundials and Hairer's radau
  • Arbitrary precision with BigFloats and Arbfloats
  • Arbitrary array types, allowing the definition of differential equations on matrices and distributed arrays
  • Unit-checked arithmetic with Unitful

Optimization.jl: Unified Interface for Optimization

Optimization.jl is the canonical library for solving OptimizationProblems. It includes wrappers of most of the Julia nonlinear optimization ecosystem, allowing one syntax to use all packages in a uniform manner. This covers:

Integrals.jl: Unified Interface for Numerical Integration

Integrals.jl is the canonical library for solving IntegralsProblems. It includes wrappers of most of the Julia quadrature ecosystem, allowing one syntax to use all packages in a uniform manner. This covers:

  • Gauss-Kronrod quadrature
  • Cubature methods (both h and p cubature)
  • Adaptive Monte Carlo methods

JumpProcesses.jl: Stochastic Simulation Algorithms for Jump Processes, Jump-ODEs, and Jump-Diffusions

JumpProcesses.jl is the library for Poisson jump processes, also known as chemical master equations or Gillespie simulations, for simulating chemical reaction networks and other applications. It allows for solving with many methods, including:

  • Direct: the Gillespie Direct method SSA.
  • RDirect: A variant of Gillespie's Direct method that uses rejection to sample the next reaction.
  • DirectCR: The Composition-Rejection Direct method of Slepoy et al. For large networks and linear chain-type networks, it will often give better performance than Direct. (Requires dependency graph, see below.)
  • DirectFW: the Gillespie Direct method SSA with FunctionWrappers. This aggregator uses a different internal storage format for collections of ConstantRateJumps.
  • FRM: the Gillespie first reaction method SSA. Direct should generally offer better performance and be preferred to FRM.
  • FRMFW: the Gillespie first reaction method SSA with FunctionWrappers.
  • NRM: The Gibson-Bruck Next Reaction Method. For some reaction network structures, this may offer better performance than Direct (for example, large, linear chains of reactions). (Requires dependency graph, see below.)
  • RSSA: The Rejection SSA (RSSA) method of Thanh et al. With RSSACR, for very large reaction networks, it often offers the best performance of all methods. (Requires dependency graph, see below.)
  • RSSACR: The Rejection SSA (RSSA) with Composition-Rejection method of Thanh et al. With RSSA, for very large reaction networks, it often offers the best performance of all methods. (Requires dependency graph, see below.)
  • SortingDirect: The Sorting Direct Method of McCollum et al. It will usually offer performance as good as Direct, and for some systems can offer substantially better performance. (Requires dependency graph, see below.)

The design of JumpProcesses.jl composes with DifferentialEquations.jl, allowing for discrete stochastic chemical reactions to be easily mixed with differential equation models, allowing for simulation of hybrid systems, jump diffusions, and differential equations driven by Levy processes.

In addition, JumpProcesses's interfaces allow for solving with regular jump methods, such as adaptive Tau-Leaping.

Third-Party Libraries to Note

JuMP.jl: Julia for Mathematical Programming

While Optimization.jl is the preferred library for nonlinear optimization, for all other forms of optimization Julia for Mathematical Programming (JuMP) is the star. JuMP is the leading choice in Julia for doing:

  • Linear Programming
  • Quadratic Programming
  • Convex Programming
  • Conic Programming
  • Semidefinite Programming
  • Mixed-Complementarity Programming
  • Integer Programming
  • Mixed Integer (nonlinear/linear) Programming
  • (Mixed Integer) Second Order Conic Programming

JuMP can also be used for some nonlinear programming, though the Optimization.jl bindings to the JuMP solvers (via MathOptInterface.jl) is generally preferred.

FractionalDiffEq.jl: Fractional Differential Equation Solvers

FractionalDiffEq.jl is a set of high-performance solvers for fractional differential equations.

ManifoldDiffEq.jl: Solvers for Differential Equations on Manifolds

ManifoldDiffEq.jl is a set of high-performance solvers for differential equations on manifolds using methods such as Lie Group actions and frozen coefficients (Crouch-Grossman methods). These solvers can in many cases out-perform the OrdinaryDiffEq.jl nonautonomous operator ODE solvers by using methods specialized on manifold definitions of ManifoldsBase.

Manopt.jl: Optimization on Manifolds

ManOpt.jl allows for easy and efficient solving of nonlinear optimization problems on manifolds.

+Equation Solvers · Overview of Julia's SciML
GitHub

Equation Solvers

The SciML Equation Solvers cover a large set of SciMLProblems with SciMLAlgorithms that are efficient, numerically stable, and flexible. These methods tie into libraries like SciMLSensitivity.jl to be fully differentiable and compatible with machine learning pipelines, and are designed for integration with applications like parameter estimation, global sensitivity analysis, and more.

LinearSolve.jl: Unified Interface for Linear Solvers

LinearSolve.jl is the canonical library for solving LinearProblems. It includes:

  • Fast pure Julia LU factorizations which outperform standard BLAS
  • KLU for faster sparse LU factorization on unstructured matrices
  • UMFPACK for faster sparse LU factorization on matrices with some repeated structure
  • MKLPardiso wrappers for handling many sparse matrices faster than SuiteSparse (KLU, UMFPACK) methods
  • GPU-offloading for large dense matrices
  • Wrappers to all of the Krylov implementations (Krylov.jl, IterativeSolvers.jl, KrylovKit.jl) for easy testing of all of them. LinearSolve.jl handles the API differences, especially with the preconditioner definitions
  • A polyalgorithm that smartly chooses between these methods
  • A caching interface which automates caching of symbolic factorizations and numerical factorizations as optimally as possible
  • Compatible with arbitrary AbstractArray and Number types, such as GPU-based arrays, uncertainty quantification number types, and more.

NonlinearSolve.jl: Unified Interface for Nonlinear Solvers

NonlinearSolve.jl is the canonical library for solving NonlinearProblems. It includes:

  • Fast non-allocating implementations on static arrays of common methods (Newton-Rhapson)
  • Bracketing methods (Bisection, Falsi) for methods with known upper and lower bounds (IntervalNonlinearProblem)
  • Wrappers to common other solvers (NLsolve.jl, MINPACK, KINSOL from Sundials) for trust region methods, line search-based approaches, etc.
  • Built over the LinearSolve.jl API for maximum flexibility and performance in the solving approach
  • Compatible with arbitrary AbstractArray and Number types, such as GPU-based arrays, uncertainty quantification number types, and more.

DifferentialEquations.jl: Unified Interface for Differential Equation Solvers

DifferentialEquations.jl is the canonical library for solving DEProblems. This includes:

  • Discrete equations (function maps, discrete stochastic (Gillespie/Markov) simulations) (DiscreteProblem)
  • Ordinary differential equations (ODEs) (ODEProblem)
  • Split and Partitioned ODEs (Symplectic integrators, IMEX Methods) (SplitODEProblem)
  • Stochastic ordinary differential equations (SODEs or SDEs) (SDEProblem)
  • Stochastic differential-algebraic equations (SDAEs) (SDEProblem with mass matrices)
  • Random differential equations (RODEs or RDEs) (RODEProblem)
  • Differential algebraic equations (DAEs) (DAEProblem and ODEProblem with mass matrices)
  • Delay differential equations (DDEs) (DDEProblem)
  • Neutral, retarded, and algebraic delay differential equations (NDDEs, RDDEs, and DDAEs)
  • Stochastic delay differential equations (SDDEs) (SDDEProblem)
  • Experimental support for stochastic neutral, retarded, and algebraic delay differential equations (SNDDEs, SRDDEs, and SDDAEs)
  • Mixed discrete and continuous equations (Hybrid Equations, Jump Diffusions) (DEProblems with callbacks and JumpProblem)

The well-optimized DifferentialEquations solvers benchmark as some of the fastest implementations of classic algorithms. It also includes algorithms from recent research which routinely outperform the “standard” C/Fortran methods, and algorithms optimized for high-precision and HPC applications. Simultaneously, it wraps the classic C/Fortran methods, making it easy to switch over to them whenever necessary. Solving differential equations with different methods from different languages and packages can be done by changing one line of code, allowing for easy benchmarking to ensure you are using the fastest method possible.

DifferentialEquations.jl integrates with the Julia package sphere. Examples are:

  • GPU acceleration through CUDAnative.jl and CuArrays.jl
  • Automated sparsity detection with Symbolics.jl
  • Automatic Jacobian coloring with SparseDiffTools.jl, allowing for fast solutions to problems with sparse or structured (Tridiagonal, Banded, BlockBanded, etc.) Jacobians
  • Allowing the specification of linear solvers for maximal efficiency
  • Progress meter integration with the Juno IDE for estimated time to solution
  • Automatic plotting of time series and phase plots
  • Built-in interpolations
  • Wraps for common C/Fortran methods, like Sundials and Hairer's radau
  • Arbitrary precision with BigFloats and Arbfloats
  • Arbitrary array types, allowing the definition of differential equations on matrices and distributed arrays
  • Unit-checked arithmetic with Unitful

Optimization.jl: Unified Interface for Optimization

Optimization.jl is the canonical library for solving OptimizationProblems. It includes wrappers of most of the Julia nonlinear optimization ecosystem, allowing one syntax to use all packages in a uniform manner. This covers:

Integrals.jl: Unified Interface for Numerical Integration

Integrals.jl is the canonical library for solving IntegralsProblems. It includes wrappers of most of the Julia quadrature ecosystem, allowing one syntax to use all packages in a uniform manner. This covers:

  • Gauss-Kronrod quadrature
  • Cubature methods (both h and p cubature)
  • Adaptive Monte Carlo methods

JumpProcesses.jl: Stochastic Simulation Algorithms for Jump Processes, Jump-ODEs, and Jump-Diffusions

JumpProcesses.jl is the library for Poisson jump processes, also known as chemical master equations or Gillespie simulations, for simulating chemical reaction networks and other applications. It allows for solving with many methods, including:

  • Direct: the Gillespie Direct method SSA.
  • RDirect: A variant of Gillespie's Direct method that uses rejection to sample the next reaction.
  • DirectCR: The Composition-Rejection Direct method of Slepoy et al. For large networks and linear chain-type networks, it will often give better performance than Direct. (Requires dependency graph, see below.)
  • DirectFW: the Gillespie Direct method SSA with FunctionWrappers. This aggregator uses a different internal storage format for collections of ConstantRateJumps.
  • FRM: the Gillespie first reaction method SSA. Direct should generally offer better performance and be preferred to FRM.
  • FRMFW: the Gillespie first reaction method SSA with FunctionWrappers.
  • NRM: The Gibson-Bruck Next Reaction Method. For some reaction network structures, this may offer better performance than Direct (for example, large, linear chains of reactions). (Requires dependency graph, see below.)
  • RSSA: The Rejection SSA (RSSA) method of Thanh et al. With RSSACR, for very large reaction networks, it often offers the best performance of all methods. (Requires dependency graph, see below.)
  • RSSACR: The Rejection SSA (RSSA) with Composition-Rejection method of Thanh et al. With RSSA, for very large reaction networks, it often offers the best performance of all methods. (Requires dependency graph, see below.)
  • SortingDirect: The Sorting Direct Method of McCollum et al. It will usually offer performance as good as Direct, and for some systems can offer substantially better performance. (Requires dependency graph, see below.)

The design of JumpProcesses.jl composes with DifferentialEquations.jl, allowing for discrete stochastic chemical reactions to be easily mixed with differential equation models, allowing for simulation of hybrid systems, jump diffusions, and differential equations driven by Levy processes.

In addition, JumpProcesses's interfaces allow for solving with regular jump methods, such as adaptive Tau-Leaping.

Third-Party Libraries to Note

JuMP.jl: Julia for Mathematical Programming

While Optimization.jl is the preferred library for nonlinear optimization, for all other forms of optimization Julia for Mathematical Programming (JuMP) is the star. JuMP is the leading choice in Julia for doing:

  • Linear Programming
  • Quadratic Programming
  • Convex Programming
  • Conic Programming
  • Semidefinite Programming
  • Mixed-Complementarity Programming
  • Integer Programming
  • Mixed Integer (nonlinear/linear) Programming
  • (Mixed Integer) Second Order Conic Programming

JuMP can also be used for some nonlinear programming, though the Optimization.jl bindings to the JuMP solvers (via MathOptInterface.jl) is generally preferred.

FractionalDiffEq.jl: Fractional Differential Equation Solvers

FractionalDiffEq.jl is a set of high-performance solvers for fractional differential equations.

ManifoldDiffEq.jl: Solvers for Differential Equations on Manifolds

ManifoldDiffEq.jl is a set of high-performance solvers for differential equations on manifolds using methods such as Lie Group actions and frozen coefficients (Crouch-Grossman methods). These solvers can in many cases out-perform the OrdinaryDiffEq.jl nonautonomous operator ODE solvers by using methods specialized on manifold definitions of ManifoldsBase.

Manopt.jl: Optimization on Manifolds

ManOpt.jl allows for easy and efficient solving of nonlinear optimization problems on manifolds.

diff --git a/dev/highlevels/function_approximation/index.html b/dev/highlevels/function_approximation/index.html index 161b0ab3a6e..c175b0b3b6e 100644 --- a/dev/highlevels/function_approximation/index.html +++ b/dev/highlevels/function_approximation/index.html @@ -1,2 +1,2 @@ -Function Approximation · Overview of Julia's SciML
GitHub

Function Approximation

While SciML is not an ecosystem for machine learning, SciML has many libraries for doing machine learning with its equation solver libraries and machine learning libraries which are integrated into the equation solvers.

Surrogates.jl: Easy Generation of Differentiable Surrogate Models

Surrogates.jl is a library for generating surrogate approximations to computationally expensive simulations. It has the following high-dimensional function approximators:

  • Kriging
  • Kriging using Stheno
  • Radial Basis
  • Wendland
  • Linear
  • Second Order Polynomial
  • Support Vector Machines (Wait for LIBSVM resolution)
  • Neural Networks
  • Random Forests
  • Lobachevsky splines
  • Inverse-distance
  • Polynomial expansions
  • Variable fidelity
  • Mixture of experts (Waiting GaussianMixtures package to work on v1.5)
  • Earth
  • Gradient Enhanced Kriging

ReservoirComputing.jl: Fast and Flexible Reservoir Computing Methods

ReservoirComputing.jl is a library for doing machine learning using reservoir computing techniques, such as with methods like Echo State Networks (ESNs). Its reservoir computing methods make it stabilized for usage with difficult equations like stiff dynamics, chaotic equations, and more.

Third-Party Libraries to Note

Flux.jl: the ML library that doesn't make you tensor

Flux.jl is the most popular machine learning library in the Julia programming language. SciML's libraries are heavily tested with it and its automatic differentiation engine Zygote.jl for composability and compatibility.

Lux.jl: Explicitly Parameterized Neural Networks in Julia

Lux.jl is a library for fully explicitly parameterized neural networks. Thus, while alternative interfaces are required to use Flux with many equation solvers (i.e. Flux.destructure), Lux.jl's explicit design marries effortlessly with the SciML equation solver libraries. For this reason, SciML's library are also heavily tested with Lux to ensure compatibility with neural network definitions from here.

SimpleChains.jl: Fast Small-Scale Machine Learning

SimpleChains.jl is a library specialized for small-scale machine learning. It uses non-allocating mutating forms to be highly efficient for the cases where matrix multiplication kernels cannot overcome the common overheads of machine learning libraries. Thus for SciML cases with small neural networks (<100 node layers) and non-batched usage (many/most use cases), SimpleChains.jl can be the fastest choice for the neural network definitions.

NNLib.jl: Neural Network Primitives with Multiple Backends

NNLib.jl is the core library which defines the handling of common functions, like conv and how they map to device accelerators such as the NVIDIA cudnn. This library can thus be used to directly grab many of the core functions used in machine learning, such as common activation functions and gather/scatter operations, without depending on the given style of any machine learning library.

GeometricFlux.jl: Geometric Deep Learning and Graph Neural Networks

GeometricFlux.jl is a library for graph neural networks and geometric deep learning. It is the one that is used and tested by the SciML developers for mixing with equation solver applications.

AbstractGPs.jl: Fast and Flexible Gaussian Processes

AbstractGPs.jl is the fast and flexible Gaussian Process library that is used by the SciML packages and recommended for downstream usage.

MLDatasets.jl: Common Machine Learning Datasets

MLDatasets.jl is a common interface for accessing common machine learning datasets. For example, if you want to run a test on MNIST data, MLDatasets is the quickest way to obtain it.

MLUtils.jl: Utility Functions for Machine Learning Pipelines

MLUtils.jl is a library of utility functions for making writing common machine learning pipelines easier. This includes functionality for:

  • An extensible dataset interface (numobs and getobs).
  • Data iteration and data loaders (eachobs and DataLoader).
  • Lazy data views (obsview).
  • Resampling procedures (undersample and oversample).
  • Train/test splits (splitobs)
  • Data partitioning and aggregation tools (batch, unbatch, chunk, group_counts, group_indices).
  • Folds for cross-validation (kfolds, leavepout).
  • Datasets lazy transformations (mapobs, filterobs, groupobs, joinobs, shuffleobs).
  • Toy datasets for demonstration purpose.
  • Other data handling utilities (flatten, normalise, unsqueeze, stack, unstack).
+Function Approximation · Overview of Julia's SciML
GitHub

Function Approximation

While SciML is not an ecosystem for machine learning, SciML has many libraries for doing machine learning with its equation solver libraries and machine learning libraries which are integrated into the equation solvers.

Surrogates.jl: Easy Generation of Differentiable Surrogate Models

Surrogates.jl is a library for generating surrogate approximations to computationally expensive simulations. It has the following high-dimensional function approximators:

  • Kriging
  • Kriging using Stheno
  • Radial Basis
  • Wendland
  • Linear
  • Second Order Polynomial
  • Support Vector Machines (Wait for LIBSVM resolution)
  • Neural Networks
  • Random Forests
  • Lobachevsky splines
  • Inverse-distance
  • Polynomial expansions
  • Variable fidelity
  • Mixture of experts (Waiting GaussianMixtures package to work on v1.5)
  • Earth
  • Gradient Enhanced Kriging

ReservoirComputing.jl: Fast and Flexible Reservoir Computing Methods

ReservoirComputing.jl is a library for doing machine learning using reservoir computing techniques, such as with methods like Echo State Networks (ESNs). Its reservoir computing methods make it stabilized for usage with difficult equations like stiff dynamics, chaotic equations, and more.

Third-Party Libraries to Note

Flux.jl: the ML library that doesn't make you tensor

Flux.jl is the most popular machine learning library in the Julia programming language. SciML's libraries are heavily tested with it and its automatic differentiation engine Zygote.jl for composability and compatibility.

Lux.jl: Explicitly Parameterized Neural Networks in Julia

Lux.jl is a library for fully explicitly parameterized neural networks. Thus, while alternative interfaces are required to use Flux with many equation solvers (i.e. Flux.destructure), Lux.jl's explicit design marries effortlessly with the SciML equation solver libraries. For this reason, SciML's library are also heavily tested with Lux to ensure compatibility with neural network definitions from here.

SimpleChains.jl: Fast Small-Scale Machine Learning

SimpleChains.jl is a library specialized for small-scale machine learning. It uses non-allocating mutating forms to be highly efficient for the cases where matrix multiplication kernels cannot overcome the common overheads of machine learning libraries. Thus for SciML cases with small neural networks (<100 node layers) and non-batched usage (many/most use cases), SimpleChains.jl can be the fastest choice for the neural network definitions.

NNLib.jl: Neural Network Primitives with Multiple Backends

NNLib.jl is the core library which defines the handling of common functions, like conv and how they map to device accelerators such as the NVIDIA cudnn. This library can thus be used to directly grab many of the core functions used in machine learning, such as common activation functions and gather/scatter operations, without depending on the given style of any machine learning library.

GeometricFlux.jl: Geometric Deep Learning and Graph Neural Networks

GeometricFlux.jl is a library for graph neural networks and geometric deep learning. It is the one that is used and tested by the SciML developers for mixing with equation solver applications.

AbstractGPs.jl: Fast and Flexible Gaussian Processes

AbstractGPs.jl is the fast and flexible Gaussian Process library that is used by the SciML packages and recommended for downstream usage.

MLDatasets.jl: Common Machine Learning Datasets

MLDatasets.jl is a common interface for accessing common machine learning datasets. For example, if you want to run a test on MNIST data, MLDatasets is the quickest way to obtain it.

MLUtils.jl: Utility Functions for Machine Learning Pipelines

MLUtils.jl is a library of utility functions for making writing common machine learning pipelines easier. This includes functionality for:

  • An extensible dataset interface (numobs and getobs).
  • Data iteration and data loaders (eachobs and DataLoader).
  • Lazy data views (obsview).
  • Resampling procedures (undersample and oversample).
  • Train/test splits (splitobs)
  • Data partitioning and aggregation tools (batch, unbatch, chunk, group_counts, group_indices).
  • Folds for cross-validation (kfolds, leavepout).
  • Datasets lazy transformations (mapobs, filterobs, groupobs, joinobs, shuffleobs).
  • Toy datasets for demonstration purpose.
  • Other data handling utilities (flatten, normalise, unsqueeze, stack, unstack).
diff --git a/dev/highlevels/implicit_layers/index.html b/dev/highlevels/implicit_layers/index.html index 02dda1df3bf..5ec312582d2 100644 --- a/dev/highlevels/implicit_layers/index.html +++ b/dev/highlevels/implicit_layers/index.html @@ -1,2 +1,2 @@ -Implicit Layer Deep Learning · Overview of Julia's SciML
GitHub

Implicit Layer Deep Learning

Implicit layer deep learning is a field which uses implicit rules, such as differential equations and nonlinear solvers, to define the layers of neural networks. This field has brought the potential to automatically optimize network depth and improve training performance. SciML's differentiable solver ecosystem is specifically designed to accommodate implicit layer methodologies, and provides libraries with pre-built layers for common methods.

DiffEqFlux.jl: High Level Pre-Built Architectures for Implicit Deep Learning

DiffEqFlux.jl is a library of pre-built architectures for implicit deep learning, including layer definitions for methods like:

DeepEquilibriumNetworks.jl: Deep Equilibrium Models Made Fast

DeepEquilibriumNetworks.jl is a library of optimized layer implementations for Deep Equilibrium Models (DEQs). It uses special training techniques such as implicit-explicit regularization in order to accelerate the convergence over traditional implementations, all while using the optimized and flexible SciML libraries under the hood.

+Implicit Layer Deep Learning · Overview of Julia's SciML
GitHub

Implicit Layer Deep Learning

Implicit layer deep learning is a field which uses implicit rules, such as differential equations and nonlinear solvers, to define the layers of neural networks. This field has brought the potential to automatically optimize network depth and improve training performance. SciML's differentiable solver ecosystem is specifically designed to accommodate implicit layer methodologies, and provides libraries with pre-built layers for common methods.

DiffEqFlux.jl: High Level Pre-Built Architectures for Implicit Deep Learning

DiffEqFlux.jl is a library of pre-built architectures for implicit deep learning, including layer definitions for methods like:

DeepEquilibriumNetworks.jl: Deep Equilibrium Models Made Fast

DeepEquilibriumNetworks.jl is a library of optimized layer implementations for Deep Equilibrium Models (DEQs). It uses special training techniques such as implicit-explicit regularization in order to accelerate the convergence over traditional implementations, all while using the optimized and flexible SciML libraries under the hood.

diff --git a/dev/highlevels/interfaces/index.html b/dev/highlevels/interfaces/index.html index 20388edf21e..35c35817785 100644 --- a/dev/highlevels/interfaces/index.html +++ b/dev/highlevels/interfaces/index.html @@ -1,2 +1,2 @@ -The SciML Interface Libraries · Overview of Julia's SciML
GitHub

The SciML Interface Libraries

SciMLBase.jl: The SciML Common Interface

SciMLBase.jl defines the core interfaces of the SciML libraries, such as the definitions of abstract types like SciMLProblem, along with their instantiations like ODEProblem. While SciMLBase.jl is insufficient to solve any equations, it holds all the equation definitions, and thus downstream libraries which wish to allow for using SciML solvers without depending on any solvers can directly depend on SciMLBase.jl.

SciMLOperators.jl: The AbstractSciMLOperator Interface

SciMLOperators.jl defines the interface for how matrix-free linear and affine operators are defined and used throughout the SciML ecosystem.

DiffEqNoiseProcess.jl: The SciML Common Noise Interface

DiffEqNoiseProcess.jl defines the common interface for stochastic noise processes used by the equation solvers of the SciML ecosystem.

CommonSolve.jl: The Common Definition of Solve

CommonSolve.jl is the library that defines the solve, solve!, and init interfaces which are used throughout all the SciML equation solvers. It's defined as an extremely lightweight library so that other ecosystems can build on the same solve definition without clashing with SciML when both export.

Static.jl: A Shared Interface for Static Compile-Time Computation

Static.jl is a set of statically parameterized types for performing operations in a statically-defined (compiler-optimized) way with respect to values.

DiffEqBase.jl: A Library of Shared Components for Differential Equation Solvers

DiffEqBase.jl is the core shared component of the DifferentialEquations.jl ecosystem. It's not intended for non-developer users to interface directly with, instead it's used for the common functionality for uniformity of implementation between the solver libraries.

Third-Party Libraries to Note

ArrayInterface.jl: Extensions to the Julia AbstractArray Interface

ArrayInterface.jl are traits and functions which extend the Julia Base AbstractArray interface, giving a much larger set of queries to allow for writing high-performance generic code over all array types. For example, functions include can_change_size to know if an AbstractArray type is compatible with resize!, fast_scalar_indexing to know whether direct scalar indexing A[i] is optimized, and functions like findstructralnz to get the structural non-zeros of arbitrary sparse and structured matrices.

Adapt.jl: Conversion to Allow Chip-Generic Programs

Adapt.jl makes it possible to write code that is generic to the compute devices, i.e. code that works on both CPUs and GPUs. It defines the adapt function which acts like convert(T, x), but without the restriction of returning a T. This allows you to “convert” wrapper types, like Adjoint to be GPU compatible (for example) without throwing away the wrapper.

Example usage:

adapt(CuArray, ::Adjoint{Array})::Adjoint{CuArray}

AbstractFFTs.jl: High Level Shared Interface for Fast Fourier Transformation Libraries

AbstractFFTs.jl defines the common interface for Fast Fourier Transformations (FFTs) in Julia. Similar to SciMLBase.jl, AbstractFFTs.jl is not a solver library but instead a shared API which is extended by solver libraries such as FFTW.jl. Code written using AbstractFFTs.jl can be made compatible with FFT libraries without having an explicit dependency on a solver.

GPUArrays.jl: Common Interface for GPU-Based Array Types

GPUArrays.jl defines the shared higher-level operations for GPU-based array types, like CUDA.jl's CuArray and AMDGPU.jl's ROCmArray. Packages in SciML use the designation x isa AbstractGPUArray in order to find out if a user's operation is on the GPU and specialize computations.

RecipesBase.jl: Standard Plotting Recipe Interface

RecipesBase.jl defines the common interface for plotting recipes, composable transformations of Julia data types into simpler data types for visualization with libraries such as Plots.jl and Makie.jl. SciML libraries attempt to always include plot recipes wherever possible for ease of visualization.

Tables.jl: Common Interface for Tabular Data Types

Tables.jl is a common interface for defining tabular data structures, such as DataFrames.jl. SciML's libraries extend the Tables.jl interface to allow for automated conversions into data frame libraries without explicit dependence on any singular implementation.

+The SciML Interface Libraries · Overview of Julia's SciML
GitHub

The SciML Interface Libraries

SciMLBase.jl: The SciML Common Interface

SciMLBase.jl defines the core interfaces of the SciML libraries, such as the definitions of abstract types like SciMLProblem, along with their instantiations like ODEProblem. While SciMLBase.jl is insufficient to solve any equations, it holds all the equation definitions, and thus downstream libraries which wish to allow for using SciML solvers without depending on any solvers can directly depend on SciMLBase.jl.

SciMLOperators.jl: The AbstractSciMLOperator Interface

SciMLOperators.jl defines the interface for how matrix-free linear and affine operators are defined and used throughout the SciML ecosystem.

DiffEqNoiseProcess.jl: The SciML Common Noise Interface

DiffEqNoiseProcess.jl defines the common interface for stochastic noise processes used by the equation solvers of the SciML ecosystem.

CommonSolve.jl: The Common Definition of Solve

CommonSolve.jl is the library that defines the solve, solve!, and init interfaces which are used throughout all the SciML equation solvers. It's defined as an extremely lightweight library so that other ecosystems can build on the same solve definition without clashing with SciML when both export.

Static.jl: A Shared Interface for Static Compile-Time Computation

Static.jl is a set of statically parameterized types for performing operations in a statically-defined (compiler-optimized) way with respect to values.

DiffEqBase.jl: A Library of Shared Components for Differential Equation Solvers

DiffEqBase.jl is the core shared component of the DifferentialEquations.jl ecosystem. It's not intended for non-developer users to interface directly with, instead it's used for the common functionality for uniformity of implementation between the solver libraries.

Third-Party Libraries to Note

ArrayInterface.jl: Extensions to the Julia AbstractArray Interface

ArrayInterface.jl are traits and functions which extend the Julia Base AbstractArray interface, giving a much larger set of queries to allow for writing high-performance generic code over all array types. For example, functions include can_change_size to know if an AbstractArray type is compatible with resize!, fast_scalar_indexing to know whether direct scalar indexing A[i] is optimized, and functions like findstructralnz to get the structural non-zeros of arbitrary sparse and structured matrices.

Adapt.jl: Conversion to Allow Chip-Generic Programs

Adapt.jl makes it possible to write code that is generic to the compute devices, i.e. code that works on both CPUs and GPUs. It defines the adapt function which acts like convert(T, x), but without the restriction of returning a T. This allows you to “convert” wrapper types, like Adjoint to be GPU compatible (for example) without throwing away the wrapper.

Example usage:

adapt(CuArray, ::Adjoint{Array})::Adjoint{CuArray}

AbstractFFTs.jl: High Level Shared Interface for Fast Fourier Transformation Libraries

AbstractFFTs.jl defines the common interface for Fast Fourier Transformations (FFTs) in Julia. Similar to SciMLBase.jl, AbstractFFTs.jl is not a solver library but instead a shared API which is extended by solver libraries such as FFTW.jl. Code written using AbstractFFTs.jl can be made compatible with FFT libraries without having an explicit dependency on a solver.

GPUArrays.jl: Common Interface for GPU-Based Array Types

GPUArrays.jl defines the shared higher-level operations for GPU-based array types, like CUDA.jl's CuArray and AMDGPU.jl's ROCmArray. Packages in SciML use the designation x isa AbstractGPUArray in order to find out if a user's operation is on the GPU and specialize computations.

RecipesBase.jl: Standard Plotting Recipe Interface

RecipesBase.jl defines the common interface for plotting recipes, composable transformations of Julia data types into simpler data types for visualization with libraries such as Plots.jl and Makie.jl. SciML libraries attempt to always include plot recipes wherever possible for ease of visualization.

Tables.jl: Common Interface for Tabular Data Types

Tables.jl is a common interface for defining tabular data structures, such as DataFrames.jl. SciML's libraries extend the Tables.jl interface to allow for automated conversions into data frame libraries without explicit dependence on any singular implementation.

diff --git a/dev/highlevels/inverse_problems/index.html b/dev/highlevels/inverse_problems/index.html index 4804ecf88ea..4c71777b592 100644 --- a/dev/highlevels/inverse_problems/index.html +++ b/dev/highlevels/inverse_problems/index.html @@ -1,2 +1,2 @@ -Parameter Estimation, Bayesian Analysis, and Inverse Problems · Overview of Julia's SciML
GitHub

Parameter Estimation, Bayesian Analysis, and Inverse Problems

Parameter estimation for models and equations, also known as dynamic data analysis, solving the inverse problem, or Bayesian posterior estimation (when done probabilistically), is provided by the SciML tools for the equations in its set. In this introduction, we briefly present the relevant packages that facilitate parameter estimation, namely:

We also provide information regarding the respective strengths of these packages so that you can easily decide which one suits your needs best.

SciMLSensitivity.jl: Local Sensitivity Analysis and Automatic Differentiation Support for Solvers

SciMLSensitivity.jl is the system for local sensitivity, which all other inverse problem methods rely on. This package defines the interactions between the equation solvers and automatic differentiation, defining fast overloads for forward and adjoint (reverse) sensitivity analysis for fast gradient and Jacobian calculations with respect to model inputs. Its documentation covers how to use direct differentiation of equation solvers in conjunction with tools like Optimization.jl to perform model calibration of ODEs against data, PDE-constrained optimization, nonlinear optimal controls analysis, and much more. As a lower level tool, this library is very versatile, feature-rich, and high-performance, giving all the tools required but not directly providing a higher level interface.

Note

Sensitivity analysis is kept in a separate library from the solvers (SciMLSensitivity.jl), in order to not require all equation solvers to have a dependency on all automatic differentiation libraries. If automatic differentiation is applied to a solver library without importing SciMLSensitivity.jl, an error is thrown letting the user know to import SciMLSensitivity.jl for the functionality to exist.

DataDrivenDiffEq.jl: Data-Driven Modeling and Equation Discovery

The distinguishing feature of this package is that its ultimate goal is to identify the differential equation model that generated the input data. Depending on the user's needs, the package can provide structural identification of a given differential equation (output in a symbolic form) or structural estimation (output as a function for prediction purposes).

DiffEqParamEstim.jl: Simplified Parameter Estimation Interface

This package is for simplified parameter estimation. While not as flexible of a system like DiffEqFlux.jl, it provides ready-made functions for doing standard optimization procedures like L2 fitting and MAP estimates. Among other features, it allows for the optimization of parameters in ODEs, stochastic problems, and delay differential equations.

DiffEqBayes.jl: Simplified Bayesian Estimation Interface

As the name suggests, this package has been designed to provide the estimation of differential equations parameters by Bayesian methods. It works in conjunction with Turing.jl, CmdStan.jl, DynamicHMC.jl, and ApproxBayes.jl. While not as flexible as direct usage of DiffEqFlux.jl or Turing.jl, DiffEqBayes.jl can be an approachable interface for those not familiar with Bayesian estimation, and provides a nice way to use Stan from pure Julia.

Third-Party Tools of Note

Turing.jl: A Flexible Probabilistic Programming Language for Bayesian Analysis

In the context of differential equations and parameter estimation, Turing.jl allows for a Bayesian estimation of differential equations (used in conjunction with the high-level package DiffEqBayes.jl). For more examples on combining Turing.jl with DiffEqBayes.jl, see the documentation below. It is important to note that Turing.jl can also perform Bayesian estimation without relying on DiffEqBayes.jl (for an example, consult this tutorial).

Topopt.jl: Topology Optimization in Julia

Topopt.jl solves topology optimization problems which are inverse problems on partial differential equations, solving for an optimal domain.

Recommended Automatic Differentiation Libraries

Solving inverse problems commonly requires using automatic differentiation (AD). SciML includes extensive support for automatic differentiation throughout its solvers, though some AD libraries are more tested than others. The following libraries are the current recommendations of the SciML developers.

ForwardDiff.jl: Operator-Overloading Forward Mode Automatic Differentiation

ForwardDiff.jl is a library for operator-overloading based forward-mode automatic differentiation. It's commonly used as the default method for generating Jacobians throughout the SciML solver libraries.

Note

Because ForwardDiff.jl uses an operator overloading approach, uses of ForwardDiff.jl require that any caches for non-allocating mutating code allows for Dual numbers. To allow such code to be ForwardDiff.jl-compatible, see PreallocationTools.jl.

Enzyme.jl: LLVM-Level Forward and Reverse Mode Automatic Differentiation

Enzyme.jl is an LLVM-level AD library for forward and reverse automatic differentiation. It supports many features required for high performance, such as being able to differentiate mutating and interleave compiler optimization with the AD passes. However, it does not support all of the Julia runtime, and thus some code with many dynamic behaviors and garbage collection (GC) invocations can be incompatible with Enzyme. Enzyme.jl is quickly becoming the new standard AD for SciML.

Zygote.jl: Julia-Level Source-to-Source Reverse Mode Automatic Differentiation

Zygote.jl is the current standard user-level reverse-mode automatic differentiation library for the SciML solvers. User-level means that many library tutorials, like in SciMLSensitivity.jl and DiffEqFlux.jl, showcase user code using Zygote.jl. This is because Zygote.jl is the AD engine associated with the Flux machine learning library. However, Zygote.jl has many limitations which limits its performance in equation solver contexts, such as an inability to handle mutation and introducing many small allocations and type-instabilities. For this reason, the SciML equation solvers define differentiation overloads using ChainRules.jl, meaning that the equation solvers tend not to use Zygote.jl internally even if the user code uses Zygote.gradient. In this manner, the speed and performance of more advanced techniques can be preserved while using the Julia standard.

FiniteDiff.jl: Fast Finite Difference Approximations

FiniteDiff.jl is the preferred fallback library for numerical differentiation and is commonly used by SciML solver libraries when automatic differentiation is disabled.

SparseDiffTools.jl: Tools for Fast Automatic Differentiation with Sparse Operators

SparseDiffTools.jl is a library for sparse automatic differentiation. It's used internally by many of the SciML equation solver libraries, which explicitly expose interfaces for colorvec color vectors generated by SparseDiffTools.jl's methods. SparseDiffTools.jl also includes many features useful to users, such as operators for matrix-free Jacobian-vector and Hessian-vector products.

+Parameter Estimation, Bayesian Analysis, and Inverse Problems · Overview of Julia's SciML
GitHub

Parameter Estimation, Bayesian Analysis, and Inverse Problems

Parameter estimation for models and equations, also known as dynamic data analysis, solving the inverse problem, or Bayesian posterior estimation (when done probabilistically), is provided by the SciML tools for the equations in its set. In this introduction, we briefly present the relevant packages that facilitate parameter estimation, namely:

We also provide information regarding the respective strengths of these packages so that you can easily decide which one suits your needs best.

SciMLSensitivity.jl: Local Sensitivity Analysis and Automatic Differentiation Support for Solvers

SciMLSensitivity.jl is the system for local sensitivity, which all other inverse problem methods rely on. This package defines the interactions between the equation solvers and automatic differentiation, defining fast overloads for forward and adjoint (reverse) sensitivity analysis for fast gradient and Jacobian calculations with respect to model inputs. Its documentation covers how to use direct differentiation of equation solvers in conjunction with tools like Optimization.jl to perform model calibration of ODEs against data, PDE-constrained optimization, nonlinear optimal controls analysis, and much more. As a lower level tool, this library is very versatile, feature-rich, and high-performance, giving all the tools required but not directly providing a higher level interface.

Note

Sensitivity analysis is kept in a separate library from the solvers (SciMLSensitivity.jl), in order to not require all equation solvers to have a dependency on all automatic differentiation libraries. If automatic differentiation is applied to a solver library without importing SciMLSensitivity.jl, an error is thrown letting the user know to import SciMLSensitivity.jl for the functionality to exist.

DataDrivenDiffEq.jl: Data-Driven Modeling and Equation Discovery

The distinguishing feature of this package is that its ultimate goal is to identify the differential equation model that generated the input data. Depending on the user's needs, the package can provide structural identification of a given differential equation (output in a symbolic form) or structural estimation (output as a function for prediction purposes).

DiffEqParamEstim.jl: Simplified Parameter Estimation Interface

This package is for simplified parameter estimation. While not as flexible of a system like DiffEqFlux.jl, it provides ready-made functions for doing standard optimization procedures like L2 fitting and MAP estimates. Among other features, it allows for the optimization of parameters in ODEs, stochastic problems, and delay differential equations.

DiffEqBayes.jl: Simplified Bayesian Estimation Interface

As the name suggests, this package has been designed to provide the estimation of differential equations parameters by Bayesian methods. It works in conjunction with Turing.jl, CmdStan.jl, DynamicHMC.jl, and ApproxBayes.jl. While not as flexible as direct usage of DiffEqFlux.jl or Turing.jl, DiffEqBayes.jl can be an approachable interface for those not familiar with Bayesian estimation, and provides a nice way to use Stan from pure Julia.

Third-Party Tools of Note

Turing.jl: A Flexible Probabilistic Programming Language for Bayesian Analysis

In the context of differential equations and parameter estimation, Turing.jl allows for a Bayesian estimation of differential equations (used in conjunction with the high-level package DiffEqBayes.jl). For more examples on combining Turing.jl with DiffEqBayes.jl, see the documentation below. It is important to note that Turing.jl can also perform Bayesian estimation without relying on DiffEqBayes.jl (for an example, consult this tutorial).

Topopt.jl: Topology Optimization in Julia

Topopt.jl solves topology optimization problems which are inverse problems on partial differential equations, solving for an optimal domain.

Recommended Automatic Differentiation Libraries

Solving inverse problems commonly requires using automatic differentiation (AD). SciML includes extensive support for automatic differentiation throughout its solvers, though some AD libraries are more tested than others. The following libraries are the current recommendations of the SciML developers.

ForwardDiff.jl: Operator-Overloading Forward Mode Automatic Differentiation

ForwardDiff.jl is a library for operator-overloading based forward-mode automatic differentiation. It's commonly used as the default method for generating Jacobians throughout the SciML solver libraries.

Note

Because ForwardDiff.jl uses an operator overloading approach, uses of ForwardDiff.jl require that any caches for non-allocating mutating code allows for Dual numbers. To allow such code to be ForwardDiff.jl-compatible, see PreallocationTools.jl.

Enzyme.jl: LLVM-Level Forward and Reverse Mode Automatic Differentiation

Enzyme.jl is an LLVM-level AD library for forward and reverse automatic differentiation. It supports many features required for high performance, such as being able to differentiate mutating and interleave compiler optimization with the AD passes. However, it does not support all of the Julia runtime, and thus some code with many dynamic behaviors and garbage collection (GC) invocations can be incompatible with Enzyme. Enzyme.jl is quickly becoming the new standard AD for SciML.

Zygote.jl: Julia-Level Source-to-Source Reverse Mode Automatic Differentiation

Zygote.jl is the current standard user-level reverse-mode automatic differentiation library for the SciML solvers. User-level means that many library tutorials, like in SciMLSensitivity.jl and DiffEqFlux.jl, showcase user code using Zygote.jl. This is because Zygote.jl is the AD engine associated with the Flux machine learning library. However, Zygote.jl has many limitations which limits its performance in equation solver contexts, such as an inability to handle mutation and introducing many small allocations and type-instabilities. For this reason, the SciML equation solvers define differentiation overloads using ChainRules.jl, meaning that the equation solvers tend not to use Zygote.jl internally even if the user code uses Zygote.gradient. In this manner, the speed and performance of more advanced techniques can be preserved while using the Julia standard.

FiniteDiff.jl: Fast Finite Difference Approximations

FiniteDiff.jl is the preferred fallback library for numerical differentiation and is commonly used by SciML solver libraries when automatic differentiation is disabled.

SparseDiffTools.jl: Tools for Fast Automatic Differentiation with Sparse Operators

SparseDiffTools.jl is a library for sparse automatic differentiation. It's used internally by many of the SciML equation solver libraries, which explicitly expose interfaces for colorvec color vectors generated by SparseDiffTools.jl's methods. SparseDiffTools.jl also includes many features useful to users, such as operators for matrix-free Jacobian-vector and Hessian-vector products.

diff --git a/dev/highlevels/learning_resources/index.html b/dev/highlevels/learning_resources/index.html index 59220719033..ec9bf357384 100644 --- a/dev/highlevels/learning_resources/index.html +++ b/dev/highlevels/learning_resources/index.html @@ -1,2 +1,2 @@ -Curated Learning, Teaching, and Training Resources · Overview of Julia's SciML
GitHub

Curated Learning, Teaching, and Training Resources

While the SciML documentation is made to be comprehensive, there will always be good alternative resources. The purpose of this section of the documentation is to highlight the alternative resources which can be helpful for learning how to use the SciML Open-Source Software libraries.

JuliaCon and SciMLCon Videos

Many tutorials and introductions to packages have been taught through previous JuliaCon/SciMLCon workshops and talks. The following is a curated list of such training videos:

SciML Book: Parallel Computing and Scientific Machine Learning (SciML): Methods and Applications

The book Parallel Computing and Scientific Machine Learning (SciML): Methods and Applications is a compilation of the lecture notes from the MIT Course 18.337J/6.338J: Parallel Computing and Scientific Machine Learning. It contains a walkthrough of many of the methods implemented in the SciML libraries, as well as how to understand much of the functionality at a deeper level. This course was intended for MIT graduate students in engineering, computer science, and mathematics and thus may have a high prerequisite requirement than many other resources.

sir-julia: Various implementations of the classical SIR model in Julia

For those who like to learn by example, the repository sir-julia is a great resource! It showcases how to use the SciML libraries in many different ways to simulate different variations of the classic SIR epidemic model.

Other Books Featuring SciML

+Curated Learning, Teaching, and Training Resources · Overview of Julia's SciML
GitHub

Curated Learning, Teaching, and Training Resources

While the SciML documentation is made to be comprehensive, there will always be good alternative resources. The purpose of this section of the documentation is to highlight the alternative resources which can be helpful for learning how to use the SciML Open-Source Software libraries.

JuliaCon and SciMLCon Videos

Many tutorials and introductions to packages have been taught through previous JuliaCon/SciMLCon workshops and talks. The following is a curated list of such training videos:

SciML Book: Parallel Computing and Scientific Machine Learning (SciML): Methods and Applications

The book Parallel Computing and Scientific Machine Learning (SciML): Methods and Applications is a compilation of the lecture notes from the MIT Course 18.337J/6.338J: Parallel Computing and Scientific Machine Learning. It contains a walkthrough of many of the methods implemented in the SciML libraries, as well as how to understand much of the functionality at a deeper level. This course was intended for MIT graduate students in engineering, computer science, and mathematics and thus may have a high prerequisite requirement than many other resources.

sir-julia: Various implementations of the classical SIR model in Julia

For those who like to learn by example, the repository sir-julia is a great resource! It showcases how to use the SciML libraries in many different ways to simulate different variations of the classic SIR epidemic model.

Other Books Featuring SciML

diff --git a/dev/highlevels/model_libraries_and_importers/index.html b/dev/highlevels/model_libraries_and_importers/index.html index 12a98ccef0f..0bf16081ff6 100644 --- a/dev/highlevels/model_libraries_and_importers/index.html +++ b/dev/highlevels/model_libraries_and_importers/index.html @@ -1,2 +1,2 @@ -Model Libraries and Importers · Overview of Julia's SciML
GitHub

Model Libraries and Importers

Models are passed on from generation to generation. Many models are not built from scratch but have a legacy of the known physics, biology, and chemistry embedded into them. Julia's SciML offers a range of pre-built modeling tools, from reusable acausal components to direct imports from common file formats.

ModelingToolkitStandardLibrary.jl: A Standard Library for ModelingToolkit

Given the composable nature of acausal modeling systems, it's helpful to not have to define every component from scratch and instead build off a common base of standard components. ModelingToolkitStandardLibrary.jl is that library. It provides components for standard models to start building everything from circuits and engines to robots.

DiffEqCallbacks.jl: Pre-Made Callbacks for DifferentialEquations.jl

DiffEqCallbacks.jl has many event handling and callback definitions which allow for quickly building up complex differential equation models. It includes:

  • Callbacks for specialized output and saving procedures
  • Callbacks for enforcing domain constraints, positivity, and manifolds
  • Timed callbacks for periodic dosing, presetting of tstops, and more
  • Callbacks for determining and terminating at steady state
  • Callbacks for controlling stepsizes and enforcing CFL conditions
  • Callbacks for quantifying uncertainty with respect to numerical errors

SBMLToolkit.jl: SBML Import

SBMLToolkit.jl is a library for reading SBML files into the standard formats for Catalyst.jl and ModelingToolkit.jl. There are well over one thousand biological models available in the BioModels Repository.

CellMLToolkit.jl: CellML Import

CellMLToolkit.jl is a library for reading CellML files into the standard formats for ModelingToolkit.jl. There are several hundred biological models available in the CellML Model Repository.

ReactionNetworkImporters.jl: BioNetGen Import

ReactionNetworkImporters.jl is a library for reading BioNetGen .net files and various stoichiometry matrix representations into the standard formats for Catalyst.jl and ModelingToolkit.jl.

+Model Libraries and Importers · Overview of Julia's SciML
GitHub

Model Libraries and Importers

Models are passed on from generation to generation. Many models are not built from scratch but have a legacy of the known physics, biology, and chemistry embedded into them. Julia's SciML offers a range of pre-built modeling tools, from reusable acausal components to direct imports from common file formats.

ModelingToolkitStandardLibrary.jl: A Standard Library for ModelingToolkit

Given the composable nature of acausal modeling systems, it's helpful to not have to define every component from scratch and instead build off a common base of standard components. ModelingToolkitStandardLibrary.jl is that library. It provides components for standard models to start building everything from circuits and engines to robots.

DiffEqCallbacks.jl: Pre-Made Callbacks for DifferentialEquations.jl

DiffEqCallbacks.jl has many event handling and callback definitions which allow for quickly building up complex differential equation models. It includes:

  • Callbacks for specialized output and saving procedures
  • Callbacks for enforcing domain constraints, positivity, and manifolds
  • Timed callbacks for periodic dosing, presetting of tstops, and more
  • Callbacks for determining and terminating at steady state
  • Callbacks for controlling stepsizes and enforcing CFL conditions
  • Callbacks for quantifying uncertainty with respect to numerical errors

SBMLToolkit.jl: SBML Import

SBMLToolkit.jl is a library for reading SBML files into the standard formats for Catalyst.jl and ModelingToolkit.jl. There are well over one thousand biological models available in the BioModels Repository.

CellMLToolkit.jl: CellML Import

CellMLToolkit.jl is a library for reading CellML files into the standard formats for ModelingToolkit.jl. There are several hundred biological models available in the CellML Model Repository.

ReactionNetworkImporters.jl: BioNetGen Import

ReactionNetworkImporters.jl is a library for reading BioNetGen .net files and various stoichiometry matrix representations into the standard formats for Catalyst.jl and ModelingToolkit.jl.

diff --git a/dev/highlevels/modeling_languages/index.html b/dev/highlevels/modeling_languages/index.html index 28051f1f575..1d906f0677a 100644 --- a/dev/highlevels/modeling_languages/index.html +++ b/dev/highlevels/modeling_languages/index.html @@ -1,2 +1,2 @@ -Modeling Languages · Overview of Julia's SciML
GitHub

Modeling Languages

While in theory one can build perfect code for all models from scratch, in practice many scientists and engineers need or want some help! The SciML modeling tools provide a higher level interface over the equation solver, which helps the translation from good models to good simulations in a way that abstracts away the mathematical and computational details without giving up performance.

ModelingToolkit.jl: Acausal Symbolic Modeling

Acausal modeling is an extension of causal modeling that is more composable and allows for more code reuse. Build a model of an electric engine, then build a model of a battery, and now declare connections by stating "the voltage at the engine equals the voltage at the connector of the battery", and generate the composed model. The tool for this is ModelingToolkit.jl. ModelingToolkit.jl is a sophisticated symbolic modeling library which allows for specifying these types of large-scale differential equation models in a simple way, abstracting away the computational details. However, its symbolic analysis allows for generating much more performant code for differential-algebraic equations than most users could ever write by hand, with its structural_simplify automatically correcting the model to improve parallelism, numerical stability, and automatically remove variables which it can show are redundant.

ModelingToolkit.jl is the base of the SciML symbolic modeling ecosystem, defining the AbstractSystem types, such as ODESystem, SDESystem, OptimizationSystem, PDESystem, and more, which are then used by all the other modeling tools. As such, when using other modeling tools like Catalyst.jl, the reference for all the things that can be done with the symbolic representation is simply ModelingToolkit.jl.

Catalyst.jl: Chemical Reaction Networks (CRN), Systems Biology, and Quantitative Systems Pharmacology (QSP) Modeling

Catalyst.jl is a modeling interface for efficient simulation of mass action ODE, chemical Langevin SDE, and stochastic chemical kinetics jump process (i.e. chemical master equation) models for chemical reaction networks and population processes. It uses a highly intuitive chemical reaction syntax interface, which generates all the extra functionality necessary for the fastest use with JumpProcesses.jl, DifferentialEquations.jl, and higher level SciML libraries. Its ReactionSystem type is a programmable extension of the ModelingToolkit AbstractSystem interface, meaning that complex reaction systems are represented symbolically, and then compiled to optimized representations automatically when converting ReactionSystems to concrete ODE/SDE/jump process representations. Catalyst also provides functionality to support chemical reaction network and steady-state analysis.

For an overview of the library, see Modeling Biochemical Systems with Catalyst.jl - Samuel Isaacson

NBodySimulator.jl: A differentiable simulator for N-body problems, including astrophysical and molecular dynamics

NBodySimulator.jl is a differentiable simulator for N-body problems, including astrophysical and molecular dynamics. It uses the DifferentialEquations.jl solvers, allowing for one to choose between a large variety of symplectic integration schemes. It implements many of the thermostats required for doing standard molecular dynamics approximations.

DiffEqFinancial.jl: Financial models for use in the DifferentialEquations ecosystem

The goal of DiffEqFinancial.jl is to be a feature-complete set of solvers for the types of problems found in libraries like QuantLib, such as the Heston process or the Black-Scholes model.

ParameterizedFunctions.jl: Simple Differential Equation Definitions Made Easy

This image that went viral is actually runnable code from ParameterizedFunctions.jl. Define equations and models using a very simple high-level syntax and let the code generation tools build symbolic fast Jacobian, gradient, etc. functions for you.

Third-Party Tools of Note

MomentClosure.jl: Automated Generation of Moment Closure Equations

MomentClosure.jl is a library for generating the moment closure equations for a given chemical master equation or stochastic differential equation. Thus instead of solving a stochastic model thousands of times to find the mean and variance, this library can generate the deterministic equations for how the mean and variance evolve in order to be solved in a single run. MomentClosure.jl uses Catalyst ReactionSystem and ModelingToolkit SDESystem types as the input for its symbolic generation processes.

Agents.jl: Agent-Based Modeling Framework in Julia

If one wants to do agent-based modeling in Julia, Agents.jl is the go-to library. It's fast and flexible, making it a solid foundation for any agent-based model.

Unitful.jl: A Julia package for physical units

Supports not only SI units, but also any other unit system. Unitful.jl has minimal run-time penalty of units. Includes facilities for dimensional analysis, and integrates easily with the usual mathematical operations and collections that are defined in Julia.

ReactionMechanismSimulator.jl: Simulation and Analysis of Large Chemical Reaction Systems

ReactionMechanismSimulator.jl is a tool for simulating and analyzing large chemical reaction mechanisms. It interfaces with the ReactionMechanismGenerator suite for automatically constructing reaction pathways from chemical components to quickly build realistic models of chemical systems.

FiniteStateProjection.jl: Direct Solution of Chemical Master Equations

FiniteStateProjection.jl is a library for finite state projection direct solving of the chemical master equation. It automatically converts the Catalyst ReactionSystem definitions into ModelingToolkit ODESystem representations for the evolution of probability distributions to allow for directly solving the weak form of the stochastic model.

AlgebraicPetri.jl: Applied Category Theory of Modeling

AlgebraicPetri.jl is a library for automating the intuitive generation of dynamical models using a Category theory-based approach.

QuantumOptics.jl: Simulating quantum systems.

QuantumOptics.jl makes it easy to simulate various kinds of quantum systems. It is inspired by the Quantum Optics Toolbox for MATLAB and the Python framework QuTiP.

+Modeling Languages · Overview of Julia's SciML
GitHub

Modeling Languages

While in theory one can build perfect code for all models from scratch, in practice many scientists and engineers need or want some help! The SciML modeling tools provide a higher level interface over the equation solver, which helps the translation from good models to good simulations in a way that abstracts away the mathematical and computational details without giving up performance.

ModelingToolkit.jl: Acausal Symbolic Modeling

Acausal modeling is an extension of causal modeling that is more composable and allows for more code reuse. Build a model of an electric engine, then build a model of a battery, and now declare connections by stating "the voltage at the engine equals the voltage at the connector of the battery", and generate the composed model. The tool for this is ModelingToolkit.jl. ModelingToolkit.jl is a sophisticated symbolic modeling library which allows for specifying these types of large-scale differential equation models in a simple way, abstracting away the computational details. However, its symbolic analysis allows for generating much more performant code for differential-algebraic equations than most users could ever write by hand, with its structural_simplify automatically correcting the model to improve parallelism, numerical stability, and automatically remove variables which it can show are redundant.

ModelingToolkit.jl is the base of the SciML symbolic modeling ecosystem, defining the AbstractSystem types, such as ODESystem, SDESystem, OptimizationSystem, PDESystem, and more, which are then used by all the other modeling tools. As such, when using other modeling tools like Catalyst.jl, the reference for all the things that can be done with the symbolic representation is simply ModelingToolkit.jl.

Catalyst.jl: Chemical Reaction Networks (CRN), Systems Biology, and Quantitative Systems Pharmacology (QSP) Modeling

Catalyst.jl is a modeling interface for efficient simulation of mass action ODE, chemical Langevin SDE, and stochastic chemical kinetics jump process (i.e. chemical master equation) models for chemical reaction networks and population processes. It uses a highly intuitive chemical reaction syntax interface, which generates all the extra functionality necessary for the fastest use with JumpProcesses.jl, DifferentialEquations.jl, and higher level SciML libraries. Its ReactionSystem type is a programmable extension of the ModelingToolkit AbstractSystem interface, meaning that complex reaction systems are represented symbolically, and then compiled to optimized representations automatically when converting ReactionSystems to concrete ODE/SDE/jump process representations. Catalyst also provides functionality to support chemical reaction network and steady-state analysis.

For an overview of the library, see Modeling Biochemical Systems with Catalyst.jl - Samuel Isaacson

NBodySimulator.jl: A differentiable simulator for N-body problems, including astrophysical and molecular dynamics

NBodySimulator.jl is a differentiable simulator for N-body problems, including astrophysical and molecular dynamics. It uses the DifferentialEquations.jl solvers, allowing for one to choose between a large variety of symplectic integration schemes. It implements many of the thermostats required for doing standard molecular dynamics approximations.

DiffEqFinancial.jl: Financial models for use in the DifferentialEquations ecosystem

The goal of DiffEqFinancial.jl is to be a feature-complete set of solvers for the types of problems found in libraries like QuantLib, such as the Heston process or the Black-Scholes model.

ParameterizedFunctions.jl: Simple Differential Equation Definitions Made Easy

This image that went viral is actually runnable code from ParameterizedFunctions.jl. Define equations and models using a very simple high-level syntax and let the code generation tools build symbolic fast Jacobian, gradient, etc. functions for you.

Third-Party Tools of Note

MomentClosure.jl: Automated Generation of Moment Closure Equations

MomentClosure.jl is a library for generating the moment closure equations for a given chemical master equation or stochastic differential equation. Thus instead of solving a stochastic model thousands of times to find the mean and variance, this library can generate the deterministic equations for how the mean and variance evolve in order to be solved in a single run. MomentClosure.jl uses Catalyst ReactionSystem and ModelingToolkit SDESystem types as the input for its symbolic generation processes.

Agents.jl: Agent-Based Modeling Framework in Julia

If one wants to do agent-based modeling in Julia, Agents.jl is the go-to library. It's fast and flexible, making it a solid foundation for any agent-based model.

Unitful.jl: A Julia package for physical units

Supports not only SI units, but also any other unit system. Unitful.jl has minimal run-time penalty of units. Includes facilities for dimensional analysis, and integrates easily with the usual mathematical operations and collections that are defined in Julia.

ReactionMechanismSimulator.jl: Simulation and Analysis of Large Chemical Reaction Systems

ReactionMechanismSimulator.jl is a tool for simulating and analyzing large chemical reaction mechanisms. It interfaces with the ReactionMechanismGenerator suite for automatically constructing reaction pathways from chemical components to quickly build realistic models of chemical systems.

FiniteStateProjection.jl: Direct Solution of Chemical Master Equations

FiniteStateProjection.jl is a library for finite state projection direct solving of the chemical master equation. It automatically converts the Catalyst ReactionSystem definitions into ModelingToolkit ODESystem representations for the evolution of probability distributions to allow for directly solving the weak form of the stochastic model.

AlgebraicPetri.jl: Applied Category Theory of Modeling

AlgebraicPetri.jl is a library for automating the intuitive generation of dynamical models using a Category theory-based approach.

QuantumOptics.jl: Simulating quantum systems.

QuantumOptics.jl makes it easy to simulate various kinds of quantum systems. It is inspired by the Quantum Optics Toolbox for MATLAB and the Python framework QuTiP.

diff --git a/dev/highlevels/numerical_utilities/index.html b/dev/highlevels/numerical_utilities/index.html index 86e590cf84e..8c5fbbe24cc 100644 --- a/dev/highlevels/numerical_utilities/index.html +++ b/dev/highlevels/numerical_utilities/index.html @@ -1,2 +1,2 @@ -SciML Numerical Utility Libraries · Overview of Julia's SciML
GitHub

SciML Numerical Utility Libraries

ExponentialUtilities.jl: Faster Matrix Exponentials

ExponentialUtilities.jl is a library for efficient computation of matrix exponentials. While Julia has a built-in exp(A) method, ExponentialUtilities.jl offers many features around this to improve performance in scientific contexts, including:

  • Faster methods for (non-allocating) matrix exponentials via exponential!
  • Methods for computing matrix exponential that are generic to number types and arrays (i.e. GPUs)
  • Methods for computing Arnoldi iterations on Krylov subspaces
  • Direct computation of exp(t*A)*v, i.e. exponentiation of a matrix times a vector, without computing the matrix exponential
  • Direct computation of ϕ_m(t*A)*v operations, where ϕ_0(z) = exp(z) and ϕ_(k+1)(z) = (ϕ_k(z) - 1) / z

ExponentialUtilities.jl includes complex adaptive time stepping techniques such as KIOPS in order to perform these calculations in a fast and numerically-stable way.

QuasiMonteCarlo.jl: Fast Quasi-Random Number Generation

QuasiMonteCarlo.jl is a library for fast generation of low discrepancy Quasi-Monte Carlo samples, using methods like:

  • GridSample(dx) where the grid is given by lb:dx[i]:ub in the ith direction.
  • UniformSample for uniformly distributed random numbers.
  • SobolSample for the Sobol sequence.
  • LatinHypercubeSample for a Latin Hypercube.
  • LatticeRuleSample for a randomly-shifted rank-1 lattice rule.
  • LowDiscrepancySample(base) where base[i] is the base in the ith direction.
  • GoldenSample for a Golden Ratio sequence.
  • KroneckerSample(alpha, s0) for a Kronecker sequence, where alpha is a length-d vector of irrational numbers (often sqrt(d)) and s0 is a length-d seed vector (often 0).
  • SectionSample(x0, sampler) where sampler is any sampler above and x0 is a vector of either NaN for a free dimension or some scalar for a constrained dimension.

DataInterpolations.jl: One-Dimensional Interpolations

DataInterpolations.jl is a library of one-dimensional interpolation schemes which are composable with automatic differentiation and the SciML ecosystem. It includes direct interpolation methods and regression techniques for handling noisy data. Its methods include:

  • ConstantInterpolation(u,t) - A piecewise constant interpolation.

  • LinearInterpolation(u,t) - A linear interpolation.

  • QuadraticInterpolation(u,t) - A quadratic interpolation.

  • LagrangeInterpolation(u,t,n) - A Lagrange interpolation of order n.

  • QuadraticSpline(u,t) - A quadratic spline interpolation.

  • CubicSpline(u,t) - A cubic spline interpolation.

  • BSplineInterpolation(u,t,d,pVec,knotVec) - An interpolation B-spline. This is a B-spline which hits each of the data points. The argument choices are:

    • d - degree of B-spline
    • pVec - Symbol to Parameters Vector, pVec = :Uniform for uniform spaced parameters and pVec = :ArcLen for parameters generated by chord length method.
    • knotVec - Symbol to Knot Vector, knotVec = :Uniform for uniform knot vector, knotVec = :Average for average spaced knot vector.

  • BSplineApprox(u,t,d,h,pVec,knotVec) - A regression B-spline which smooths the fitting curve. The argument choices are the same as the BSplineInterpolation, with the additional parameter h<length(t) which is the number of control points to use, with smaller h indicating more smoothing.

  • Curvefit(u,t,m,p,alg) - An interpolation which is done by fitting a user-given functional form m(t,p) where p is the vector of parameters. The user's input p is an initial value for a least-square fitting, alg is the algorithm choice used to optimize the cost function (sum of squared deviations) via Optim.jl and optimal ps are used in the interpolation.

These interpolations match the SciML interfaces and have direct support for packages like ModelingToolkit.jl.

PoissonRandom.jl: Fast Poisson Random Number Generation

PoissonRandom.jl is just fast Poisson random number generation for Poisson processes, like chemical master equations.

PreallocationTools.jl: Write Non-Allocating Code Easier

PreallocationTools.jl is a library of tools for writing non-allocating code that interacts well with advanced features like automatic differentiation and symbolics.

RuntimeGeneratedFunctions.jl: Efficient Staged Programming in Julia

RuntimeGeneratedFunctions.jl allows for staged programming in Julia, compiling functions at runtime with full optimizations. This is used by many libraries such as ModelingToolkit.jl to allow for runtime code generation for improved performance.

EllipsisNotation.jl: Implementation of Ellipsis Array Slicing

EllipsisNotation.jl defines the ellipsis array slicing notation for Julia. It uses .. as a catch-all for “all dimensions”, allowing for indexing like [..,1] to mean [:,:,:,1] on four dimensional arrays, in a way that is generic to the number of dimensions in the underlying array.

Third-Party Libraries to Note

Distributions.jl: Representations of Probability Distributions

Distributions.jl is a library for defining distributions in Julia. It's used all throughout the SciML libraries for specifications of probability distributions.

Note

For full compatibility with automatic differentiation, see DistributionsAD.jl

FFTW.jl: Fastest Fourier Transformation in the West

FFTW.jl is the preferred library for fast Fourier Transformations on the CPU.

SpecialFunctions.jl: Implementations of Mathematical Special Functions

SpecialFunctions.jl is a library of implementations of special functions, like Bessel functions and error functions (erf). This library is compatible with automatic differentiation.

LoopVectorization.jl: Automated Loop Accelerator

LoopVectorization.jl is a library which provides the @turbo and @tturbo macros for accelerating the computation of loops. This can be used to accelerating the model functions sent to the equation solvers, for example, accelerating handwritten PDE discretizations.

Polyester.jl: Cheap Threads

Polyester.jl is a cheaper version of threads for Julia, which use a set pool of threads for lower overhead. Note that Polyester does not compose with the standard Julia composable threading infrastructure, and thus one must take care not to compose two levels of Polyester, as this will oversubscribe the computation and lead to performance degradation. Many SciML solvers have options to use Polyester for threading to achieve the top performance.

Tullio.jl: Fast Tensor Calculations and Einstein Notation

Tullio.jl is a library for fast tensor calculations with Einstein notation. It allows for defining operations which are compatible with automatic differentiation, GPUs, and more.

ParallelStencil.jl: High-Level Code for Parallelized Stencil Computations

ParallelStencil.jl is a library for writing high-level code for parallelized stencil computations. It is compatible with SciML equation solvers and is thus a good way to generate GPU and distributed parallel model code.

Julia Utilities

StaticCompiler.jl

StaticCompiler.jl is a package for generating static binaries from Julia code. It only supports a subset of Julia, so not all equation solver algorithms are compatible with StaticCompiler.jl.

PackageCompiler.jl

PackageCompiler.jl is a package for generating shared libraries from Julia code. It builds the entirety of Julia by bundling a system image with the Julia runtime. It thus builds complete binaries that can hold all the functionality of SciML. Furthermore, it can also be used to generate new system images to decrease startup times and remove JIT-compilation from SciML usage.

+SciML Numerical Utility Libraries · Overview of Julia's SciML
GitHub

SciML Numerical Utility Libraries

ExponentialUtilities.jl: Faster Matrix Exponentials

ExponentialUtilities.jl is a library for efficient computation of matrix exponentials. While Julia has a built-in exp(A) method, ExponentialUtilities.jl offers many features around this to improve performance in scientific contexts, including:

  • Faster methods for (non-allocating) matrix exponentials via exponential!
  • Methods for computing matrix exponential that are generic to number types and arrays (i.e. GPUs)
  • Methods for computing Arnoldi iterations on Krylov subspaces
  • Direct computation of exp(t*A)*v, i.e. exponentiation of a matrix times a vector, without computing the matrix exponential
  • Direct computation of ϕ_m(t*A)*v operations, where ϕ_0(z) = exp(z) and ϕ_(k+1)(z) = (ϕ_k(z) - 1) / z

ExponentialUtilities.jl includes complex adaptive time stepping techniques such as KIOPS in order to perform these calculations in a fast and numerically-stable way.

QuasiMonteCarlo.jl: Fast Quasi-Random Number Generation

QuasiMonteCarlo.jl is a library for fast generation of low discrepancy Quasi-Monte Carlo samples, using methods like:

  • GridSample(dx) where the grid is given by lb:dx[i]:ub in the ith direction.
  • UniformSample for uniformly distributed random numbers.
  • SobolSample for the Sobol sequence.
  • LatinHypercubeSample for a Latin Hypercube.
  • LatticeRuleSample for a randomly-shifted rank-1 lattice rule.
  • LowDiscrepancySample(base) where base[i] is the base in the ith direction.
  • GoldenSample for a Golden Ratio sequence.
  • KroneckerSample(alpha, s0) for a Kronecker sequence, where alpha is a length-d vector of irrational numbers (often sqrt(d)) and s0 is a length-d seed vector (often 0).
  • SectionSample(x0, sampler) where sampler is any sampler above and x0 is a vector of either NaN for a free dimension or some scalar for a constrained dimension.

DataInterpolations.jl: One-Dimensional Interpolations

DataInterpolations.jl is a library of one-dimensional interpolation schemes which are composable with automatic differentiation and the SciML ecosystem. It includes direct interpolation methods and regression techniques for handling noisy data. Its methods include:

  • ConstantInterpolation(u,t) - A piecewise constant interpolation.

  • LinearInterpolation(u,t) - A linear interpolation.

  • QuadraticInterpolation(u,t) - A quadratic interpolation.

  • LagrangeInterpolation(u,t,n) - A Lagrange interpolation of order n.

  • QuadraticSpline(u,t) - A quadratic spline interpolation.

  • CubicSpline(u,t) - A cubic spline interpolation.

  • BSplineInterpolation(u,t,d,pVec,knotVec) - An interpolation B-spline. This is a B-spline which hits each of the data points. The argument choices are:

    • d - degree of B-spline
    • pVec - Symbol to Parameters Vector, pVec = :Uniform for uniform spaced parameters and pVec = :ArcLen for parameters generated by chord length method.
    • knotVec - Symbol to Knot Vector, knotVec = :Uniform for uniform knot vector, knotVec = :Average for average spaced knot vector.

  • BSplineApprox(u,t,d,h,pVec,knotVec) - A regression B-spline which smooths the fitting curve. The argument choices are the same as the BSplineInterpolation, with the additional parameter h<length(t) which is the number of control points to use, with smaller h indicating more smoothing.

  • Curvefit(u,t,m,p,alg) - An interpolation which is done by fitting a user-given functional form m(t,p) where p is the vector of parameters. The user's input p is an initial value for a least-square fitting, alg is the algorithm choice used to optimize the cost function (sum of squared deviations) via Optim.jl and optimal ps are used in the interpolation.

These interpolations match the SciML interfaces and have direct support for packages like ModelingToolkit.jl.

PoissonRandom.jl: Fast Poisson Random Number Generation

PoissonRandom.jl is just fast Poisson random number generation for Poisson processes, like chemical master equations.

PreallocationTools.jl: Write Non-Allocating Code Easier

PreallocationTools.jl is a library of tools for writing non-allocating code that interacts well with advanced features like automatic differentiation and symbolics.

RuntimeGeneratedFunctions.jl: Efficient Staged Programming in Julia

RuntimeGeneratedFunctions.jl allows for staged programming in Julia, compiling functions at runtime with full optimizations. This is used by many libraries such as ModelingToolkit.jl to allow for runtime code generation for improved performance.

EllipsisNotation.jl: Implementation of Ellipsis Array Slicing

EllipsisNotation.jl defines the ellipsis array slicing notation for Julia. It uses .. as a catch-all for “all dimensions”, allowing for indexing like [..,1] to mean [:,:,:,1] on four dimensional arrays, in a way that is generic to the number of dimensions in the underlying array.

Third-Party Libraries to Note

Distributions.jl: Representations of Probability Distributions

Distributions.jl is a library for defining distributions in Julia. It's used all throughout the SciML libraries for specifications of probability distributions.

Note

For full compatibility with automatic differentiation, see DistributionsAD.jl

FFTW.jl: Fastest Fourier Transformation in the West

FFTW.jl is the preferred library for fast Fourier Transformations on the CPU.

SpecialFunctions.jl: Implementations of Mathematical Special Functions

SpecialFunctions.jl is a library of implementations of special functions, like Bessel functions and error functions (erf). This library is compatible with automatic differentiation.

LoopVectorization.jl: Automated Loop Accelerator

LoopVectorization.jl is a library which provides the @turbo and @tturbo macros for accelerating the computation of loops. This can be used to accelerating the model functions sent to the equation solvers, for example, accelerating handwritten PDE discretizations.

Polyester.jl: Cheap Threads

Polyester.jl is a cheaper version of threads for Julia, which use a set pool of threads for lower overhead. Note that Polyester does not compose with the standard Julia composable threading infrastructure, and thus one must take care not to compose two levels of Polyester, as this will oversubscribe the computation and lead to performance degradation. Many SciML solvers have options to use Polyester for threading to achieve the top performance.

Tullio.jl: Fast Tensor Calculations and Einstein Notation

Tullio.jl is a library for fast tensor calculations with Einstein notation. It allows for defining operations which are compatible with automatic differentiation, GPUs, and more.

ParallelStencil.jl: High-Level Code for Parallelized Stencil Computations

ParallelStencil.jl is a library for writing high-level code for parallelized stencil computations. It is compatible with SciML equation solvers and is thus a good way to generate GPU and distributed parallel model code.

Julia Utilities

StaticCompiler.jl

StaticCompiler.jl is a package for generating static binaries from Julia code. It only supports a subset of Julia, so not all equation solver algorithms are compatible with StaticCompiler.jl.

PackageCompiler.jl

PackageCompiler.jl is a package for generating shared libraries from Julia code. It builds the entirety of Julia by bundling a system image with the Julia runtime. It thus builds complete binaries that can hold all the functionality of SciML. Furthermore, it can also be used to generate new system images to decrease startup times and remove JIT-compilation from SciML usage.

diff --git a/dev/highlevels/parameter_analysis/index.html b/dev/highlevels/parameter_analysis/index.html index c8e450c87f2..0931fae2383 100644 --- a/dev/highlevels/parameter_analysis/index.html +++ b/dev/highlevels/parameter_analysis/index.html @@ -1,2 +1,2 @@ -Parameter Analysis Utilities · Overview of Julia's SciML
GitHub

Parameter Analysis Utilities

GlobalSensitivity.jl: Global Sensitivity Analysis

Derivatives calculate the local sensitivity of a model, i.e. the change in the simulation's outcome if one were to change the parameter with respect to some chosen part of the parameter space. But how does a simulation's output change “in general” with respect to a given parameter? That is what global sensitivity analysis (GSA) computes, and thus GlobalSensitivity.jl is the way to answer that question. GlobalSensitivity.jl includes a wide array of methods, including:

  • Morris's method
  • Sobol's method
  • Regression methods (PCC, SRC, Pearson)
  • eFAST
  • Delta Moment-Independent method
  • Derivative-based Global Sensitivity Measures (DGSM)
  • EASI
  • Fractional Factorial method
  • Random Balance Design FAST method

StructuralIdentifiability.jl: Identifiability Analysis Made Simple

Performing parameter estimation from a data set means attempting to recover parameters like reaction rates by fitting some model to the data. But how do you know whether you have enough data to even consider getting the “correct” parameters back? StructuralIdentifiability.jl allows for running a structural identifiability analysis on a given model to determine whether it's theoretically possible to recover the correct parameters. It can state whether a given type of output data can be used to globally recover the parameters (i.e. only a unique parameter set for the model produces a given output), whether the parameters are only locally identifiable (i.e. there are finitely many parameter sets which could generate the seen data), or whether it's unidentifiable (there are infinitely many parameters which generate the same output data).

For more information on what StructuralIdentifiability.jl is all about, see the SciMLCon 2022 tutorial video.

MinimallyDisruptiveCurves.jl

MinimallyDisruptiveCurves.jl is a library for finding relationships between parameters of models, finding the curves on which the solution is constant.

Third-Party Libraries to Note

SIAN.jl: Structural Identifiability Analyzer

SIAN.jl is a structural identifiability analysis package which uses an entirely different algorithm from StructuralIdentifiability.jl. For information on the differences between the two approaches, see the Structural Identifiability Tools in Julia tutorial.

DynamicalSystems.jl: A Suite of Dynamical Systems Analysis

DynamicalSystems.jl is an entire ecosystem of dynamical systems analysis methods, for computing measures of chaos (dimension estimation, Lyapunov coefficients), generating delay embeddings, and much more. It uses the SciML tools for its internal equation solving and thus shares much of its API, adding a layer of new tools for extended analyses.

For more information, watch the tutorial Introduction to DynamicalSystems.jl.

BifurcationKit.jl

BifurcationKit.jl is a tool for performing bifurcation analysis. It uses and composes with many SciML equation solvers.

ReachabilityAnalysis.jl

ReachabilityAnalysis.jl is a library for performing reachability analysis of dynamical systems, determining for a given uncertainty interval the full set of possible outcomes from a dynamical system.

ControlSystems.jl

ControlSystems.jl is a library for building and analyzing control systems.

+Parameter Analysis Utilities · Overview of Julia's SciML
GitHub

Parameter Analysis Utilities

GlobalSensitivity.jl: Global Sensitivity Analysis

Derivatives calculate the local sensitivity of a model, i.e. the change in the simulation's outcome if one were to change the parameter with respect to some chosen part of the parameter space. But how does a simulation's output change “in general” with respect to a given parameter? That is what global sensitivity analysis (GSA) computes, and thus GlobalSensitivity.jl is the way to answer that question. GlobalSensitivity.jl includes a wide array of methods, including:

  • Morris's method
  • Sobol's method
  • Regression methods (PCC, SRC, Pearson)
  • eFAST
  • Delta Moment-Independent method
  • Derivative-based Global Sensitivity Measures (DGSM)
  • EASI
  • Fractional Factorial method
  • Random Balance Design FAST method

StructuralIdentifiability.jl: Identifiability Analysis Made Simple

Performing parameter estimation from a data set means attempting to recover parameters like reaction rates by fitting some model to the data. But how do you know whether you have enough data to even consider getting the “correct” parameters back? StructuralIdentifiability.jl allows for running a structural identifiability analysis on a given model to determine whether it's theoretically possible to recover the correct parameters. It can state whether a given type of output data can be used to globally recover the parameters (i.e. only a unique parameter set for the model produces a given output), whether the parameters are only locally identifiable (i.e. there are finitely many parameter sets which could generate the seen data), or whether it's unidentifiable (there are infinitely many parameters which generate the same output data).

For more information on what StructuralIdentifiability.jl is all about, see the SciMLCon 2022 tutorial video.

MinimallyDisruptiveCurves.jl

MinimallyDisruptiveCurves.jl is a library for finding relationships between parameters of models, finding the curves on which the solution is constant.

Third-Party Libraries to Note

SIAN.jl: Structural Identifiability Analyzer

SIAN.jl is a structural identifiability analysis package which uses an entirely different algorithm from StructuralIdentifiability.jl. For information on the differences between the two approaches, see the Structural Identifiability Tools in Julia tutorial.

DynamicalSystems.jl: A Suite of Dynamical Systems Analysis

DynamicalSystems.jl is an entire ecosystem of dynamical systems analysis methods, for computing measures of chaos (dimension estimation, Lyapunov coefficients), generating delay embeddings, and much more. It uses the SciML tools for its internal equation solving and thus shares much of its API, adding a layer of new tools for extended analyses.

For more information, watch the tutorial Introduction to DynamicalSystems.jl.

BifurcationKit.jl

BifurcationKit.jl is a tool for performing bifurcation analysis. It uses and composes with many SciML equation solvers.

ReachabilityAnalysis.jl

ReachabilityAnalysis.jl is a library for performing reachability analysis of dynamical systems, determining for a given uncertainty interval the full set of possible outcomes from a dynamical system.

ControlSystems.jl

ControlSystems.jl is a library for building and analyzing control systems.

diff --git a/dev/highlevels/partial_differential_equation_solvers/index.html b/dev/highlevels/partial_differential_equation_solvers/index.html index 9c7d28f9686..710d8110395 100644 --- a/dev/highlevels/partial_differential_equation_solvers/index.html +++ b/dev/highlevels/partial_differential_equation_solvers/index.html @@ -1,2 +1,2 @@ -Partial Differential Equations (PDE) · Overview of Julia's SciML
GitHub

Partial Differential Equations (PDE)

NeuralPDE.jl: Physics-Informed Neural Network (PINN) PDE Solvers

NeuralPDE.jl is a partial differential equation solver library which uses physics-informed neural networks (PINNs) to solve the equations. It uses the ModelingToolkit.jl symbolic PDESystem as its input and can handle a wide variety of equation types, including systems of partial differential equations, partial differential-algebraic equations, and integro-differential equations. Its benefit is its flexibility, and it can be used to easily generate surrogate solutions over entire parameter ranges. However, its downside is solver speed: PINN solvers tend to be a lot slower than other methods for solving PDEs.

MethodOflines.jl: Automated Finite Difference Method (FDM)

MethodOflines.jl is a partial differential equation solver library which automates the discretization of PDEs via the finite difference method. It uses the ModelingToolkit.jl symbolic PDESystem as its input, and generates AbstractSystems and SciMLProblems whose numerical solution gives the solution to the PDE.

FEniCS.jl: Wrappers for the Finite Element Method (FEM)

FEniCS.jl is a wrapper for the popular FEniCS finite element method library.

HighDimPDE.jl: High-dimensional PDE Solvers

HighDimPDE.jl is a partial differential equation solver library which implements algorithms that break down the curse of dimensionality to solve the equations. It implements deep-learning based and Picard-iteration based methods to approximately solve high-dimensional, nonlinear, non-local PDEs in up to 10,000 dimensions. Its cons are accuracy: high-dimensional solvers are stochastic, and might result in wrong solutions if the solver meta-parameters are not appropriate.

NeuralOperators.jl: (Fourier) Neural Operators and DeepONets for PDE Solving

NeuralOperators.jl is a library for operator learning based PDE solvers. This includes techniques like:

  • Fourier Neural Operators (FNO)
  • Deep Operator Networks (DeepONets)
  • Markov Neural Operators (MNO)

Currently, its connection to PDE solving must be specified manually, though an interface for ModelingToolkit PDESystems is in progress.

DiffEqOperators.jl: Operators for Finite Difference Method (FDM) Discretizations

DiffEqOperators.jl is a library for defining finite difference operators to easily perform manual FDM semi-discretizations of partial differential equations. This library is fairly incomplete and most cases should receive better performance using MethodOflines.jl.

Third-Party Libraries to Note

ApproxFun.jl: Automated Spectral Discretizations

ApproxFun.jl is a package for approximating functions in basis sets. One particular use case is with spectral basis sets, such as Chebyshev functions and Fourier decompositions, making it easy to represent spectral and pseudospectral discretizations of partial differential equations as ordinary differential equations for the SciML equation solvers.

Gridap.jl: Julia-Based Tools for Finite Element Discretizations

Gridap.jl is a package for grid-based approximation of partial differential equations, particularly notable for its use of conforming and nonconforming finite element (FEM) discretizations.

Trixi.jl: Adaptive High-Order Numerical Simulations of Hyperbolic Equations

Trixi.jl is a package for numerical simulation of hyperbolic conservation laws, i.e. a large set of hyperbolic partial differential equations, which interfaces and uses the SciML ordinary differential equation solvers.

VoronoiFVM.jl: Tools for the Voronoi Finite Volume Discretizations

VoronoiFVM.jl is a library for generating FVM discretizations of systems of PDEs. It interfaces with many of the SciML equation solver libraries to allow for ease of discretization and flexibility in the solver choice.

+Partial Differential Equations (PDE) · Overview of Julia's SciML
GitHub

Partial Differential Equations (PDE)

NeuralPDE.jl: Physics-Informed Neural Network (PINN) PDE Solvers

NeuralPDE.jl is a partial differential equation solver library which uses physics-informed neural networks (PINNs) to solve the equations. It uses the ModelingToolkit.jl symbolic PDESystem as its input and can handle a wide variety of equation types, including systems of partial differential equations, partial differential-algebraic equations, and integro-differential equations. Its benefit is its flexibility, and it can be used to easily generate surrogate solutions over entire parameter ranges. However, its downside is solver speed: PINN solvers tend to be a lot slower than other methods for solving PDEs.

MethodOflines.jl: Automated Finite Difference Method (FDM)

MethodOflines.jl is a partial differential equation solver library which automates the discretization of PDEs via the finite difference method. It uses the ModelingToolkit.jl symbolic PDESystem as its input, and generates AbstractSystems and SciMLProblems whose numerical solution gives the solution to the PDE.

FEniCS.jl: Wrappers for the Finite Element Method (FEM)

FEniCS.jl is a wrapper for the popular FEniCS finite element method library.

HighDimPDE.jl: High-dimensional PDE Solvers

HighDimPDE.jl is a partial differential equation solver library which implements algorithms that break down the curse of dimensionality to solve the equations. It implements deep-learning based and Picard-iteration based methods to approximately solve high-dimensional, nonlinear, non-local PDEs in up to 10,000 dimensions. Its cons are accuracy: high-dimensional solvers are stochastic, and might result in wrong solutions if the solver meta-parameters are not appropriate.

NeuralOperators.jl: (Fourier) Neural Operators and DeepONets for PDE Solving

NeuralOperators.jl is a library for operator learning based PDE solvers. This includes techniques like:

  • Fourier Neural Operators (FNO)
  • Deep Operator Networks (DeepONets)
  • Markov Neural Operators (MNO)

Currently, its connection to PDE solving must be specified manually, though an interface for ModelingToolkit PDESystems is in progress.

DiffEqOperators.jl: Operators for Finite Difference Method (FDM) Discretizations

DiffEqOperators.jl is a library for defining finite difference operators to easily perform manual FDM semi-discretizations of partial differential equations. This library is fairly incomplete and most cases should receive better performance using MethodOflines.jl.

Third-Party Libraries to Note

ApproxFun.jl: Automated Spectral Discretizations

ApproxFun.jl is a package for approximating functions in basis sets. One particular use case is with spectral basis sets, such as Chebyshev functions and Fourier decompositions, making it easy to represent spectral and pseudospectral discretizations of partial differential equations as ordinary differential equations for the SciML equation solvers.

Gridap.jl: Julia-Based Tools for Finite Element Discretizations

Gridap.jl is a package for grid-based approximation of partial differential equations, particularly notable for its use of conforming and nonconforming finite element (FEM) discretizations.

Trixi.jl: Adaptive High-Order Numerical Simulations of Hyperbolic Equations

Trixi.jl is a package for numerical simulation of hyperbolic conservation laws, i.e. a large set of hyperbolic partial differential equations, which interfaces and uses the SciML ordinary differential equation solvers.

VoronoiFVM.jl: Tools for the Voronoi Finite Volume Discretizations

VoronoiFVM.jl is a library for generating FVM discretizations of systems of PDEs. It interfaces with many of the SciML equation solver libraries to allow for ease of discretization and flexibility in the solver choice.

diff --git a/dev/highlevels/plots_visualization/index.html b/dev/highlevels/plots_visualization/index.html index 8ff9474ccad..8279a2a7d66 100644 --- a/dev/highlevels/plots_visualization/index.html +++ b/dev/highlevels/plots_visualization/index.html @@ -1,2 +1,2 @@ -SciML-Supported Plotting and Visualization Libraries · Overview of Julia's SciML
GitHub

SciML-Supported Plotting and Visualization Libraries

The following libraries are the plotting and visualization libraries which are supported and co-developed by the SciML developers. Other libraries may be used, though these are the libraries used in the tutorials and which have special hooks to ensure ergonomic usage with SciML tooling.

Plots.jl

Plots.jl is the current standard plotting system for the SciML ecosystem. SciML types attempt to include plot recipes for as many types as possible, allowing for automatic visualization with the Plots.jl system. All current tutorials and documentation default to using Plots.jl.

Makie.jl

Makie.jl is a high-performance interactive plotting system for the Julia programming language. It's planned to be the default plotting system used by the SciML organization in the near future.

+SciML-Supported Plotting and Visualization Libraries · Overview of Julia's SciML
GitHub

SciML-Supported Plotting and Visualization Libraries

The following libraries are the plotting and visualization libraries which are supported and co-developed by the SciML developers. Other libraries may be used, though these are the libraries used in the tutorials and which have special hooks to ensure ergonomic usage with SciML tooling.

Plots.jl

Plots.jl is the current standard plotting system for the SciML ecosystem. SciML types attempt to include plot recipes for as many types as possible, allowing for automatic visualization with the Plots.jl system. All current tutorials and documentation default to using Plots.jl.

Makie.jl

Makie.jl is a high-performance interactive plotting system for the Julia programming language. It's planned to be the default plotting system used by the SciML organization in the near future.

diff --git a/dev/highlevels/symbolic_learning/index.html b/dev/highlevels/symbolic_learning/index.html index 504ea81658f..40153a14a7e 100644 --- a/dev/highlevels/symbolic_learning/index.html +++ b/dev/highlevels/symbolic_learning/index.html @@ -1,2 +1,2 @@ -Symbolic Learning and Artificial Intelligence · Overview of Julia's SciML
GitHub

Symbolic Learning and Artificial Intelligence

Symbolic learning, the classical artificial intelligence, is a set of methods for learning symbolic equations from data and numerical functions. SciML offers an array of symbolic learning utilities which connect with the other machine learning and equation solver functionalities to make it easy to embed prior knowledge and discover missing physics. For more information, see Universal Differential Equations for Scientific Machine Learning.

DataDrivenDiffEq.jl: Data-Driven Modeling and Automated Discovery of Dynamical Systems

DataDrivenDiffEq.jl is a general interface for data-driven modeling, containing a large array of techniques such as:

  • Koopman operator methods (Dynamic-Mode Decomposition (DMD) and variations)
  • Sparse Identification of Dynamical Systems (SINDy and variations like iSINDy)
  • Sparse regression methods (STSLQ, SR3, etc.)
  • PDEFind
  • Wrappers for SymbolicRegression.jl
  • AI Feynman
  • OccamNet

SymbolicNumericIntegration.jl: Symbolic Integration via Numerical Methods

SymbolicNumericIntegration.jl is a package computing the solution to symbolic integration problem using numerical methods (numerical integration mixed with sparse regression).

Third-Party Libraries to Note

SymbolicRegression.jl

SymbolicRegression.jl is a symbolic regression library which uses genetic algorithms with parallelization to achieve fast and robust symbolic learning.

+Symbolic Learning and Artificial Intelligence · Overview of Julia's SciML
GitHub

Symbolic Learning and Artificial Intelligence

Symbolic learning, the classical artificial intelligence, is a set of methods for learning symbolic equations from data and numerical functions. SciML offers an array of symbolic learning utilities which connect with the other machine learning and equation solver functionalities to make it easy to embed prior knowledge and discover missing physics. For more information, see Universal Differential Equations for Scientific Machine Learning.

DataDrivenDiffEq.jl: Data-Driven Modeling and Automated Discovery of Dynamical Systems

DataDrivenDiffEq.jl is a general interface for data-driven modeling, containing a large array of techniques such as:

  • Koopman operator methods (Dynamic-Mode Decomposition (DMD) and variations)
  • Sparse Identification of Dynamical Systems (SINDy and variations like iSINDy)
  • Sparse regression methods (STSLQ, SR3, etc.)
  • PDEFind
  • Wrappers for SymbolicRegression.jl
  • AI Feynman
  • OccamNet

SymbolicNumericIntegration.jl: Symbolic Integration via Numerical Methods

SymbolicNumericIntegration.jl is a package computing the solution to symbolic integration problem using numerical methods (numerical integration mixed with sparse regression).

Third-Party Libraries to Note

SymbolicRegression.jl

SymbolicRegression.jl is a symbolic regression library which uses genetic algorithms with parallelization to achieve fast and robust symbolic learning.

diff --git a/dev/highlevels/symbolic_tools/index.html b/dev/highlevels/symbolic_tools/index.html index 06634a18eef..43969549296 100644 --- a/dev/highlevels/symbolic_tools/index.html +++ b/dev/highlevels/symbolic_tools/index.html @@ -1,2 +1,2 @@ -Symbolic Model Tooling and JuliaSymbolics · Overview of Julia's SciML
GitHub

Symbolic Model Tooling and JuliaSymbolics

JuliaSymbolics is a sister organization of SciML. It spawned out of the symbolic modeling tools being developed within SciML (ModelingToolkit.jl) to become its own organization dedicated to building a fully-featured Julia-based Computer Algebra System (CAS). As such, the two organizations are closely aligned in terms of its developer community, and many of the SciML libraries use Symbolics.jl extensively.

ModelOrderReduction.jl: Automated Model Reduction for Fast Approximations of Solutions

ModelOrderReduction.jl is a package for automating the reduction of models. These methods function a submodel with a projection, where solving the smaller model provides approximation information about the full model. MOR.jl uses ModelingToolkit.jl as a system description and automatically transforms equations to the subform, defining the observables to automatically lazily reconstruct the full model on-demand in a fast and stable form.

Symbolics.jl: The Computer Algebra System (CAS) of the Julia Programming Language

Symbolics.jl is the CAS of the Julia programming language. If something needs to be done symbolically, most likely Symbolics.jl is the answer.

MetaTheory.jl: E-Graphs to Automate Symbolic Transformations

Metatheory.jl is a library for defining e-graph rewriters for use on the common symbolic interface. This can be used to do all sorts of analysis and code transformations, such as improving code performance, numerical stability, and more. See Automated Code Optimization with E-Graphs for more details.

SymbolicUtils.jl: Define Your Own Computer Algebra System

SymbolicUtils.jl is the underlying utility library and rule-based rewriting language on which Symbolics.jl is developed. Symbolics.jl is standardized type and rule definitions built using SymbolicUtils.jl. However, if non-standard types are required, such as symbolic computing over Fock algebras, then SymbolicUtils.jl is the library from which the new symbolic types can be implemented.

+Symbolic Model Tooling and JuliaSymbolics · Overview of Julia's SciML
GitHub

Symbolic Model Tooling and JuliaSymbolics

JuliaSymbolics is a sister organization of SciML. It spawned out of the symbolic modeling tools being developed within SciML (ModelingToolkit.jl) to become its own organization dedicated to building a fully-featured Julia-based Computer Algebra System (CAS). As such, the two organizations are closely aligned in terms of its developer community, and many of the SciML libraries use Symbolics.jl extensively.

ModelOrderReduction.jl: Automated Model Reduction for Fast Approximations of Solutions

ModelOrderReduction.jl is a package for automating the reduction of models. These methods function a submodel with a projection, where solving the smaller model provides approximation information about the full model. MOR.jl uses ModelingToolkit.jl as a system description and automatically transforms equations to the subform, defining the observables to automatically lazily reconstruct the full model on-demand in a fast and stable form.

Symbolics.jl: The Computer Algebra System (CAS) of the Julia Programming Language

Symbolics.jl is the CAS of the Julia programming language. If something needs to be done symbolically, most likely Symbolics.jl is the answer.

MetaTheory.jl: E-Graphs to Automate Symbolic Transformations

Metatheory.jl is a library for defining e-graph rewriters for use on the common symbolic interface. This can be used to do all sorts of analysis and code transformations, such as improving code performance, numerical stability, and more. See Automated Code Optimization with E-Graphs for more details.

SymbolicUtils.jl: Define Your Own Computer Algebra System

SymbolicUtils.jl is the underlying utility library and rule-based rewriting language on which Symbolics.jl is developed. Symbolics.jl is standardized type and rule definitions built using SymbolicUtils.jl. However, if non-standard types are required, such as symbolic computing over Fock algebras, then SymbolicUtils.jl is the library from which the new symbolic types can be implemented.

diff --git a/dev/highlevels/uncertainty_quantification/index.html b/dev/highlevels/uncertainty_quantification/index.html index 4789faf22d2..11ad93bc88b 100644 --- a/dev/highlevels/uncertainty_quantification/index.html +++ b/dev/highlevels/uncertainty_quantification/index.html @@ -1,2 +1,2 @@ -Uncertainty Quantification · Overview of Julia's SciML
GitHub

Uncertainty Quantification

There's always uncertainty in our models. Whether it's in the form of the model's equations or in the model's parameters, the uncertainty in our simulation's output often needs to be quantified. The following tools automate this process.

For Measurements.jl vs MonteCarloMeasurements.jl vs Intervals.jl, and the relation to other methods, see the Uncertainty Programming chapter of the SciML Book.

PolyChaos.jl: Intrusive Polynomial Chaos Expansions Made Unintrusive

PolyChaos.jl is a library for calculating intrusive polynomial chaos expansions (PCE) on arbitrary Julia functions. This allows for inputting representations of probability distributions into functions to compute the output distribution in an expansion representation. While normally this would require deriving the PCE-expanded equations by hand, PolyChaos.jl does this at the compiler level using Julia's multiple dispatch, giving a high-performance implementation to a normally complex and tedious mathematical transformation.

SciMLExpectations.jl: Fast Calculations of Expectations of Equation Solutions

SciMLExpectations.jl is a library for accelerating the calculation of expectations of equation solutions with respect to input probability distributions, allowing for applications like robust optimization with respect to uncertainty. It uses Koopman operator techniques to calculate these expectations without requiring the propagation of uncertainties through a solver, effectively performing the adjoint of uncertainty quantification and being much more efficient in the process.

Third-Party Libraries to Note

Measurements.jl: Automated Linear Error Propagation

Measurements.jl is a library for automating linear error propagation. Uncertain numbers are defined as x = 3.8 ± 0.4 and are pushed through calculations using a normal distribution approximation in order to compute an approximate uncertain output. Measurements.jl uses a dictionary-based approach to keep track of correlations to improve the accuracy over naive implementations, though note that linear error propagation theory still has some major issues handling some types of equations, as described in detail in the MonteCarloMeasurements.jl documentation.

MonteCarloMeasurements.jl: Automated Monte Carlo Error Propagation

MonteCarloMeasurements.jl is a library for automating the uncertainty quantification of equation solution using Monte Carlo methods. It defines number types which sample from an input distribution to receive a representative set of parameters that propagate through the solver to calculate a representative set of possible solutions. Note that Monte Carlo techniques can be expensive but are exact, in the sense that as the number of sample points increases to infinity it will compute a correct approximation of the output uncertainty.

ProbNumDiffEq.jl: Probabilistic Numerics Based Differential Equation Solvers

ProbNumDiffEq.jl is a set of probabilistic numerical ODE solvers which compute the solution of a differential equation along with a posterior distribution to estimate its numerical approximation error. Thus these specialized integrators compute an uncertainty output similar to the ProbInts technique of DiffEqUncertainty, but use specialized integration techniques in order to do it much faster for specific kinds of equations.

TaylorIntegration.jl: Taylor Series Integration for Rigorous Numerical Bounds

TaylorIntegration.jl is a library for Taylor series integrators, which has special functionality for computing the interval bound of possible solutions with respect to numerical approximation error.

IntervalArithmetic.jl: Rigorous Numerical Intervals

IntervalArithmetic.jl is a library for performing interval arithmetic calculations on arbitrary Julia code. Interval arithmetic computes rigorous computations with respect to finite-precision floating-point arithmetic, i.e. its intervals are guaranteed to include the true solution. However, interval arithmetic intervals can grow at exponential rates in many problems, thus being unsuitable for analyses in many equation solver contexts.

+Uncertainty Quantification · Overview of Julia's SciML
GitHub

Uncertainty Quantification

There's always uncertainty in our models. Whether it's in the form of the model's equations or in the model's parameters, the uncertainty in our simulation's output often needs to be quantified. The following tools automate this process.

For Measurements.jl vs MonteCarloMeasurements.jl vs Intervals.jl, and the relation to other methods, see the Uncertainty Programming chapter of the SciML Book.

PolyChaos.jl: Intrusive Polynomial Chaos Expansions Made Unintrusive

PolyChaos.jl is a library for calculating intrusive polynomial chaos expansions (PCE) on arbitrary Julia functions. This allows for inputting representations of probability distributions into functions to compute the output distribution in an expansion representation. While normally this would require deriving the PCE-expanded equations by hand, PolyChaos.jl does this at the compiler level using Julia's multiple dispatch, giving a high-performance implementation to a normally complex and tedious mathematical transformation.

SciMLExpectations.jl: Fast Calculations of Expectations of Equation Solutions

SciMLExpectations.jl is a library for accelerating the calculation of expectations of equation solutions with respect to input probability distributions, allowing for applications like robust optimization with respect to uncertainty. It uses Koopman operator techniques to calculate these expectations without requiring the propagation of uncertainties through a solver, effectively performing the adjoint of uncertainty quantification and being much more efficient in the process.

Third-Party Libraries to Note

Measurements.jl: Automated Linear Error Propagation

Measurements.jl is a library for automating linear error propagation. Uncertain numbers are defined as x = 3.8 ± 0.4 and are pushed through calculations using a normal distribution approximation in order to compute an approximate uncertain output. Measurements.jl uses a dictionary-based approach to keep track of correlations to improve the accuracy over naive implementations, though note that linear error propagation theory still has some major issues handling some types of equations, as described in detail in the MonteCarloMeasurements.jl documentation.

MonteCarloMeasurements.jl: Automated Monte Carlo Error Propagation

MonteCarloMeasurements.jl is a library for automating the uncertainty quantification of equation solution using Monte Carlo methods. It defines number types which sample from an input distribution to receive a representative set of parameters that propagate through the solver to calculate a representative set of possible solutions. Note that Monte Carlo techniques can be expensive but are exact, in the sense that as the number of sample points increases to infinity it will compute a correct approximation of the output uncertainty.

ProbNumDiffEq.jl: Probabilistic Numerics Based Differential Equation Solvers

ProbNumDiffEq.jl is a set of probabilistic numerical ODE solvers which compute the solution of a differential equation along with a posterior distribution to estimate its numerical approximation error. Thus these specialized integrators compute an uncertainty output similar to the ProbInts technique of DiffEqUncertainty, but use specialized integration techniques in order to do it much faster for specific kinds of equations.

TaylorIntegration.jl: Taylor Series Integration for Rigorous Numerical Bounds

TaylorIntegration.jl is a library for Taylor series integrators, which has special functionality for computing the interval bound of possible solutions with respect to numerical approximation error.

IntervalArithmetic.jl: Rigorous Numerical Intervals

IntervalArithmetic.jl is a library for performing interval arithmetic calculations on arbitrary Julia code. Interval arithmetic computes rigorous computations with respect to finite-precision floating-point arithmetic, i.e. its intervals are guaranteed to include the true solution. However, interval arithmetic intervals can grow at exponential rates in many problems, thus being unsuitable for analyses in many equation solver contexts.

diff --git a/dev/index.html b/dev/index.html index a91cc7098ad..c9ae4a1d79c 100644 --- a/dev/index.html +++ b/dev/index.html @@ -1,5 +1,5 @@ -SciML: Open Source Software for Scientific Machine Learning with Julia · Overview of Julia's SciML
GitHub

SciML: Differentiable Modeling and Simulation Combined with Machine Learning

The SciML organization is a collection of tools for solving equations and modeling systems developed in the Julia programming language with bindings to other languages such as R and Python. The organization provides well-maintained tools which compose together as a coherent ecosystem. It has a coherent development principle, unified APIs over large collections of equation solvers, pervasive differentiability and sensitivity analysis, and features many of the highest performance and parallel implementations one can find.

Scientific Machine Learning (SciML) = Scientific Computing + Machine Learning

Where to Start?

And for diving into the details, use the bar on the top to navigate to the submodule of interest!

Reproducibility

The documentation of the [SciML Showcase](@ref showcase) was built using these direct dependencies,
Status `/var/lib/buildkite-agent/builds/gpuci-13/julialang/scimldocs/docs/Project.toml`
+SciML: Open Source Software for Scientific Machine Learning with Julia · Overview of Julia's SciML
GitHub
SciML: Differentiable Modeling and Simulation Combined with Machine Learning
The SciML organization is a collection of tools for solving equations and modeling systems developed in the Julia programming language with bindings to other languages such as R and Python. The organization provides well-maintained tools which compose together as a coherent ecosystem. It has a coherent development principle, unified APIs over large collections of equation solvers, pervasive differentiability and sensitivity analysis, and features many of the highest performance and parallel implementations one can find.
Scientific Machine Learning (SciML) = Scientific Computing + Machine Learning
Where to Start?
And for diving into the details, use the bar on the top to navigate to the submodule of interest!
Reproducibility
The documentation of the [SciML Showcase](@ref showcase) was built using these direct dependencies,
Status `/var/lib/buildkite-agent/builds/gpuci-9/julialang/scimldocs/docs/Project.toml`
   [0bf59076] AdvancedHMC v0.5.5
   [6e4b80f9] BenchmarkTools v1.3.2
   [336ed68f] CSV v0.10.11
@@ -66,7 +66,7 @@
   JULIA_DEPOT_PATH = /root/.cache/julia-buildkite-plugin/depots/0183cc98-c3b4-4959-aaaa-6c0d5f351407
   LD_LIBRARY_PATH = /usr/local/nvidia/lib:/usr/local/nvidia/lib64
   JULIA_PKG_SERVER =
-  JULIA_IMAGE_THREADS = 1
A more complete overview of all dependencies and their versions is also provided.
Status `/var/lib/buildkite-agent/builds/gpuci-13/julialang/scimldocs/docs/Manifest.toml`
+  JULIA_IMAGE_THREADS = 1
A more complete overview of all dependencies and their versions is also provided.
Status `/var/lib/buildkite-agent/builds/gpuci-9/julialang/scimldocs/docs/Manifest.toml`
   [47edcb42] ADTypes v0.2.4
   [a4c015fc] ANSIColoredPrinters v0.0.1
 ⌅ [c3fe647b] AbstractAlgebra v0.32.5
@@ -565,4 +565,4 @@
   [8e850b90] libblastrampoline_jll v5.8.0+0
   [8e850ede] nghttp2_jll v1.48.0+0
   [3f19e933] p7zip_jll v17.4.0+0
-Info Packages marked with ⌃ and ⌅ have new versions available, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated -m`
You can also download the manifest file and the project file.

+Info Packages marked with ⌃ and ⌅ have new versions available, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated -m`

You can also download the manifest file and the project file.

diff --git a/dev/overview/index.html b/dev/overview/index.html index 5545236f264..eddf3b5e3ac 100644 --- a/dev/overview/index.html +++ b/dev/overview/index.html @@ -1,2 +1,2 @@ -Detailed Overview of the SciML Software Ecosystem · Overview of Julia's SciML
GitHub

Detailed Overview of the SciML Software Ecosystem

SciML: Combining High-Performance Scientific Computing and Machine Learning

SciML is not standard machine learning, SciML is the combination of scientific computing techniques with machine learning. Thus the SciML organization is not an organization for machine learning libraries (see FluxML for machine learning in Julia), rather SciML is an organization dedicated to the development of scientific computing tools which work seamlessly in conjunction with next-generation machine learning workflows. This includes:

  • High-performance and accurate tools for standard scientific computing modeling and simulation
  • Compatibility with differentiable programming and automatic differentiation
  • Tools for building complex multiscale models
  • Methods for handling inverse problems, model calibration, controls, and Bayesian analysis
  • Symbolic modeling tools for generating efficient code for numerical equation solvers
  • Methods for automatic discovery of (bio)physical equations

and much more. For an overview of the broad goals of the SciML organization, watch:

Overview of Computational Science in Julia with SciML

Below is a simplification of the user-facing packages for use in scientific computing and SciML workflows.

Workflow ElementSciML-Supported Julia packages
Plotting and VisualizationPlots*, Makie*
Sparse matrixSparseArrays*
Interpolation/approximationDataInterpolations*, ApproxFun*
Linear system / least squaresLinearSolve
Nonlinear system / rootfindingNonlinearSolve
Polynomial rootsPolynomials*
IntegrationIntegrals
Nonlinear OptimizationOptimization
Other Optimization (linear, quadratic, convex, etc.)JuMP*
Initial-value problemDifferentialEquations
Boundary-value problemDifferentialEquations
Continuous-Time Markov Chains (Poisson Jumps), Jump DiffusionsJumpProcesses
Finite differencesFiniteDifferences*, FiniteDiff*
Automatic DifferentiationForwardDiff*, Enzyme*, DiffEqSensitivity
Bayesian ModelingTuring*
Deep LearningFlux*
Acausal Modeling / DAEsModelingToolkit
Chemical Reaction NetworksCatalyst
Symbolic ComputingSymbolics
Fast Fourier TransformFFTW*
Partial Differential Equation DiscretizationsAssociated Julia packages
–-–-
Finite DifferencesMethodOfLines
Discontinuous GalerkinTrixi*
Finite ElementGridap*
Physics-Informed Neural NetworksNeuralPDE
Neural OperatorsNeuralOperators
High Dimensional Deep LearningHighDimPDE

* Denotes a non-SciML package that is heavily tested against as part of SciML workflows and has frequent collaboration with the SciML developers.

SciML Mind Map

Domains of SciML

The SciML common interface covers the following domains:

  • Linear systems (LinearProblem)

    • Direct methods for dense and sparse
    • Iterative solvers with preconditioning

  • Nonlinear Systems (NonlinearProblem)

    • Systems of nonlinear equations
    • Scalar bracketing systems

  • Integrals (quadrature) (IntegralProblem)

  • Differential Equations

    • Discrete equations (function maps, discrete stochastic (Gillespie/Markov) simulations) (DiscreteProblem and JumpProblem)
    • Ordinary differential equations (ODEs) (ODEProblem)
    • Split and Partitioned ODEs (Symplectic integrators, IMEX Methods) (SplitODEProblem)
    • Stochastic ordinary differential equations (SODEs or SDEs) (SDEProblem)
    • Stochastic differential-algebraic equations (SDAEs) (SDEProblem with mass matrices)
    • Random differential equations (RODEs or RDEs) (RODEProblem)
    • Differential algebraic equations (DAEs) (DAEProblem and ODEProblem with mass matrices)
    • Delay differential equations (DDEs) (DDEProblem)
    • Neutral, retarded, and algebraic delay differential equations (NDDEs, RDDEs, and DDAEs)
    • Stochastic delay differential equations (SDDEs) (SDDEProblem)
    • Experimental support for stochastic neutral, retarded, and algebraic delay differential equations (SNDDEs, SRDDEs, and SDDAEs)
    • Mixed discrete and continuous equations (Hybrid Equations, Jump Diffusions) (DEProblems with callbacks and JumpProblem)

  • Optimization (OptimizationProblem)

    • Nonlinear (constrained) optimization

  • (Stochastic/Delay/Differential-Algebraic) Partial Differential Equations (PDESystem)

    • Finite difference and finite volume methods
    • Interfaces to finite element methods
    • Physics-Informed Neural Networks (PINNs)
    • Integro-Differential Equations
    • Fractional Differential Equations

  • Specialized Forms

    • Partial Integro-Differential Equations (PIPDEProblem)

  • Data-driven modeling

    • Discrete-time data-driven dynamical systems (DiscreteDataDrivenProblem)
    • Continuous-time data-driven dynamical systems (ContinuousDataDrivenProblem)
    • Symbolic regression (DirectDataDrivenProblem)

  • Uncertainty quantification and expected values (ExpectationProblem)

The SciML common interface also includes ModelingToolkit.jl for defining such systems symbolically, allowing for optimizations like automated generation of parallel code, symbolic simplification, and generation of sparsity patterns.

Inverse Problems, Parameter Estimation, and Structural Identification

Parameter estimation and inverse problems are solved directly on their constituent problem types using tools like SciMLSensitivity.jl. Thus for example, there is no ODEInverseProblem, and instead ODEProblem is used to find the parameters p that solve the inverse problem. Check out the SciMLSensitivity documentation for a discussion on connections to automatic differentiation, optimization, and adjoints.

Common Interface High-Level Overview

The SciML interface is common as the usage of arguments is standardized across all of the problem domains. Underlying high-level ideas include:

  • All domains use the same interface of defining a AbstractSciMLProblem which is then solved via solve(prob,alg;kwargs), where alg is a AbstractSciMLAlgorithm. The keyword argument namings are standardized across the organization.
  • AbstractSciMLProblems are generally defined by a AbstractSciMLFunction which can define extra details about a model function, such as its analytical Jacobian, its sparsity patterns and so on.
  • There is an organization-wide method for defining linear and nonlinear solvers used within other solvers, giving maximum control of performance to the user.
  • Types used within the packages are defined by the input types. For example, packages attempt to internally use the type of the initial condition as the type for the state within differential equation solvers.
  • solve calls should be thread-safe and parallel-safe.
  • init(prob,alg;kwargs) returns an iterator which allows for directly iterating over the solution process
  • High performance is key. Any performance that is not at the top level is considered a bug and should be reported as such.
  • All functions have an in-place and out-of-place form, where the in-place form is made to utilize mutation for high performance on large-scale problems and the out-of-place form is for compatibility with tooling like static arrays and some reverse-mode automatic differentiation systems.

Flowchart Example for PDE-Constrained Optimal Control

The following example showcases how the pieces of the common interface connect to solve a problem that mixes inference, symbolics, and numerics.

External Binding Libraries

  • diffeqr

    • Solving differential equations in R using DifferentialEquations.jl with ModelingToolkit for JIT compilation and GPU-acceleration

  • diffeqpy

    • Solving differential equations in Python using DifferentialEquations.jl

Note About Third-Party Libraries

The SciML documentation references and recommends many third-party libraries for improving ones modeling, simulation, and analysis workflow in Julia. Take these as a positive affirmation of the quality of these libraries, as these libraries are commonly tested by SciML developers who are in contact with the development teams of these groups. It also documents the libraries which are commonly chosen by SciML as dependencies. Do not take omissions as negative affirmations against a given library, i.e. a library left off of the list by SciML is not a negative endorsement. Rather, it means that compatibility with SciML is untested, SciML developers may have a personal preference for another choice, or SciML developers may be simply unaware of the library's existence. If one would like to add a third-party library to the SciML documentation, open a pull request with the requested text.

Note that the libraries in this documentation are only those that are meant to be used in the SciML extended universe of modeling, simulation, and analysis and thus there are many high-quality libraries in other domains (machine learning, data science, etc.) which are purposefully not included. For an overview of the Julia package ecosystem, see the JuliaHub Search Engine.

+Detailed Overview of the SciML Software Ecosystem · Overview of Julia's SciML
GitHub

Detailed Overview of the SciML Software Ecosystem

SciML: Combining High-Performance Scientific Computing and Machine Learning

SciML is not standard machine learning, SciML is the combination of scientific computing techniques with machine learning. Thus the SciML organization is not an organization for machine learning libraries (see FluxML for machine learning in Julia), rather SciML is an organization dedicated to the development of scientific computing tools which work seamlessly in conjunction with next-generation machine learning workflows. This includes:

  • High-performance and accurate tools for standard scientific computing modeling and simulation
  • Compatibility with differentiable programming and automatic differentiation
  • Tools for building complex multiscale models
  • Methods for handling inverse problems, model calibration, controls, and Bayesian analysis
  • Symbolic modeling tools for generating efficient code for numerical equation solvers
  • Methods for automatic discovery of (bio)physical equations

and much more. For an overview of the broad goals of the SciML organization, watch:

Overview of Computational Science in Julia with SciML

Below is a simplification of the user-facing packages for use in scientific computing and SciML workflows.

Workflow ElementSciML-Supported Julia packages
Plotting and VisualizationPlots*, Makie*
Sparse matrixSparseArrays*
Interpolation/approximationDataInterpolations*, ApproxFun*
Linear system / least squaresLinearSolve
Nonlinear system / rootfindingNonlinearSolve
Polynomial rootsPolynomials*
IntegrationIntegrals
Nonlinear OptimizationOptimization
Other Optimization (linear, quadratic, convex, etc.)JuMP*
Initial-value problemDifferentialEquations
Boundary-value problemDifferentialEquations
Continuous-Time Markov Chains (Poisson Jumps), Jump DiffusionsJumpProcesses
Finite differencesFiniteDifferences*, FiniteDiff*
Automatic DifferentiationForwardDiff*, Enzyme*, DiffEqSensitivity
Bayesian ModelingTuring*
Deep LearningFlux*
Acausal Modeling / DAEsModelingToolkit
Chemical Reaction NetworksCatalyst
Symbolic ComputingSymbolics
Fast Fourier TransformFFTW*
Partial Differential Equation DiscretizationsAssociated Julia packages
–-–-
Finite DifferencesMethodOfLines
Discontinuous GalerkinTrixi*
Finite ElementGridap*
Physics-Informed Neural NetworksNeuralPDE
Neural OperatorsNeuralOperators
High Dimensional Deep LearningHighDimPDE

* Denotes a non-SciML package that is heavily tested against as part of SciML workflows and has frequent collaboration with the SciML developers.

SciML Mind Map

Domains of SciML

The SciML common interface covers the following domains:

  • Linear systems (LinearProblem)

    • Direct methods for dense and sparse
    • Iterative solvers with preconditioning

  • Nonlinear Systems (NonlinearProblem)

    • Systems of nonlinear equations
    • Scalar bracketing systems

  • Integrals (quadrature) (IntegralProblem)

  • Differential Equations

    • Discrete equations (function maps, discrete stochastic (Gillespie/Markov) simulations) (DiscreteProblem and JumpProblem)
    • Ordinary differential equations (ODEs) (ODEProblem)
    • Split and Partitioned ODEs (Symplectic integrators, IMEX Methods) (SplitODEProblem)
    • Stochastic ordinary differential equations (SODEs or SDEs) (SDEProblem)
    • Stochastic differential-algebraic equations (SDAEs) (SDEProblem with mass matrices)
    • Random differential equations (RODEs or RDEs) (RODEProblem)
    • Differential algebraic equations (DAEs) (DAEProblem and ODEProblem with mass matrices)
    • Delay differential equations (DDEs) (DDEProblem)
    • Neutral, retarded, and algebraic delay differential equations (NDDEs, RDDEs, and DDAEs)
    • Stochastic delay differential equations (SDDEs) (SDDEProblem)
    • Experimental support for stochastic neutral, retarded, and algebraic delay differential equations (SNDDEs, SRDDEs, and SDDAEs)
    • Mixed discrete and continuous equations (Hybrid Equations, Jump Diffusions) (DEProblems with callbacks and JumpProblem)

  • Optimization (OptimizationProblem)

    • Nonlinear (constrained) optimization

  • (Stochastic/Delay/Differential-Algebraic) Partial Differential Equations (PDESystem)

    • Finite difference and finite volume methods
    • Interfaces to finite element methods
    • Physics-Informed Neural Networks (PINNs)
    • Integro-Differential Equations
    • Fractional Differential Equations

  • Specialized Forms

    • Partial Integro-Differential Equations (PIPDEProblem)

  • Data-driven modeling

    • Discrete-time data-driven dynamical systems (DiscreteDataDrivenProblem)
    • Continuous-time data-driven dynamical systems (ContinuousDataDrivenProblem)
    • Symbolic regression (DirectDataDrivenProblem)

  • Uncertainty quantification and expected values (ExpectationProblem)

The SciML common interface also includes ModelingToolkit.jl for defining such systems symbolically, allowing for optimizations like automated generation of parallel code, symbolic simplification, and generation of sparsity patterns.

Inverse Problems, Parameter Estimation, and Structural Identification

Parameter estimation and inverse problems are solved directly on their constituent problem types using tools like SciMLSensitivity.jl. Thus for example, there is no ODEInverseProblem, and instead ODEProblem is used to find the parameters p that solve the inverse problem. Check out the SciMLSensitivity documentation for a discussion on connections to automatic differentiation, optimization, and adjoints.

Common Interface High-Level Overview

The SciML interface is common as the usage of arguments is standardized across all of the problem domains. Underlying high-level ideas include:

  • All domains use the same interface of defining a AbstractSciMLProblem which is then solved via solve(prob,alg;kwargs), where alg is a AbstractSciMLAlgorithm. The keyword argument namings are standardized across the organization.
  • AbstractSciMLProblems are generally defined by a AbstractSciMLFunction which can define extra details about a model function, such as its analytical Jacobian, its sparsity patterns and so on.
  • There is an organization-wide method for defining linear and nonlinear solvers used within other solvers, giving maximum control of performance to the user.
  • Types used within the packages are defined by the input types. For example, packages attempt to internally use the type of the initial condition as the type for the state within differential equation solvers.
  • solve calls should be thread-safe and parallel-safe.
  • init(prob,alg;kwargs) returns an iterator which allows for directly iterating over the solution process
  • High performance is key. Any performance that is not at the top level is considered a bug and should be reported as such.
  • All functions have an in-place and out-of-place form, where the in-place form is made to utilize mutation for high performance on large-scale problems and the out-of-place form is for compatibility with tooling like static arrays and some reverse-mode automatic differentiation systems.

Flowchart Example for PDE-Constrained Optimal Control

The following example showcases how the pieces of the common interface connect to solve a problem that mixes inference, symbolics, and numerics.

External Binding Libraries

  • diffeqr

    • Solving differential equations in R using DifferentialEquations.jl with ModelingToolkit for JIT compilation and GPU-acceleration

  • diffeqpy

    • Solving differential equations in Python using DifferentialEquations.jl

Note About Third-Party Libraries

The SciML documentation references and recommends many third-party libraries for improving ones modeling, simulation, and analysis workflow in Julia. Take these as a positive affirmation of the quality of these libraries, as these libraries are commonly tested by SciML developers who are in contact with the development teams of these groups. It also documents the libraries which are commonly chosen by SciML as dependencies. Do not take omissions as negative affirmations against a given library, i.e. a library left off of the list by SciML is not a negative endorsement. Rather, it means that compatibility with SciML is untested, SciML developers may have a personal preference for another choice, or SciML developers may be simply unaware of the library's existence. If one would like to add a third-party library to the SciML documentation, open a pull request with the requested text.

Note that the libraries in this documentation are only those that are meant to be used in the SciML extended universe of modeling, simulation, and analysis and thus there are many high-quality libraries in other domains (machine learning, data science, etc.) which are purposefully not included. For an overview of the Julia package ecosystem, see the JuliaHub Search Engine.

diff --git a/dev/showcase/bayesian_neural_ode/3171e8ff.svg b/dev/showcase/bayesian_neural_ode/3171e8ff.svg deleted file mode 100644 index 3e4d1599ba8..00000000000 --- a/dev/showcase/bayesian_neural_ode/3171e8ff.svg +++ /dev/null @@ -1,316 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dev/showcase/bayesian_neural_ode/45b3565a.svg b/dev/showcase/bayesian_neural_ode/45b3565a.svg deleted file mode 100644 index cabc1e72e49..00000000000 --- a/dev/showcase/bayesian_neural_ode/45b3565a.svg +++ /dev/null @@ -1,389 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dev/showcase/bayesian_neural_ode/5447e78c.svg b/dev/showcase/bayesian_neural_ode/5447e78c.svg new file mode 100644 index 00000000000..3491667f6bc --- /dev/null +++ b/dev/showcase/bayesian_neural_ode/5447e78c.svg @@ -0,0 +1,328 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/bayesian_neural_ode/5fc0d793.svg b/dev/showcase/bayesian_neural_ode/5fc0d793.svg deleted file mode 100644 index 3363355faf8..00000000000 --- a/dev/showcase/bayesian_neural_ode/5fc0d793.svg +++ /dev/null @@ -1,731 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dev/showcase/bayesian_neural_ode/d5f26c8c.svg b/dev/showcase/bayesian_neural_ode/d5f26c8c.svg new file mode 100644 index 00000000000..b8f23d40624 --- /dev/null +++ b/dev/showcase/bayesian_neural_ode/d5f26c8c.svg @@ -0,0 +1,389 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/bayesian_neural_ode/da41ed8f.svg b/dev/showcase/bayesian_neural_ode/da41ed8f.svg deleted file mode 100644 index 75da78631a3..00000000000 --- a/dev/showcase/bayesian_neural_ode/da41ed8f.svg +++ /dev/null @@ -1,169 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dev/showcase/bayesian_neural_ode/de435bca.svg b/dev/showcase/bayesian_neural_ode/de435bca.svg new file mode 100644 index 00000000000..4e6c0fa6575 --- /dev/null +++ b/dev/showcase/bayesian_neural_ode/de435bca.svg @@ -0,0 +1,169 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/bayesian_neural_ode/e38ccbe5.svg b/dev/showcase/bayesian_neural_ode/e38ccbe5.svg new file mode 100644 index 00000000000..f1b681f0d4c --- /dev/null +++ b/dev/showcase/bayesian_neural_ode/e38ccbe5.svg @@ -0,0 +1,731 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/bayesian_neural_ode/index.html b/dev/showcase/bayesian_neural_ode/index.html index 53a4d7ae440..f57a4725a7b 100644 --- a/dev/showcase/bayesian_neural_ode/index.html +++ b/dev/showcase/bayesian_neural_ode/index.html @@ -20,24 +20,24 @@ prob_neuralode = NeuralODE(dudt2, tspan, Tsit5(), saveat = tsteps) rng = Random.default_rng() p = Float64.(prob_neuralode.p)
252-element Vector{Float64}:
- -0.1854008138179779
- -0.28647381067276
-  0.145919531583786
-  0.08644620329141617
- -0.053269289433956146
-  0.20180708169937134
- -0.2493487000465393
-  0.2379695624113083
-  0.08451325446367264
- -0.14735132455825806
+  0.09884993731975555
+ -0.026836948469281197
+ -0.10988697409629822
+ -0.15215255320072174
+ -0.19330763816833496
+ -0.05161436274647713
+  0.17740000784397125
+  0.02958742156624794
+  0.16917160153388977
+  0.24170111119747162
   ⋮
- -0.2585447132587433
- -0.18830345571041107
- -0.20958977937698364
-  0.20926401019096375
-  0.06813614815473557
- -0.0686485543847084
-  0.24162498116493225
+ -0.1010826975107193
+ -0.028700130060315132
+ -0.3326646685600281
+ -0.05639230087399483
+  0.20527678728103638
+ -0.11461251229047775
+ -0.15055108070373535
   0.0
   0.0

Note that the f64 is required to put the Flux neural network into Float64 precision.

Step 3: Define the loss function for the Neural ODE.

function predict_neuralode(p)
     Array(prob_neuralode(u0, p))
@@ -57,11 +57,11 @@
 h = Hamiltonian(metric, l, dldθ)
Hamiltonian(metric=DiagEuclideanMetric([1.0, 1.0, 1.0, 1.0, 1.0, 1 ...]), kinetic=AdvancedHMC.GaussianKinetic())

We use the NUTS sampler with an acceptance ratio of δ= 0.45 in this example. In addition, we use Nesterov Dual Averaging for the Step Size adaptation.

We sample using 500 warmup samples and 500 posterior samples.

integrator = Leapfrog(find_good_stepsize(h, p))
 kernel = HMCKernel(Trajectory{MultinomialTS}(integrator, GeneralisedNoUTurn()))
 adaptor = StanHMCAdaptor(MassMatrixAdaptor(metric), StepSizeAdaptor(0.45, integrator))
-samples, stats = sample(h, kernel, p, 500, adaptor, 500; progress = true)
([[-0.20195876267595317, -0.295888890818682, 0.1662975020168111, 0.11969505376051837, -0.10541647864435945, 0.20240011763585647, -0.25975764989626765, 0.22512779049136045, 0.15103335890539243, -0.12453813720421568  …  0.1325867910434564, -0.27468069855830995, -0.18762779708105376, -0.215585694615452, 0.14883444765602305, 0.11571766320590968, -0.035623720035776185, 0.23568143981828252, -0.05774864770964133, 0.01324054902501625], [-0.20195876267595317, -0.295888890818682, 0.1662975020168111, 0.11969505376051837, -0.10541647864435945, 0.20240011763585647, -0.25975764989626765, 0.22512779049136045, 0.15103335890539243, -0.12453813720421568  …  0.1325867910434564, -0.27468069855830995, -0.18762779708105376, -0.215585694615452, 0.14883444765602305, 0.11571766320590968, -0.035623720035776185, 0.23568143981828252, -0.05774864770964133, 0.01324054902501625], [-0.20195876267595317, -0.295888890818682, 0.1662975020168111, 0.11969505376051837, -0.10541647864435945, 0.20240011763585647, -0.25975764989626765, 0.22512779049136045, 0.15103335890539243, -0.12453813720421568  …  0.1325867910434564, -0.27468069855830995, -0.18762779708105376, -0.215585694615452, 0.14883444765602305, 0.11571766320590968, -0.035623720035776185, 0.23568143981828252, -0.05774864770964133, 0.01324054902501625], [-0.01063868459176577, -0.30203759474730535, 0.4350741050444473, 0.32382495955422597, -0.16265723466732526, 0.20361181368772732, -0.2318922958206614, 0.17251468692328004, 0.2589206084437615, -0.2599537618028735  …  0.09247425815593843, -0.14913541335452848, -0.2557885798679649, -0.017459005474394454, 0.19975488288205642, -0.04882018221765329, 0.3659476050322732, -0.04800561142836687, -0.6109491715364483, 0.039170875871582286], [-0.01063868459176577, -0.30203759474730535, 0.4350741050444473, 0.32382495955422597, -0.16265723466732526, 0.20361181368772732, -0.2318922958206614, 0.17251468692328004, 0.2589206084437615, -0.2599537618028735  …  0.09247425815593843, -0.14913541335452848, -0.2557885798679649, -0.017459005474394454, 0.19975488288205642, -0.04882018221765329, 0.3659476050322732, -0.04800561142836687, -0.6109491715364483, 0.039170875871582286], [-0.2628048500381224, 0.36382683613807787, 0.5473839025159579, 0.5534955410407074, -0.12913231571318234, 0.09177389361754668, 0.11252606301004123, 0.37002305696401244, 0.20313483198484303, -0.24024506786146835  …  -0.2590586234510648, -0.14846385952985813, -0.24006092521758357, 0.009805736232099162, 0.5768544049813645, 0.2783270589014398, 0.21041312943507628, -0.13694193815375014, -0.7965079764367498, -0.33261004212671824], [-0.2628048500381224, 0.36382683613807787, 0.5473839025159579, 0.5534955410407074, -0.12913231571318234, 0.09177389361754668, 0.11252606301004123, 0.37002305696401244, 0.20313483198484303, -0.24024506786146835  …  -0.2590586234510648, -0.14846385952985813, -0.24006092521758357, 0.009805736232099162, 0.5768544049813645, 0.2783270589014398, 0.21041312943507628, -0.13694193815375014, -0.7965079764367498, -0.33261004212671824], [-0.5130162285979032, 0.27882554507359153, 0.07461818214032315, 0.6008216811508267, -0.2848687846346512, 0.16876278109980172, 0.16243580990828055, 0.7423299008069145, 0.12311197128923522, -0.10581045697542651  …  -0.19177632793376276, -0.022181033127268716, -0.1743692005554265, 0.2674793526633006, 0.3839670153548103, 0.06926223570995862, 0.5813611663785454, -0.2423082596372466, -1.177425000526609, 0.09725894133939575], [-0.3522644679370758, 0.24190311004206588, 0.06619960347690382, 0.5998416337982442, -0.08208782410908647, 0.09791629582469905, -0.0023754669384323523, 0.7507277276398213, 0.17522380838680437, 0.14645764697662067  …  -0.1956780930728858, -0.08402574812253183, -0.2545522518212365, 0.24385793591269717, 0.2655493508811622, -0.08536520927699714, 0.6479527382798499, -0.22138842935395997, -1.2083716341987758, 0.014577140445390935], [-0.244092366817952, 0.4183458755957361, 0.03166144612457119, 0.6636013484198847, -0.11575764923991892, -0.12637210334953325, 0.17817817507100556, 0.5298627923131769, -0.027610650414718788, 0.21724888550103397  …  -0.12185677471306847, -0.07941714771275109, -0.3453043021406518, 0.36267265262915493, 0.5106566186785413, -0.0612726197860721, 0.526993260845142, -0.19925136242077204, -1.190982559276584, 0.05693534966062949]  …  [-0.6540384496617972, 0.20959603157720916, -0.4610368053959579, -0.059451826899776114, -0.7644825171684507, -0.19478786862401162, -0.443774135858259, 0.9917111430200972, 0.8360804336332271, 0.542593331691558  …  -0.028920145198117322, -0.022912745767317853, 1.1370169557540195, -0.8381621843512818, -0.5359076306968904, 0.2851182910229418, 1.2733825697376693, -0.7655325207285797, -0.6834627714575232, 0.3465548501002312], [0.7189421776245865, 1.4571125589969824, -0.046730990987365474, -0.2898956095784886, 0.1618187786936289, -0.14097105475228455, -0.20932162496524803, 0.23631807667185742, -1.2829648304139054, 0.34909254793279415  …  -0.24134803129227947, -1.0201930595670399, 0.18435903212130947, -0.06409366578691958, -0.24043375254122723, -0.5691863591568159, -0.6116167107378795, 0.49724360745869794, -0.08776697334435163, 0.5741375020286772], [0.7229803541493339, 1.3688413446603653, -0.05918744894460669, -0.28696294124933575, 0.22101516927667642, -0.22527410817634042, -0.24734391105261608, 0.24218872998743943, -1.3127311386174165, 0.3199051652023352  …  -0.2082456268889229, -1.113596333447264, 0.16290659115253295, 0.0016803059388919295, -0.3106309839659377, -0.531410328198809, -0.5397853659023535, 0.5278978854320676, -0.17002897172639223, 0.6165192872600933], [-0.2393996960383563, -1.539562089904845, -0.01872464020778533, -0.6029589482465989, -0.06039544958048246, -0.8591187187993209, 0.27344841808932535, -0.49348577027871415, 1.7931756763698274, 0.2463424020416241  …  -1.050396318611741, -0.34843175190653836, 0.49749920887603405, 0.14425768087835528, 1.166581936650357, 0.8293447094823717, 0.39099454902042685, -0.43151282160521354, -0.4955865865509851, 0.587958106517797], [-0.19696578072409135, -1.5896399979203935, 0.004384908813991643, -0.5292026171015116, 0.034845576838305276, -0.6088992181418081, 0.32227974067516196, -0.5163611197678781, 1.8298489167839778, 0.23865341185820887  …  -0.9626634504535343, -0.3331749898616082, 0.4742155142897585, 0.0645902050483428, 1.1845746203180638, 0.7639915068786245, 0.3211240558502264, -0.37141148863773504, -0.47083260247824243, 0.604088920919415], [-0.12267929462813751, 0.9226584056403362, -0.7037474869241253, 0.8422308323074238, -0.04121649746038319, 1.1151661868877019, -0.021439977245173915, 0.16126747281033052, -1.6430624089361299, -1.333941683467823  …  1.622140406119137, -0.354666618306937, -0.953547950732601, 0.5239658922738892, -0.5135546508779886, 0.0422746845336749, -0.7326168032618618, -0.1714970159965355, -0.43497197147707856, -0.12483495958148856], [-0.6612725777077195, 0.8916641570211171, -0.6945039711883678, -0.26796809765529167, -0.8386734494466127, 0.8476721749190959, -0.6764724286524583, 1.5954321747950475, -2.0482414032299165, 0.05250036288317849  …  0.6221641349439341, -0.6820367573896906, 0.4153295444765244, -1.004942885779065, 0.08792304358770131, -0.6592057141982548, -1.0243362674894074, 0.675128757122807, 0.33334250431913354, 1.1309252228820719], [-0.5303427071600754, 0.593831287951956, -0.48886129575999643, -0.035994173168126274, -0.6231747309684597, 1.0127725797808524, -0.44783717207224144, 0.9547149530042321, -1.492366499229945, 0.20602035161194965  …  0.6560501389173528, -0.8951992469506258, 0.39846044542710607, -0.973742268485942, -0.15731768722183376, -0.3792180620751779, -0.5769211577561462, 0.46716370783554334, 0.1116328656315099, 0.8874168189518616], [-0.8277505950586208, 0.416543011406437, -0.6018547890335453, 0.39022423685713425, -0.3673315354061011, -0.7665148856883363, -0.11276736570460288, 0.4661797022896723, 0.43813665447566547, 0.27627256257143057  …  1.636730216559227, -1.011940064322991, -1.185346968062319, 0.2524262603231247, -0.6310165269306147, -0.41722067310940203, -0.4747372282766852, -0.7685022569577759, 0.2822811761229283, -0.2914250116340868], [0.5359878270694306, 0.8941702482344012, 0.13254635720515648, 0.5062183078293413, -0.6498585097976213, -0.3995270334581228, 0.3399050599593619, 1.216515046050826, 1.0050802379463903, -0.04184755574397878  …  0.8802546837581103, -1.1688354917277426, -0.5907125688417723, -0.2791503611033702, -1.2749766913436908, 0.13437120406751646, 0.5076205798896505, -1.7272856398053649, 1.086889171735326, 0.13875778069680417]], NamedTuple[(n_steps = 15, is_accept = true, acceptance_rate = 1.0, log_density = -239.20695997096897, hamiltonian_energy = 487.776077320803, hamiltonian_energy_error = -31.09565989499106, max_hamiltonian_energy_error = -31.09565989499106, tree_depth = 4, numerical_error = false, step_size = 0.025, nom_step_size = 0.025, is_adapt = true), (n_steps = 1, is_accept = true, acceptance_rate = 0.0, log_density = -239.20695997096897, hamiltonian_energy = 363.49249704056274, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 8703.965163263738, tree_depth = 0, numerical_error = true, step_size = 0.6795704571147614, nom_step_size = 0.6795704571147614, is_adapt = true), (n_steps = 4, is_accept = true, acceptance_rate = 1.0113829159801262e-157, log_density = -239.20695997096897, hamiltonian_energy = 356.53105243032485, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 231794.75395363185, tree_depth = 2, numerical_error = true, step_size = 0.3164493440113639, nom_step_size = 0.3164493440113639, is_adapt = true), (n_steps = 9, is_accept = true, acceptance_rate = 0.8115662089225205, log_density = -127.55919195124403, hamiltonian_energy = 336.91454140730184, hamiltonian_energy_error = -12.920849107465585, max_hamiltonian_energy_error = 5787.992805546289, tree_depth = 3, numerical_error = true, step_size = 0.09837809577101383, nom_step_size = 0.09837809577101383, is_adapt = true), (n_steps = 2, is_accept = true, acceptance_rate = 1.7316035861225546e-13, log_density = -127.55919195124403, hamiltonian_energy = 243.02222557679386, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 37035.18632694272, tree_depth = 1, numerical_error = true, step_size = 0.2583996013307102, nom_step_size = 0.2583996013307102, is_adapt = true), (n_steps = 11, is_accept = true, acceptance_rate = 0.7886191039802813, log_density = -104.1425361893657, hamiltonian_energy = 258.8133179193627, hamiltonian_energy_error = 0.002794694247540974, max_hamiltonian_energy_error = 4157.534305605343, tree_depth = 3, numerical_error = true, step_size = 0.06764706062195602, nom_step_size = 0.06764706062195602, is_adapt = true), (n_steps = 1, is_accept = true, acceptance_rate = 0.0, log_density = -104.1425361893657, hamiltonian_energy = 225.32618973891758, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 1725.5850759499654, tree_depth = 0, numerical_error = true, step_size = 0.18416684307251557, nom_step_size = 0.18416684307251557, is_adapt = true), (n_steps = 17, is_accept = true, acceptance_rate = 0.7344687287226063, log_density = -91.0361916460281, hamiltonian_energy = 235.2843668806269, hamiltonian_energy_error = -0.19967133892262723, max_hamiltonian_energy_error = 1095.7890568114951, tree_depth = 4, numerical_error = true, step_size = 0.045154096250633234, nom_step_size = 0.045154096250633234, is_adapt = true), (n_steps = 5, is_accept = true, acceptance_rate = 0.30895699274845756, log_density = -89.82947930263427, hamiltonian_energy = 203.6483622872051, hamiltonian_energy_error = -0.984481841093725, max_hamiltonian_energy_error = 2658.2007994009987, tree_depth = 2, numerical_error = true, step_size = 0.10858779564080324, nom_step_size = 0.10858779564080324, is_adapt = true), (n_steps = 6, is_accept = true, acceptance_rate = 0.3156152996862584, log_density = -78.81254532349969, hamiltonian_energy = 221.9433804011028, hamiltonian_energy_error = 0.15850653078311439, max_hamiltonian_energy_error = 3395.572656910815, tree_depth = 2, numerical_error = true, step_size = 0.06927799630915323, nom_step_size = 0.06927799630915323, is_adapt = true)  …  (n_steps = 127, is_accept = true, acceptance_rate = 0.40415270489839855, log_density = -127.30619977478361, hamiltonian_energy = 235.20552784948325, hamiltonian_energy_error = -0.06043814526066171, max_hamiltonian_energy_error = 41.33597630825358, tree_depth = 7, numerical_error = false, step_size = 0.03278060904852817, nom_step_size = 0.03278060904852817, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.7320488095277013, log_density = -135.39035250000504, hamiltonian_energy = 250.51493153761254, hamiltonian_energy_error = 0.7660064792663945, max_hamiltonian_energy_error = 0.9395336560788508, tree_depth = 7, numerical_error = false, step_size = 0.02981872072444736, nom_step_size = 0.02981872072444736, is_adapt = true), (n_steps = 26, is_accept = true, acceptance_rate = 0.04493494397083665, log_density = -135.12537688252945, hamiltonian_energy = 281.14379540912705, hamiltonian_energy_error = -0.7375259400469645, max_hamiltonian_energy_error = 2016.2729014043975, tree_depth = 4, numerical_error = true, step_size = 0.06150080045894998, nom_step_size = 0.06150080045894998, is_adapt = true), (n_steps = 255, is_accept = true, acceptance_rate = 0.7911267612786576, log_density = -131.3318642953299, hamiltonian_energy = 267.10586522689374, hamiltonian_energy_error = 0.24221788748201334, max_hamiltonian_energy_error = 0.9593839445064987, tree_depth = 8, numerical_error = false, step_size = 0.022922036932219895, nom_step_size = 0.022922036932219895, is_adapt = true), (n_steps = 63, is_accept = true, acceptance_rate = 0.09997144805151288, log_density = -129.78987543370476, hamiltonian_energy = 257.9339522494364, hamiltonian_energy_error = 0.23045903918512067, max_hamiltonian_energy_error = 539.5692438345445, tree_depth = 6, numerical_error = false, step_size = 0.05418732368255087, nom_step_size = 0.05418732368255087, is_adapt = true), (n_steps = 255, is_accept = true, acceptance_rate = 0.495492347028984, log_density = -131.44169262632423, hamiltonian_energy = 266.07528073861135, hamiltonian_energy_error = 0.5963485887201614, max_hamiltonian_energy_error = 74.19430998823623, tree_depth = 8, numerical_error = false, step_size = 0.023441994203610088, nom_step_size = 0.023441994203610088, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.8261769674397197, log_density = -148.02314138659503, hamiltonian_energy = 268.80592274117396, hamiltonian_energy_error = -0.40379552348963443, max_hamiltonian_energy_error = -1.7635699891350214, tree_depth = 7, numerical_error = false, step_size = 0.02674450276237929, nom_step_size = 0.02674450276237929, is_adapt = true), (n_steps = 17, is_accept = true, acceptance_rate = 0.34373412048165164, log_density = -145.7893910943538, hamiltonian_energy = 270.36843884874986, hamiltonian_energy_error = 0.2574931094245585, max_hamiltonian_energy_error = 1567.3525656577517, tree_depth = 4, numerical_error = true, step_size = 0.06746845941716278, nom_step_size = 0.06746845941716278, is_adapt = true), (n_steps = 63, is_accept = true, acceptance_rate = 0.44035855519836925, log_density = -121.17099892637025, hamiltonian_energy = 259.9925357095706, hamiltonian_energy_error = 1.2056429593816347, max_hamiltonian_energy_error = 61.2101143278835, tree_depth = 6, numerical_error = false, step_size = 0.05307245389341003, nom_step_size = 0.05307245389341003, is_adapt = true), (n_steps = 63, is_accept = true, acceptance_rate = 0.7364592440394202, log_density = -150.15507960243528, hamiltonian_energy = 253.11925077371774, hamiltonian_energy_error = -1.5753967732196656, max_hamiltonian_energy_error = 112.37609472509718, tree_depth = 6, numerical_error = false, step_size = 0.05267347843524629, nom_step_size = 0.05267347843524629, is_adapt = true)])

Step 5: Plot diagnostics

Now let's make sure the fit is good. This can be done by looking at the chain mixing plot and the autocorrelation plot. First, let's create the chain mixing plot using the plot recipes from ????

samples = hcat(samples...)
+samples, stats = sample(h, kernel, p, 500, adaptor, 500; progress = true)
([[0.20851658930140707, -0.14403216239052466, -0.0714564967210835, -0.263824531789019, -0.2368106672607327, 0.07062313291054406, 0.14667674975477393, 0.18439265069160635, 0.14129910148377092, 0.250363887548148  …  0.11806689818923011, 0.005726960010948773, -0.11791550527132355, -0.2558172580453225, 0.0217105328133365, 0.168207430048538, -0.2029542948128623, -0.12796037905742927, -0.05684335600091137, -0.01883694010071343], [0.20851658930140707, -0.14403216239052466, -0.0714564967210835, -0.263824531789019, -0.2368106672607327, 0.07062313291054406, 0.14667674975477393, 0.18439265069160635, 0.14129910148377092, 0.250363887548148  …  0.11806689818923011, 0.005726960010948773, -0.11791550527132355, -0.2558172580453225, 0.0217105328133365, 0.168207430048538, -0.2029542948128623, -0.12796037905742927, -0.05684335600091137, -0.01883694010071343], [0.20851658930140707, -0.14403216239052466, -0.0714564967210835, -0.263824531789019, -0.2368106672607327, 0.07062313291054406, 0.14667674975477393, 0.18439265069160635, 0.14129910148377092, 0.250363887548148  …  0.11806689818923011, 0.005726960010948773, -0.11791550527132355, -0.2558172580453225, 0.0217105328133365, 0.168207430048538, -0.2029542948128623, -0.12796037905742927, -0.05684335600091137, -0.01883694010071343], [0.5935415622731366, -0.33076328281980794, 0.007440026085803311, -0.6638990922736248, -0.43296653475011865, 0.1297117934849005, 0.3958772028826445, 0.12856578185111356, 0.40104046754920786, 0.5448429328115776  …  0.040147938787318066, 0.18800881948877426, -0.09387022093714734, -0.31693135690173746, 0.18128462418640345, 0.3697327316693706, -0.33769456048719326, -0.27159365636487043, -0.4879858363329687, 0.07217389862526688], [0.5589024390660241, -0.16058484551317537, -0.37835529449410743, -0.6461042864368166, -0.5705906294435407, -0.16113335500692713, 0.05382276731726175, 0.17154978905052048, 0.9769919630089083, 0.48263766900351546  …  -0.11637042318715898, 0.6966136992554962, -0.3168908382362252, -0.41818177619928654, 0.556996020621807, 0.18281790963553962, -0.31435558930983387, -0.9536070339551311, -0.8214348262303965, 0.2844029610559024], [0.5589024390660241, -0.16058484551317537, -0.37835529449410743, -0.6461042864368166, -0.5705906294435407, -0.16113335500692713, 0.05382276731726175, 0.17154978905052048, 0.9769919630089083, 0.48263766900351546  …  -0.11637042318715898, 0.6966136992554962, -0.3168908382362252, -0.41818177619928654, 0.556996020621807, 0.18281790963553962, -0.31435558930983387, -0.9536070339551311, -0.8214348262303965, 0.2844029610559024], [0.4710192381793823, -0.16275639363576028, -0.5077346799457493, -0.6605919802378555, -0.5614259372642386, -0.16242644007563076, 0.20125102793308594, 0.13151591963202242, 1.0378781663785086, 0.461084919169054  …  -0.12783924842711525, 0.6740776063714456, -0.4203499083806923, -0.44933817800953096, 0.5634739676616467, 0.05861567466575501, -0.22673543617458475, -0.8921448757352997, -0.6697532022999294, 0.23908733049052278], [0.17911086738481874, -0.36815614666924756, 0.09448386242409608, -0.9248260148479103, 0.9115807494203861, 1.0739159508767366, -0.02674470056549238, -0.9571829991946004, 0.7039257777897954, 0.4046030637272377  …  0.42421408245323283, -0.33528709092465936, 0.4948668227733954, 0.7711274942864375, -0.666129181088873, 0.1299834743662406, 1.403469596124325, 0.8141417444339663, -0.15698759919863248, 0.6711181203345471], [0.2555127444965276, -0.253021902998301, 0.11470322335012245, -1.4221871077550594, 0.7809236353272083, 1.014735251495782, -0.03968173302276186, -1.076583278212058, 1.1325996499910642, 0.3154486820185839  …  0.6096749473174514, -0.2922885786095326, 0.8609199115594542, 0.6474602610713018, -0.14634776478549627, -0.29383945770154424, 1.2297607391849572, 0.8597997417193167, -0.08940940671474332, 0.3620952559449425], [0.1966407177642283, -0.04365496246803913, 0.09116209797124762, -1.208403603252987, 0.3888663847780302, 0.6728039407101589, -0.22958880753333097, -0.9280241623874986, 1.016305354284355, -0.10339922229678666  …  0.27989004593172884, -0.2368276737732872, 1.1219841347009198, 0.772806910737438, -0.27148821499543563, -0.1179684507504534, 1.1879606507254286, 0.3811650489548347, -0.044182203116034977, 0.4973924295139931]  …  [-0.2989228980592161, -0.6708161773599458, 0.4226910621046258, 0.5325855838092963, 0.4671421197617571, 0.6135006882375882, 0.2379051245849957, -1.4453591234213492, 0.00024199590210423816, 0.32864414723663504  …  0.08898369804457243, -0.7192193826142812, 0.8904846019045278, 0.263727476521352, 0.0799802248468427, -1.107837754547293, -0.6549249049598413, 1.029619469448303, -0.3890956836276514, 0.5597502930056677], [-0.8739796704741278, 0.16425619914882061, -0.0912351658569784, -0.0166659536912472, 0.3931767854707051, 0.4911188043480863, 0.2776449799897353, 0.4991364433487223, 0.06964385964786358, 0.7803081676801942  …  0.6047052890017711, 0.9950559680810093, 1.4564528461166988, -1.4605911649236902, 0.10347187135779486, 1.3056871589998913, -0.9292645166010004, -0.22379747062708039, -1.2362954964232014, 0.6924807401659867], [-0.8739796704741278, 0.16425619914882061, -0.0912351658569784, -0.0166659536912472, 0.3931767854707051, 0.4911188043480863, 0.2776449799897353, 0.4991364433487223, 0.06964385964786358, 0.7803081676801942  …  0.6047052890017711, 0.9950559680810093, 1.4564528461166988, -1.4605911649236902, 0.10347187135779486, 1.3056871589998913, -0.9292645166010004, -0.22379747062708039, -1.2362954964232014, 0.6924807401659867], [1.2870281190071304, 0.05403673210964098, 0.1849873719195747, 0.8555297484101422, 0.737379116947209, -0.5403937568918937, 0.3278351365827167, 0.1908802413784991, -0.28140793029731964, -0.8720957356463583  …  -0.32609495644822856, -1.0535492249003995, -1.1946197170537185, 0.8330804522858156, -0.7925623410153247, -0.5420531680020299, 0.8055600383293795, 0.29451595789406104, 0.17213864389591776, -0.008529830982010139], [1.339186493756209, 0.07500925945675496, 0.200398807448539, 0.8159667766761284, 0.7311549648762751, -0.4921207524962361, 0.33154927950856733, 0.15711119787946234, -0.2832075704826198, -0.9101401433063394  …  -0.30551574879563825, -1.0724300623006047, -1.2006727466878933, 0.9210708620038118, -0.8194031114438222, -0.5137795284377212, 0.8036141012945665, 0.26101274284935605, 0.18049151900020938, 0.01449901289299001], [1.1220538569034546, 0.47059187918063095, 0.3504339388379416, -0.6575155901157298, 0.6904789471221815, -0.2248107276535688, 0.014035168483079782, 0.5159156502870177, 0.43773322669044024, -0.48047791563529174  …  0.2691484107742369, -1.583258517015795, -0.8993386170066138, 0.8613405660283897, -2.277644190604152, 0.34870212231545294, 0.9881946739998708, 0.1460009559926363, 0.18650835493149442, 0.48558255346002593], [0.3768320785729483, -0.8247979574471489, 0.5809933220039004, -0.3164738538868977, 1.0369477791306523, 0.7493447239908617, -0.3155492844393437, -0.3876436481642083, -0.3088684608648602, -0.5829523822500583  …  1.1700693531662776, -0.337139936862073, -1.5422029345571056, 2.355904843077833, -0.6018895298265838, -0.4899308373348371, 0.03968038370264613, -1.1231486359189855, -0.786242947977857, 0.4933474011649509], [0.04386098610652933, -0.7851041200915707, 0.7356045312219996, -0.45564081480681734, 0.9223147061406979, 0.49406275730564997, -0.0862333670674578, -0.5676379087022283, -0.47960883217086536, -0.6214940056902928  …  1.256232334909616, -0.3495225043673748, -1.4367119997323958, 2.2854669754555887, -0.580997553794816, -0.546544924054245, 0.006427897386274414, -0.8580536833007388, -0.8747553840059771, 0.5885637245276606], [0.04386098610652933, -0.7851041200915707, 0.7356045312219996, -0.45564081480681734, 0.9223147061406979, 0.49406275730564997, -0.0862333670674578, -0.5676379087022283, -0.47960883217086536, -0.6214940056902928  …  1.256232334909616, -0.3495225043673748, -1.4367119997323958, 2.2854669754555887, -0.580997553794816, -0.546544924054245, 0.006427897386274414, -0.8580536833007388, -0.8747553840059771, 0.5885637245276606], [0.08419563461513827, -1.1929477403003859, 0.3719778639513844, -0.5666453019894129, 0.256815618957385, 0.18300325820131733, 0.0018417191537886115, -0.7861207088219604, -0.5024635357467634, -0.9744091521380951  …  0.8136114563852177, -0.5646230911158823, -0.9141503843272463, 2.298977946089083, -0.4546193056788054, -0.082149625139919, 0.49754957906943065, -0.07166681806504988, -0.9730174067540645, 0.5557003514899646]], NamedTuple[(n_steps = 16, is_accept = true, acceptance_rate = 0.9375, log_density = -200.23437066988697, hamiltonian_energy = 394.1564621644343, hamiltonian_energy_error = -17.877622870615937, max_hamiltonian_energy_error = 27007.64099990863, tree_depth = 4, numerical_error = true, step_size = 0.05, nom_step_size = 0.05, is_adapt = true), (n_steps = 1, is_accept = true, acceptance_rate = 0.0, log_density = -200.23437066988697, hamiltonian_energy = 311.3533539509564, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 3.937663015011049e6, tree_depth = 0, numerical_error = true, step_size = 1.2131453574546924, nom_step_size = 1.2131453574546924, is_adapt = true), (n_steps = 2, is_accept = true, acceptance_rate = 9.367017622781803e-210, log_density = -200.23437066988697, hamiltonian_energy = 365.32366542629063, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 2.0996498102410645e6, tree_depth = 1, numerical_error = true, step_size = 0.5462061377007653, nom_step_size = 0.5462061377007653, is_adapt = true), (n_steps = 5, is_accept = true, acceptance_rate = 0.2000056885230535, log_density = -139.0573890414192, hamiltonian_energy = 326.6050260977959, hamiltonian_energy_error = -3.753965310832143, max_hamiltonian_energy_error = 300716.7192273534, tree_depth = 2, numerical_error = true, step_size = 0.16657106064026125, nom_step_size = 0.16657106064026125, is_adapt = true), (n_steps = 4, is_accept = true, acceptance_rate = 0.75, log_density = -91.73251368393824, hamiltonian_energy = 282.8730285262278, hamiltonian_energy_error = -2.7291923611851985, max_hamiltonian_energy_error = 17442.39980434212, tree_depth = 2, numerical_error = true, step_size = 0.07532161975135283, nom_step_size = 0.07532161975135283, is_adapt = true), (n_steps = 1, is_accept = true, acceptance_rate = 0.0, log_density = -91.73251368393824, hamiltonian_energy = 211.99301819409857, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 7660.793672964839, tree_depth = 0, numerical_error = true, step_size = 0.1696705930216829, nom_step_size = 0.1696705930216829, is_adapt = true), (n_steps = 72, is_accept = true, acceptance_rate = 0.11803786751152456, log_density = -86.86609111859295, hamiltonian_energy = 212.57572867835543, hamiltonian_energy_error = -0.43645803759599744, max_hamiltonian_energy_error = 3858.5331531075262, tree_depth = 6, numerical_error = true, step_size = 0.04154854471929164, nom_step_size = 0.04154854471929164, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.6498915329719174, log_density = -139.6754150510992, hamiltonian_energy = 215.53607472546486, hamiltonian_energy_error = 1.142309551701942, max_hamiltonian_energy_error = 1.6313296340249508, tree_depth = 7, numerical_error = false, step_size = 0.0141868765252456, nom_step_size = 0.0141868765252456, is_adapt = true), (n_steps = 81, is_accept = true, acceptance_rate = 0.542710968127224, log_density = -138.61800406752874, hamiltonian_energy = 276.10866862912553, hamiltonian_energy_error = -0.2361804903297866, max_hamiltonian_energy_error = 1092.4889673282282, tree_depth = 6, numerical_error = true, step_size = 0.025690646346633154, nom_step_size = 0.025690646346633154, is_adapt = true), (n_steps = 173, is_accept = true, acceptance_rate = 0.0714964676474695, log_density = -130.82182498160006, hamiltonian_energy = 266.9540886802415, hamiltonian_energy_error = -3.9017678209000906, max_hamiltonian_energy_error = 1422.0700943677548, tree_depth = 7, numerical_error = true, step_size = 0.03393826081914554, nom_step_size = 0.03393826081914554, is_adapt = true)  …  (n_steps = 511, is_accept = true, acceptance_rate = 0.6835414770956141, log_density = -138.23046317044083, hamiltonian_energy = 257.49877963156473, hamiltonian_energy_error = 0.4301839541344634, max_hamiltonian_energy_error = 2.004869583136383, tree_depth = 9, numerical_error = false, step_size = 0.011621647174864868, nom_step_size = 0.011621647174864868, is_adapt = true), (n_steps = 255, is_accept = true, acceptance_rate = 0.9612235478222795, log_density = -109.53516846504532, hamiltonian_energy = 238.96234443408707, hamiltonian_energy_error = -0.30403551233447956, max_hamiltonian_energy_error = -1.3112695790690054, tree_depth = 8, numerical_error = false, step_size = 0.021460244953606503, nom_step_size = 0.021460244953606503, is_adapt = true), (n_steps = 20, is_accept = true, acceptance_rate = 0.0020563541780161503, log_density = -109.53516846504532, hamiltonian_energy = 238.04624857114027, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 1620.5644404643815, tree_depth = 4, numerical_error = true, step_size = 0.07845783648843042, nom_step_size = 0.07845783648843042, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.7699059420821199, log_density = -121.53430356563837, hamiltonian_energy = 248.5358408681304, hamiltonian_energy_error = -0.2886186339990502, max_hamiltonian_energy_error = 73.63745253708129, tree_depth = 7, numerical_error = false, step_size = 0.02622058521676995, nom_step_size = 0.02622058521676995, is_adapt = true), (n_steps = 63, is_accept = true, acceptance_rate = 0.059027647749990086, log_density = -123.24247445059356, hamiltonian_energy = 249.59988752097792, hamiltonian_energy_error = 0.7979382648097157, max_hamiltonian_energy_error = 189.78263366108865, tree_depth = 6, numerical_error = false, step_size = 0.05871020668676345, nom_step_size = 0.05871020668676345, is_adapt = true), (n_steps = 255, is_accept = true, acceptance_rate = 0.5178281017646383, log_density = -138.66183473478986, hamiltonian_energy = 260.9559916177511, hamiltonian_energy_error = 0.01157672294027634, max_hamiltonian_energy_error = 9.373420248131765, tree_depth = 8, numerical_error = false, step_size = 0.022944639470881382, nom_step_size = 0.022944639470881382, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.9225676800809205, log_density = -124.65446465068905, hamiltonian_energy = 256.0767275327611, hamiltonian_energy_error = 0.021742849859776925, max_hamiltonian_energy_error = 0.639492192388218, tree_depth = 7, numerical_error = false, step_size = 0.027604648118847427, nom_step_size = 0.027604648118847427, is_adapt = true), (n_steps = 6, is_accept = true, acceptance_rate = 0.46063692620191726, log_density = -129.8254496901303, hamiltonian_energy = 245.3502356970231, hamiltonian_energy_error = 0.3212233725491842, max_hamiltonian_energy_error = 1316.0529821069485, tree_depth = 2, numerical_error = true, step_size = 0.08769138639088146, nom_step_size = 0.08769138639088146, is_adapt = true), (n_steps = 4, is_accept = true, acceptance_rate = 0.03551856922507148, log_density = -129.8254496901303, hamiltonian_energy = 248.32808525929897, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 1389.7583428803673, tree_depth = 2, numerical_error = true, step_size = 0.09093876529218092, nom_step_size = 0.09093876529218092, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.32423511203526956, log_density = -134.0927678991807, hamiltonian_energy = 248.9929398364308, hamiltonian_energy_error = -0.11265407318384746, max_hamiltonian_energy_error = 21.224880522906346, tree_depth = 7, numerical_error = false, step_size = 0.03437220892866808, nom_step_size = 0.03437220892866808, is_adapt = true)])

Step 5: Plot diagnostics

Now let's make sure the fit is good. This can be done by looking at the chain mixing plot and the autocorrelation plot. First, let's create the chain mixing plot using the plot recipes from ????

samples = hcat(samples...)
 samples_reduced = samples[1:5, :]
 samples_reshape = reshape(samples_reduced, (500, 5, 1))
 Chain_Spiral = Chains(samples_reshape)
-plot(Chain_Spiral)
Example block output

Now we check the autocorrelation plot:

autocorplot(Chain_Spiral)
Example block output

As another diagnostic, let's check the result on retrodicted data. To do this, we generate solutions of the Neural ODE on samples of the neural network parameters, and check the results of the predictions against the data. Let's start by looking at the time series:

pl = scatter(tsteps, ode_data[1, :], color = :red, label = "Data: Var1", xlabel = "t",
+plot(Chain_Spiral)
Example block output

Now we check the autocorrelation plot:

autocorplot(Chain_Spiral)
Example block output

As another diagnostic, let's check the result on retrodicted data. To do this, we generate solutions of the Neural ODE on samples of the neural network parameters, and check the results of the predictions against the data. Let's start by looking at the time series:

pl = scatter(tsteps, ode_data[1, :], color = :red, label = "Data: Var1", xlabel = "t",
              title = "Spiral Neural ODE")
 scatter!(tsteps, ode_data[2, :], color = :blue, label = "Data: Var2")
 for k in 1:300
@@ -75,11 +75,11 @@
 prediction = predict_neuralode(samples[:, idx])
 plot!(tsteps, prediction[1, :], color = :black, w = 2, label = "")
 plot!(tsteps, prediction[2, :], color = :black, w = 2, label = "Best fit prediction",
-      ylims = (-2.5, 3.5))
Example block output

That showed the time series form. We can similarly do a phase-space plot:

pl = scatter(ode_data[1, :], ode_data[2, :], color = :red, label = "Data", xlabel = "Var1",
+      ylims = (-2.5, 3.5))
Example block output

That showed the time series form. We can similarly do a phase-space plot:

pl = scatter(ode_data[1, :], ode_data[2, :], color = :red, label = "Data", xlabel = "Var1",
              ylabel = "Var2", title = "Spiral Neural ODE")
 for k in 1:300
     resol = predict_neuralode(samples[:, 100:end][:, rand(1:400)])
     plot!(resol[1, :], resol[2, :], alpha = 0.04, color = :red, label = "")
 end
 plot!(prediction[1, :], prediction[2, :], color = :black, w = 2,
-      label = "Best fit prediction", ylims = (-2.5, 3))
Example block output + label = "Best fit prediction", ylims = (-2.5, 3))Example block output diff --git a/dev/showcase/blackhole/6e6831b7.svg b/dev/showcase/blackhole/141a6e2a.svg similarity index 94% rename from dev/showcase/blackhole/6e6831b7.svg rename to dev/showcase/blackhole/141a6e2a.svg index 4245c7381c8..e19930ab034 100644 --- a/dev/showcase/blackhole/6e6831b7.svg +++ b/dev/showcase/blackhole/141a6e2a.svg @@ -1,50 +1,50 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/blackhole/92ef0486.svg b/dev/showcase/blackhole/1efc1e48.svg similarity index 93% rename from dev/showcase/blackhole/92ef0486.svg rename to dev/showcase/blackhole/1efc1e48.svg index 79b34ae2dec..53fb37521f1 100644 --- a/dev/showcase/blackhole/92ef0486.svg +++ b/dev/showcase/blackhole/1efc1e48.svg @@ -1,48 +1,48 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/blackhole/104f826e.svg b/dev/showcase/blackhole/6b039505.svg similarity index 76% rename from dev/showcase/blackhole/104f826e.svg rename to dev/showcase/blackhole/6b039505.svg index 3efd0192b4d..37062ffbb56 100644 --- a/dev/showcase/blackhole/104f826e.svg +++ b/dev/showcase/blackhole/6b039505.svg @@ -1,301 +1,301 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/blackhole/7bdbe226.svg b/dev/showcase/blackhole/84cb8445.svg similarity index 88% rename from dev/showcase/blackhole/7bdbe226.svg rename to dev/showcase/blackhole/84cb8445.svg index ad9681b9898..6fae2e61122 100644 --- a/dev/showcase/blackhole/7bdbe226.svg +++ b/dev/showcase/blackhole/84cb8445.svg @@ -1,50 +1,50 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/blackhole/8ef4842d.svg b/dev/showcase/blackhole/f90430a5.svg similarity index 86% rename from dev/showcase/blackhole/8ef4842d.svg rename to dev/showcase/blackhole/f90430a5.svg index 112d7e17a8f..30ae00b7de4 100644 --- a/dev/showcase/blackhole/8ef4842d.svg +++ b/dev/showcase/blackhole/f90430a5.svg @@ -1,54 +1,54 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/blackhole/index.html b/dev/showcase/blackhole/index.html index 722c5664c6e..3dc4a18b2c2 100644 --- a/dev/showcase/blackhole/index.html +++ b/dev/showcase/blackhole/index.html @@ -364,7 +364,7 @@ plt = plot(tsteps, waveform, markershape=:circle, markeralpha = 0.25, linewidth = 2, alpha = 0.5, - label="waveform data", xlabel="Time", ylabel="Waveform")Example block output

Looks great!

Automating the Discovery of Relativistic Equations from Newtonian Physics

Now let's learn the relativistic corrections directly from the data. To define the UDE, we will define a Lux neural network and pass it into our Newtonian Physics + Nerual Network ODE definition from above:

NN = Lux.Chain((x) -> cos.(x),
+           label="waveform data", xlabel="Time", ylabel="Waveform")
Example block output

Looks great!

Automating the Discovery of Relativistic Equations from Newtonian Physics

Now let's learn the relativistic corrections directly from the data. To define the UDE, we will define a Lux neural network and pass it into our Newtonian Physics + Nerual Network ODE definition from above:

NN = Lux.Chain((x) -> cos.(x),
     Lux.Dense(1, 32, cos),
     Lux.Dense(32, 32, cos),
     Lux.Dense(32, 2))
@@ -443,9 +443,9 @@
 Newt_orbit = soln2orbit(Newtonian_solution, model_params)
 plt = plot(true_orbit[1,:], true_orbit[2,:], linewidth = 2, label = "truth")
 plot!(plt, pred_orbit[1,:], pred_orbit[2,:], linestyle = :dash, linewidth = 2, label = "prediction")
-plot!(plt, Newt_orbit[1,:], Newt_orbit[2,:], linewidth = 2, label = "Newtonian")
Example block output
plt = plot(tsteps,true_waveform, linewidth = 2, label = "truth", xlabel="Time", ylabel="Waveform")
+plot!(plt, Newt_orbit[1,:], Newt_orbit[2,:], linewidth = 2, label = "Newtonian")
Example block output
plt = plot(tsteps,true_waveform, linewidth = 2, label = "truth", xlabel="Time", ylabel="Waveform")
 plot!(plt,tsteps,pred_waveform, linestyle = :dash, linewidth = 2, label = "prediction")
-plot!(plt,tsteps,Newt_waveform, linewidth = 2, label = "Newtonian")
Example block output

Now we'll do the same, but extrapolating the model out in time.

factor=5
+plot!(plt,tsteps,Newt_waveform, linewidth = 2, label = "Newtonian")
Example block output

Now we'll do the same, but extrapolating the model out in time.

factor=5
 
 extended_tspan = (tspan[1], factor*tspan[2])
 extended_tsteps = range(tspan[1], factor*tspan[2], length = factor*datasize)
@@ -461,9 +461,9 @@
 Newt_orbit = soln2orbit(Newtonian_solution, model_params)
 plt = plot(true_orbit[1,:], true_orbit[2,:], linewidth = 2, label = "truth")
 plot!(plt, pred_orbit[1,:], pred_orbit[2,:], linestyle = :dash, linewidth = 2, label = "prediction")
-plot!(plt, Newt_orbit[1,:], Newt_orbit[2,:], linewidth = 2, label = "Newtonian")
Example block output
true_waveform = compute_waveform(dt_data, reference_solution, mass_ratio, model_params)[1]
+plot!(plt, Newt_orbit[1,:], Newt_orbit[2,:], linewidth = 2, label = "Newtonian")
Example block output
true_waveform = compute_waveform(dt_data, reference_solution, mass_ratio, model_params)[1]
 pred_waveform = compute_waveform(dt_data, optimized_solution, mass_ratio, model_params)[1]
 Newt_waveform = compute_waveform(dt_data, Newtonian_solution, mass_ratio, model_params)[1]
 plt = plot(extended_tsteps,true_waveform, linewidth = 2, label = "truth", xlabel="Time", ylabel="Waveform")
 plot!(plt,extended_tsteps,pred_waveform, linestyle = :dash, linewidth = 2, label = "prediction")
-plot!(plt,extended_tsteps,Newt_waveform, linewidth = 2, label = "Newtonian")
Example block output +plot!(plt,extended_tsteps,Newt_waveform, linewidth = 2, label = "Newtonian")Example block output diff --git a/dev/showcase/brusselator/index.html b/dev/showcase/brusselator/index.html index 76a159a1e6b..4fb95025cab 100644 --- a/dev/showcase/brusselator/index.html +++ b/dev/showcase/brusselator/index.html @@ -378,4 +378,4 @@ ydomain:([0.0, 11.5], 0.0:0.03125:1.0, 0.0:0.03125:1.0) u: Dict{Symbolics.Num, Array{Float64, 3}} with 2 entries: u(x, y, t) => [0.0 0.115882 … 0.115882 0.0; 0.0 0.115882 … 0.115882 0.0; … ; … - v(x, y, t) => [0.0 0.0 … 0.0 0.0; 0.142219 0.142219 … 0.142219 0.142219; … ; …

And now we're zooming! For more information on these performance improvements, check out the deeper dive in the DifferentialEquations.jl tutorials.

If you're interested in figuring out what's the fastest current solver for this kind of PDE, check out the Brusselator benchmark in SciMLBenchmarks.jl

+ v(x, y, t) => [0.0 0.0 … 0.0 0.0; 0.142219 0.142219 … 0.142219 0.142219; … ; …

And now we're zooming! For more information on these performance improvements, check out the deeper dive in the DifferentialEquations.jl tutorials.

If you're interested in figuring out what's the fastest current solver for this kind of PDE, check out the Brusselator benchmark in SciMLBenchmarks.jl

diff --git a/dev/showcase/gpu_spde/761e69dc.svg b/dev/showcase/gpu_spde/11214966.svg similarity index 88% rename from dev/showcase/gpu_spde/761e69dc.svg rename to dev/showcase/gpu_spde/11214966.svg index e8913e7bbce..d7c0bb107c0 100644 --- a/dev/showcase/gpu_spde/761e69dc.svg +++ b/dev/showcase/gpu_spde/11214966.svg @@ -1,56 +1,56 @@ - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + diff --git a/dev/showcase/gpu_spde/5a053849.svg b/dev/showcase/gpu_spde/35cbe00e.svg similarity index 87% rename from dev/showcase/gpu_spde/5a053849.svg rename to dev/showcase/gpu_spde/35cbe00e.svg index 16660c184ef..6aaf8d7c593 100644 --- a/dev/showcase/gpu_spde/5a053849.svg +++ b/dev/showcase/gpu_spde/35cbe00e.svg @@ -1,58 +1,58 @@ - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + diff --git a/dev/showcase/gpu_spde/5b152400.svg b/dev/showcase/gpu_spde/5b152400.svg new file mode 100644 index 00000000000..0dce604cfef --- /dev/null +++ b/dev/showcase/gpu_spde/5b152400.svg @@ -0,0 +1,1565 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/gpu_spde/d5f23e83.svg b/dev/showcase/gpu_spde/d5f23e83.svg deleted file mode 100644 index 11f497b8cc0..00000000000 --- a/dev/showcase/gpu_spde/d5f23e83.svg +++ /dev/null @@ -1,1534 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dev/showcase/gpu_spde/index.html b/dev/showcase/gpu_spde/index.html index b878e153357..97de520d058 100644 --- a/dev/showcase/gpu_spde/index.html +++ b/dev/showcase/gpu_spde/index.html @@ -206,7 +206,7 @@ p1 = surface(X, Y, sol[end][:, :, 1], title = "[A]") p2 = surface(X, Y, sol[end][:, :, 2], title = "[B]") p3 = surface(X, Y, sol[end][:, :, 3], title = "[C]") -plot(p1, p2, p3, layout = grid(3, 1))Example block output

and see the pretty gradients. Using this 2nd order ROCK method we solve this equation in about 2 seconds. That's okay.

Some Optimizations

There are some optimizations that can still be done. When we do A*B as matrix multiplication, we create another temporary matrix. These allocations can bog down the system. Instead, we can pre-allocate the outputs and use the inplace functions mul! to make better use of memory. The easiest way to store these cache arrays are constant globals, but you can use closures (anonymous functions which capture data, i.e. (x)->f(x,y)) or call-overloaded types to do it without globals. The globals way (the easy way) is simply:

const MyA = zeros(N, N)
+plot(p1, p2, p3, layout = grid(3, 1))
Example block output

and see the pretty gradients. Using this 2nd order ROCK method we solve this equation in about 2 seconds. That's okay.

Some Optimizations

There are some optimizations that can still be done. When we do A*B as matrix multiplication, we create another temporary matrix. These allocations can bog down the system. Instead, we can pre-allocate the outputs and use the inplace functions mul! to make better use of memory. The easiest way to store these cache arrays are constant globals, but you can use closures (anonymous functions which capture data, i.e. (x)->f(x,y)) or call-overloaded types to do it without globals. The globals way (the easy way) is simply:

const MyA = zeros(N, N)
 const AMx = zeros(N, N)
 const DA = zeros(N, N)
 function f(du, u, p, t)
@@ -329,7 +329,7 @@
 p1 = surface(X, Y, sol[end][:, :, 1], title = "[A]")
 p2 = surface(X, Y, sol[end][:, :, 2], title = "[B]")
 p3 = surface(X, Y, sol[end][:, :, 3], title = "[C]")
-plot(p1, p2, p3, layout = grid(3, 1))
Example block output

Making Use of GPU Parallelism

That was all using the CPU. How do we turn on GPU parallelism with DifferentialEquations.jl? Well, you don't. DifferentialEquations.jl "doesn't have GPU bits". So, wait... can we not do GPU parallelism? No, this is the glory of type-genericness, especially in broadcasted operations. To make things use the GPU, we simply use a CuArray from CUDA.jl. If instead of zeros(N,M) we used CuArray(zeros(N,M)), then the array lives on the GPU. CuArray naturally overrides broadcast such that dotted operations are performed on the GPU. DifferentialEquations.jl uses broadcast internally, and thus just by putting the array as a CuArray, the array-type will take over how all internal updates are performed and turn this algorithm into a fully GPU-parallelized algorithm that doesn't require copying to the CPU. Wasn't that simple?

From that you can probably also see how to multithread everything, or how to set everything up with distributed parallelism. You can make the ODE solvers do whatever you want by defining an array type where the broadcast does whatever special behavior you want.

So to recap, the entire difference from above is changing to:

using CUDA
+plot(p1, p2, p3, layout = grid(3, 1))
Example block output

Making Use of GPU Parallelism

That was all using the CPU. How do we turn on GPU parallelism with DifferentialEquations.jl? Well, you don't. DifferentialEquations.jl "doesn't have GPU bits". So, wait... can we not do GPU parallelism? No, this is the glory of type-genericness, especially in broadcasted operations. To make things use the GPU, we simply use a CuArray from CUDA.jl. If instead of zeros(N,M) we used CuArray(zeros(N,M)), then the array lives on the GPU. CuArray naturally overrides broadcast such that dotted operations are performed on the GPU. DifferentialEquations.jl uses broadcast internally, and thus just by putting the array as a CuArray, the array-type will take over how all internal updates are performed and turn this algorithm into a fully GPU-parallelized algorithm that doesn't require copying to the CPU. Wasn't that simple?

From that you can probably also see how to multithread everything, or how to set everything up with distributed parallelism. You can make the ODE solvers do whatever you want by defining an array type where the broadcast does whatever special behavior you want.

So to recap, the entire difference from above is changing to:

using CUDA
 const gMx = CuArray(Float32.(Mx))
 const gMy = CuArray(Float32.(My))
 const gα₁ = CuArray(Float32.(α₁))
@@ -380,7 +380,7 @@
 end
g (generic function with 1 method)

Now we just define and solve the system of SDEs:

prob = SDEProblem(f, g, u0, (0.0, 100.0))
 @time sol = solve(prob, SRIW1());
retcode: Success
 Interpolation: 1st order linear
-t: 81670-element Vector{Float64}:
+t: 82065-element Vector{Float64}:
    0.0
    9.999999999999999e-5
    0.0002125
@@ -392,40 +392,40 @@
    0.0012526276111602785
    0.0015092060625553133
    ⋮
-  99.98283757746886
-  99.98441043518325
-  99.98617990011195
-  99.98817054815673
-  99.99041002720712
-  99.9929294411388
-  99.99576378181195
-  99.99895241506923
+  99.99767444251542
+  99.99788539576255
+  99.99812271816558
+  99.99838970586899
+  99.99869006703533
+  99.99902797334745
+  99.99940811794859
+  99.99983578062488
  100.0
-u: 81670-element Vector{Array{Float64, 3}}:
+u: 82065-element Vector{Array{Float64, 3}}:
  [0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0;;; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0;;; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0; … ; 0.0 0.0 … 0.0 0.0; 0.0 0.0 … 0.0 0.0]
- [4.999999999999999e-9 4.999999999999999e-9 … 9.996316537500258e-5 9.895557127461879e-5; 4.999999999999999e-9 4.999999999999999e-9 … 0.00010095991891832084 0.00010001786695802799; … ; 4.999999999999999e-9 4.999999999999999e-9 … 0.00010098125516900748 9.985954283146089e-5; 4.999999999999999e-9 4.999999999999999e-9 … 0.00010006697959276857 9.891598880860685e-5;;; 9.999999999999999e-5 9.999999999999999e-5 … 9.998508099839725e-5 9.997544172170106e-5; 9.999999999999999e-5 9.999999999999999e-5 … 9.998040675185174e-5 9.995422562917237e-5; … ; 9.999999999999999e-5 9.999999999999999e-5 … 0.00010004072537877081 0.00010005293389480278; 9.999999999999999e-5 9.999999999999999e-5 … 0.00010000901257908694 0.00010018103452400506;;; 9.999e-5 9.999e-5 … 0.00010007036637167426 9.99726086411411e-5; 9.999e-5 9.999e-5 … 9.984491999111786e-5 0.00010000969882915958; … ; 9.999e-5 9.999e-5 … 9.995231637281618e-5 0.00010006482751425028; 9.999e-5 9.999e-5 … 0.00010009036080664532 9.995584165289786e-5]
- [2.2341430571222796e-8 2.257447281403016e-8 … 0.00021228773726518117 0.00020798370386175517; 2.2571473816801087e-8 2.282786982163999e-8 … 0.00021717042615065443 0.00021245968457872102; … ; 2.2571081805367317e-8 2.2801389746267802e-8 … 0.000217227201732439 0.00021209397734642745; 2.2331304127404063e-8 2.25595811305304e-8 … 0.00021315567915251906 0.00020795738605897457;;; 0.00021249997907515068 0.00021250001772012549 … 0.00021251074257704858 0.0002125598502694325; 0.0002125000020095472 0.0002125000048785259 … 0.0002120697876550365 0.0002123871086053411; … ; 0.00021249997411579847 0.00021249997225982422 … 0.00021263686312119674 0.00021265233525586655; 0.00021249997022157463 0.00021250000250757118 … 0.00021249707233605945 0.000213039820287788;;; 0.00021245485010598533 0.00021245485396300661 … 0.00021282125073941618 0.0002125438625907802; 0.000212454849126795 0.00021245484448056186 … 0.00021223222200689689 0.00021252690867700825; … ; 0.00021245485314040936 0.00021245483335335783 … 0.00021215015235679366 0.0002126047186963575; 0.0002124548561384418 0.00021245484186531392 … 0.0002125019141398819 0.0002122610759514756]
- [5.626043154466438e-8 5.738302735048446e-8 … 0.00033909536355759584 0.0003284365727737196; 5.7448159636733606e-8 5.868498978177935e-8 … 0.00035074176606237256 0.0003393592508639709; … ; 5.74006059960896e-8 5.8564756555278474e-8 … 0.0003505836989910492 0.00033828514023566907; 5.631383509831211e-8 5.742641995041142e-8 … 0.0003396998669432268 0.00032777722773723886;;; 0.00033906240051638444 0.0003390625265528385 … 0.0003393496796999362 0.00033877423489570767; 0.00033906249760945787 0.0003390625492208266 … 0.0003383180631288406 0.0003390249471011938; … ; 0.0003390624719207299 0.00033906241862413124 … 0.0003392407304180433 0.00033994795983237445; 0.00033906250943773755 0.0003390625328913108 … 0.0003392631795345171 0.0003392885335627896;;; 0.00033894744979623236 0.000338947603296087 … 0.00033952942325889254 0.0003384330298899242; 0.00033894758524409806 0.00033894745679690513 … 0.0003385537555220067 0.0003393668736266786; … ; 0.0003389474993276571 0.00033894757026795534 … 0.0003384707422005556 0.000338468293529751; 0.00033894753676563523 0.0003389475648562937 … 0.00033920986572454726 0.00033899778105257164]
- [1.1225135074922173e-7 1.1559296785184639e-7 … 0.00048102154943786954 0.00045937338645831013; 1.1578844032022521e-7 1.193738680915563e-7 … 0.0005042889193993434 0.00048096641498339426; … ; 1.1568874024310677e-7 1.191648649451818e-7 … 0.0005038531661638163 0.0004799793631940536; 1.1259872530926271e-7 1.1577372725743263e-7 … 0.0004815878680253388 0.0004588665680417207;;; 0.00048144507524883315 0.00048144523267593673 … 0.0004824972472614454 0.0004811126276064454; 0.00048144527128336567 0.0004814454613573211 … 0.00048075038697833865 0.00048237786142219034; … ; 0.0004814451121047737 0.0004814451764322115 … 0.00048062725251270434 0.00048250370653237136; 0.0004814453166780955 0.0004814454756717085 … 0.0004819665712546553 0.0004811357474893174;;; 0.00048121346632678104 0.00048121338074236524 … 0.00048180792829394974 0.00048111871066105376; 0.00048121370392412366 0.0004812136000721056 … 0.0004806322101383575 0.00048053726775442026; … ; 0.000481213509889194 0.0004812134797608286 … 0.00048043049507087216 0.00048008676402341215; 0.00048121366549249285 0.0004812136301902226 … 0.0004815138992060525 0.00048156949952094424]
- [1.9743099666157477e-7 2.0514930199439282e-7 … 0.0006411975238073264 0.0006039225085037006; 2.054465355571475e-7 2.1390540793948128e-7 … 0.0006808537735002646 0.0006402372899986918; … ; 2.0540674034007046e-7 2.139421200931024e-7 … 0.0006811383018577721 0.0006408907503682987; 1.9806989531922662e-7 2.0538200549659342e-7 … 0.0006407737542034049 0.0006034474269441579;;; 0.0006416255871217625 0.0006416258284213718 … 0.0006423764092074581 0.0006417541185238921; 0.0006416258841318605 0.0006416257997615363 … 0.0006393541201287457 0.0006418520257099637; … ; 0.0006416255486488578 0.0006416254145495551 … 0.000640749641010793 0.0006426401401928532; 0.0006416261201760985 0.0006416260135253155 … 0.0006430983817549945 0.0006410059817692625;;; 0.0006412145242335606 0.0006412144322634356 … 0.0006408013159080384 0.0006411090716448495; 0.0006412147673935581 0.0006412147528280647 … 0.0006415515292381305 0.0006407356551311036; … ; 0.0006412143099449068 0.000641214344138083 … 0.0006397029192940781 0.0006406415220025991; 0.0006412143971163435 0.000641214091225042 … 0.000641541523525133 0.0006410622457272303]
- [3.1973016701369826e-7 3.36318640909115e-7 … 0.0008201431465781747 0.000762089034384867; 3.3662643833289894e-7 3.5402239430926946e-7 … 0.000884533275744549 0.000820045072603839; … ; 3.3644996930715915e-7 3.544009790489055e-7 … 0.0008849192832193356 0.0008206687218686358; 3.2076113071621193e-7 3.373161307742143e-7 … 0.0008197581096256342 0.000762035408814747;;; 0.0008218283672365214 0.0008218284658887978 … 0.0008232741338547533 0.0008218684878448381; 0.0008218289703787775 0.0008218284200340886 … 0.0008197692445619607 0.0008208047203223851; … ; 0.0008218284108678413 0.0008218287324020835 … 0.0008203080224101408 0.0008229595444036705; 0.0008218294487149609 0.0008218289994599111 … 0.0008211354345511454 0.0008211548931254631;;; 0.0008211546243022669 0.000821153912881781 … 0.0008225697804465709 0.0008197952288294865; 0.00082115488332997 0.0008211547340788269 … 0.0008214700690311991 0.0008209582857889921; … ; 0.0008211538357480292 0.0008211538494268866 … 0.0008205186642829694 0.0008211894105633469; 0.0008211541877714539 0.0008211542641144886 … 0.0008208811352884995 0.0008197628337192531]
- [4.917686601008236e-7 5.222616478924863e-7 … 0.001020231720824732 0.000931556185433739; 5.227195656671105e-7 5.555421519973026e-7 … 0.001121137583900297 0.001023225064284714; … ; 5.224453113811197e-7 5.564097575956078e-7 … 0.0011201244734662193 0.001021040515494499; 4.92766502664759e-7 5.228958040874829e-7 … 0.0010215480642118322 0.0009308045013193896;;; 0.001024558218017612 0.0010245567576037697 … 0.0010263992308332866 0.0010235866630488426; 0.0010245577312473053 0.0010245567453087891 … 0.0010232117827783651 0.0010241909615679714; … ; 0.0010245557846892662 0.0010245561428786662 … 0.0010220175819450055 0.001026445976836909; 0.0010245584629487066 0.0010245571324857526 … 0.0010213114060012556 0.0010234674042789722;;; 0.0010235097136730793 0.0010235078790510723 … 0.0010243980569096508 0.0010200725887688103; 0.0010235089277048843 0.0010235092898682376 … 0.0010231836811111414 0.0010231447933917936; … ; 0.0010235083953481746 0.001023509072484058 … 0.0010220551914364555 0.0010225253032023725; 0.0010235091837079138 0.0010235089418710174 … 0.0010246591360623474 0.0010222731269627341]
- [7.240530078503841e-7 7.804632179344636e-7 … 0.001246040243124299 0.0011177110626079366; 7.82150242051536e-7 8.3905029040236e-7 … 0.0013919960719870784 0.0012459736197672486; … ; 7.81328672754695e-7 8.431729332135892e-7 … 0.001393463260453747 0.0012458644733890061; 7.261335555660195e-7 7.801097192617937e-7 … 0.0012494757021112228 0.0011163323770332363;;; 0.001252627525521565 0.0012526273509545739 … 0.0012533398776742415 0.0012500701878737838; 0.0012526261501251093 0.001252627261292853 … 0.0012549154356124247 0.0012558690636071113; … ; 0.0012526249518478956 0.0012526240517618528 … 0.0012494304798307424 0.0012558143898237948; 0.0012526271592747196 0.0012526257442988938 … 0.0012502299555633577 0.0012531176217093125;;; 0.0012510602915027419 0.001251058901183097 … 0.0012512884354248946 0.0012475230645080534; 0.0012510581311770668 0.0012510602716214148 … 0.001253583793315309 0.001251243346714134; … ; 0.0012510585439931348 0.0012510603748136094 … 0.0012495668834646267 0.0012514501465208474; 0.001251061711300968 0.0012510609052924788 … 0.0012563811759896983 0.0012499721413527223]
- [1.0352990811901352e-6 1.1329077135235038e-6 … 0.001497308858540913 0.0013175620954026682; 1.135876111924879e-6 1.2355133025470448e-6 … 0.0017060288538511686 0.0015012317252172719; … ; 1.1345742345479431e-6 1.2382058920245104e-6 … 0.0017085936530296002 0.0014971349615113335; 1.04185108275658e-6 1.1326991171976344e-6 … 0.0015021232224829864 0.0013223286085772344;;; 0.001509204934424338 0.001509205033934461 … 0.0015095506127837284 0.0015037719142922984; 0.0015092042709155412 0.0015092040745397332 … 0.0015132502932711316 0.0015112348279266734; … ; 0.0015092027267505231 0.0015092045869849085 … 0.0015039552490115882 0.0015135166218096974; 0.0015092074247767214 0.0015092050420654594 … 0.0015052104858478334 0.0015077712690957355;;; 0.0015069316811931635 0.001506926789527071 … 0.0015056421011420645 0.001506697001210188; 0.0015069264047400846 0.001506929838864649 … 0.0015097730097021896 0.0015109202204542304; … ; 0.0015069307443498982 0.001506929519830393 … 0.0015045933802791108 0.0015056375664546074; 0.001506933273076694 0.001506931087339827 … 0.001513657043512686 0.0015048087486365146]
+ [4.999999999999999e-9 4.999999999999999e-9 … 9.985606380538476e-5 9.89785088295584e-5; 4.999999999999999e-9 4.999999999999999e-9 … 0.00010104368242204977 9.992492386922319e-5; … ; 4.999999999999999e-9 4.999999999999999e-9 … 0.00010089275028711275 9.988598021627924e-5; 4.999999999999999e-9 4.999999999999999e-9 … 9.99248224770009e-5 9.894209627443607e-5;;; 9.999999999999999e-5 9.999999999999999e-5 … 9.99319094575972e-5 9.999439015964657e-5; 9.999999999999999e-5 9.999999999999999e-5 … 9.997750414828171e-5 9.992547710401224e-5; … ; 9.999999999999999e-5 9.999999999999999e-5 … 0.00010001265600690508 0.00010003395664944076; 9.999999999999999e-5 9.999999999999999e-5 … 9.995815421602201e-5 0.0001000931657798362;;; 9.999e-5 9.999e-5 … 9.992640288082516e-5 9.999651856210098e-5; 9.999e-5 9.999e-5 … 9.996390700201499e-5 0.00010002873430914784; … ; 9.999e-5 9.999e-5 … 9.999111616991143e-5 0.00010009382690307789; 9.999e-5 9.999e-5 … 0.00010000582284580574 0.00010000884470387245]
+ [2.23328741866157e-8 2.2575916356897064e-8 … 0.0002124659869533662 0.00020809223591476686; 2.2555428675563965e-8 2.2825028911699737e-8 … 0.00021700691657590876 0.00021217583586757168; … ; 2.2553151761015648e-8 2.2803088306476972e-8 … 0.00021679526077062665 0.00021261240572683284; 2.2342838077494146e-8 2.256066375342087e-8 … 0.00021241633825092179 0.00020807118849036274;;; 0.0002124999980352286 0.0002124999949129772 … 0.0002123441644888038 0.00021245732592401964; 0.000212499990979344 0.00021250000548966728 … 0.0002122891662980347 0.00021237332058939552; … ; 0.00021249997450670916 0.0002124999979234862 … 0.00021245299871513624 0.0002126019015958212; 0.00021249998262482875 0.00021249998781491117 … 0.00021245206001991302 0.0002124252776625716;;; 0.0002124548575753772 0.00021245484717502856 … 0.00021231744251167526 0.0002123770296087958; 0.00021245482377256838 0.00021245484817934975 … 0.00021245293824687068 0.00021231382675883197; … ; 0.00021245483410055037 0.0002124548525416099 … 0.0002127032482078898 0.00021252934760125581; 0.00021245485831745657 0.00021245485428243792 … 0.00021245039367188013 0.00021248254671797087]
+ [5.6320573648598715e-8 5.750297536706503e-8 … 0.0003389553129086907 0.0003275827278533193; 5.746178898130615e-8 5.861290888508945e-8 … 0.00035076532460312135 0.0003387227576076753; … ; 5.7390860986855186e-8 5.853888014263755e-8 … 0.0003505977933157563 0.0003391881951275773; 5.643803978262597e-8 5.741383185669279e-8 … 0.0003386177612799003 0.0003277331610111669;;; 0.0003390625160112138 0.0003390623866083923 … 0.00033963294825128866 0.00033921242495929494; 0.00033906251047653057 0.00033906252826548306 … 0.000338594269199708 0.0003391103254781577; … ; 0.0003390624996629763 0.00033906245821445813 … 0.0003389694939828154 0.00033888136279589063; 0.0003390625119688104 0.0003390625523189165 … 0.00033929290646988547 0.00033866464172609937;;; 0.00033894761733397667 0.0003389475330886702 … 0.00033839206857742507 0.0003390146116374967; 0.00033894753698588954 0.00033894760810568097 … 0.0003392878989649192 0.00033852080933634705; … ; 0.0003389475958379555 0.0003389475367551459 … 0.00033900787728765246 0.0003390564497762368; 0.0003389475473135375 0.00033894757714981094 … 0.00033889274638243327 0.0003394096493226361]
+ [1.1248380900742633e-7 1.1574064504634187e-7 … 0.00048126722606024324 0.000459122940177962; 1.1588469231072962e-7 1.1929029417766356e-7 … 0.0005043845388468446 0.0004821431066906147; … ; 1.1569102016529593e-7 1.1921581585147142e-7 … 0.0005039148069360697 0.00048129886604625935; 1.1252676438102941e-7 1.1589350186170625e-7 … 0.0004801996089895221 0.0004593119496966663;;; 0.00048144526580839493 0.00048144503581478896 … 0.00048141701470744543 0.00048151238041692823; 0.000481445538345175 0.00048144536506677704 … 0.00048088705850824934 0.00048120572371132157; … ; 0.00048144503492443086 0.00048144512433884716 … 0.0004813534343917933 0.00048171243589996757; 0.0004814453010850488 0.0004814453316084206 … 0.00048122463365321083 0.0004805826810608375;;; 0.0004812137948492972 0.00048121354248699806 … 0.00048060907662766 0.000481571812828027; 0.00048121338611917225 0.00048121349777978584 … 0.0004818897050268114 0.0004808790686752324; … ; 0.0004812135882266259 0.0004812135710574945 … 0.00048080016556287036 0.0004815010288667679; 0.0004812134367257656 0.0004812136168076159 … 0.0004809291273700644 0.0004824641535439088]
+ [1.9777146709175162e-7 2.054206929607819e-7 … 0.0006412202041984875 0.0006036398797122343; 2.052706485947034e-7 2.1369528930457698e-7 … 0.0006804591507283688 0.0006414102351045131; … ; 2.0552171423897865e-7 2.136830505858551e-7 … 0.0006798932119429006 0.0006413511985580015; 1.9802659429543298e-7 2.0542495686129148e-7 … 0.0006389360946439278 0.0006040518110111532;;; 0.0006416260429074356 0.0006416256573280376 … 0.0006420756527699523 0.0006421447895751364; 0.0006416262141632883 0.0006416259377436523 … 0.0006415998985104483 0.0006409436482808643; … ; 0.0006416252745431659 0.0006416257743093767 … 0.0006411807071486415 0.0006414728153514642; 0.0006416258399649829 0.0006416260966401932 … 0.0006421314656888489 0.0006410580003488028;;; 0.0006412144383896006 0.000641214621025923 … 0.0006416549050246209 0.0006412004850411411; 0.0006412143861021362 0.0006412143836710299 … 0.0006409956976656638 0.0006403080991687155; … ; 0.0006412147430924182 0.0006412145795928175 … 0.0006411337638103483 0.0006408186308370179; 0.0006412142704606569 0.0006412145402160666 … 0.0006414601387047393 0.0006416860118830399]
+ [3.2100118262626766e-7 3.366066672115075e-7 … 0.0008193582083770284 0.0007603416100467402; 3.373846117474054e-7 3.539206603493776e-7 … 0.0008845528310767654 0.0008213037182579261; … ; 3.371251881463889e-7 3.5430120000288367e-7 … 0.0008824075684262037 0.0008217406897373382; 3.2096820909795635e-7 3.3666311101763746e-7 … 0.0008183084894391631 0.0007606951249939983;;; 0.0008218294924200721 0.0008218286168271587 … 0.0008224292616909959 0.0008232134931981074; 0.0008218294884269173 0.0008218290800413203 … 0.0008224757466676367 0.0008218430867373654; … ; 0.0008218286331119705 0.0008218290531243074 … 0.000823063792208635 0.0008211070194662142; 0.0008218286664104898 0.0008218292526544982 … 0.0008219011591793781 0.0008227870412695065;;; 0.0008211543030481589 0.0008211540894077733 … 0.0008227665571045043 0.000822006794967427; 0.0008211545211471827 0.0008211533609632802 … 0.0008207786646735432 0.0008197815364887426; … ; 0.0008211536555664258 0.0008211550045678031 … 0.0008202437216958783 0.0008215117269486186; 0.0008211545364322259 0.0008211550020522573 … 0.000821384960198112 0.0008224295237429157]
+ [4.928902526322651e-7 5.236841160434091e-7 … 0.0010214774703457318 0.0009320102259972808; 5.246898743663464e-7 5.55646852156825e-7 … 0.0011169574923569972 0.0010206348748889743; … ; 5.233369306977913e-7 5.568172750390468e-7 … 0.0011158843943450836 0.0010222960811210123; 4.930546707906812e-7 5.237234402087273e-7 … 0.0010185933920679824 0.0009306423230638478;;; 0.0010245594047399937 0.0010245572836258799 … 0.0010253884638132189 0.0010268110263241478; 0.0010245582332012522 0.0010245585722018155 … 0.0010258572800433848 0.0010227896498748946; … ; 0.0010245569481022593 0.001024557121880903 … 0.0010253799803442802 0.0010223851859976715; 0.0010245575223219964 0.0010245578680526067 … 0.0010256458338745967 0.0010263668671400361;;; 0.0010235096272399494 0.0010235084740253038 … 0.0010254883561853778 0.0010260291032416597; 0.0010235082896903015 0.0010235078071052551 … 0.0010223733356705315 0.001020671079231815; … ; 0.0010235084631538443 0.0010235093069095717 … 0.0010240186116849735 0.001025301885904364; 0.0010235097467181686 0.001023508791970006 … 0.0010256577259433386 0.0010248676193469247]
+ [7.259855443954394e-7 7.809506326853916e-7 … 0.0012454639768548095 0.0011183482064507759; 7.821679575512494e-7 8.406740658012621e-7 … 0.0013889919189014666 0.0012415225867839722; … ; 7.80607991319131e-7 8.428868048666042e-7 … 0.0013884804459984364 0.001243991037841012; 7.263025701743557e-7 7.805694918663041e-7 … 0.0012441298212133396 0.0011181578390130698;;; 0.0012526289187726383 0.001252627830944839 … 0.0012547096355593978 0.0012570667242554291; 0.001252628612604141 0.0012526284241362283 … 0.0012539670647376316 0.0012486649274872212; … ; 0.0012526264929882023 0.001252625601464593 … 0.0012537243308277347 0.001251131053227742; 0.0012526274863657793 0.0012526279335736822 … 0.0012578652620026881 0.001254013946599161;;; 0.001251060727984884 0.0012510612496033297 … 0.0012526830186759782 0.0012530724762314638; 0.001251059111033589 0.0012510592669081624 … 0.0012494610931021101 0.0012494145249762134; … ; 0.0012510601777217965 0.0012510616600973628 … 0.001250738724029174 0.001254921902639397; 0.0012510605871079803 0.0012510593294675113 … 0.0012500471662000392 0.00125296323648572]
+ [1.0410851895165245e-6 1.1300995371826151e-6 … 0.0014948922276066686 0.0013183846322513445; 1.1344739908549457e-6 1.2374900535770287e-6 … 0.0017031827246438986 0.0014917329143985098; … ; 1.1264997575382494e-6 1.2376085336866258e-6 … 0.001704980794021033 0.0014966971058111673; 1.0384991684861093e-6 1.1292751204073296e-6 … 0.0014937503002488697 0.0013196244128146387;;; 0.001509207490135736 0.0015092058431940805 … 0.0015078199164548776 0.0015128867743091849; 0.0015092066078322536 0.0015092076129765786 … 0.001508277417841423 0.001503912293062574; … ; 0.0015092034362294965 0.001509206274929207 … 0.0015074656970147742 0.0015109372067831544; 0.0015092059487641645 0.0015092084297761163 … 0.0015131123167917576 0.001507580594157866;;; 0.0015069332055410879 0.0015069328487965264 … 0.001505787167412524 0.0015087608707833732; 0.0015069306849254806 0.0015069300401536706 … 0.001501846023063347 0.0015060781570021403; … ; 0.0015069313374144258 0.0015069340250143276 … 0.0015049772024092161 0.0015115411705687296; 0.0015069291408320122 0.001506928777770267 … 0.0015012134367047493 0.0015067355058622537]
  ⋮
- [0.09095411023354408 0.18073368759568964 … 0.5433985438256818 0.2749853607857393; 0.17996664711307828 0.3618246298905481 … 1.0736444650386152 0.5405332282244737; … ; 0.18335466192478964 0.3681437455872525 … 1.0606959383532075 0.5266238495551709; 0.09294317370253823 0.186705790049562 … 0.5323073140559054 0.2672741484522486;;; 1.4528167461382795 1.3645639592884635 … 1.5009603144439962 1.2448919767996092; 1.287264872900764 1.4234940232820317 … 1.592867180170738 1.26989590801143; … ; 1.3783465195820213 1.4443492584096838 … 0.991913595818825 1.0755207157970135; 1.4848783878640988 1.4735269218327074 … 1.105081452438355 1.1329915236876058;;; 0.5909246250190522 0.7345753841171144 … 1.0960277844677402 0.6184950979143579; 0.60724493211109 0.7512006674152759 … 1.7962379070995884 0.8088384498662519; … ; 0.6357739710491221 0.784045868633366 … 0.8823538017833243 1.0754424617553644; 0.5775811592949992 0.5897427784589999 … 0.9782334043026564 0.6692023380019949]
- [0.0909686932157408 0.18069446668188865 … 0.5417617660812728 0.27419203921264446; 0.17995192211245567 0.36135201473031675 … 1.0775928672546293 0.5425763487250082; … ; 0.1845118311842056 0.36826649363096947 … 1.0674865366823763 0.5310402673920481; 0.09286337635883045 0.18653340915054253 … 0.5317977984569924 0.2682666257527216;;; 1.4532250326724026 1.3655555224088616 … 1.499105967193163 1.2444045024129906; 1.2871330771055505 1.4208422720562348 … 1.5832651729618292 1.2733680702303813; … ; 1.3784500550390877 1.4437583255218296 … 0.9917322636899936 1.0740125536609268; 1.4842304804060305 1.473487850627034 … 1.1052025106893784 1.1325586522894762;;; 0.5915275927276417 0.7344296695584942 … 1.0942588308457146 0.6187195209558951; 0.6072609503794257 0.7511585499426237 … 1.7978511739319538 0.8105197623997598; … ; 0.634304037159775 0.7832516253728063 … 0.8888084942551522 1.0730539875472989; 0.5770646874525776 0.5906277518745324 … 0.9795756976428373 0.6686088492880367]
- [0.09101756047573457 0.18102058332581372 … 0.542088124727038 0.2738172067787298; 0.17930989687908844 0.35950731557224425 … 1.070814482352316 0.5404219767585827; … ; 0.18428397833328589 0.36762424495455975 … 1.0682878206171698 0.5340429264278261; 0.09276693079382944 0.18577371716535598 … 0.5325164074870662 0.2685566330994594;;; 1.4531174705762269 1.3656939222091755 … 1.4976734359488777 1.243448458065818; 1.2870137575226344 1.420224004796199 … 1.5767985428067328 1.2686941328626957; … ; 1.3785767952164734 1.4443268461907812 … 0.9920064336724933 1.0792931696111883; 1.4844298669734572 1.4731895311058483 … 1.1074957242875003 1.1325722610656186;;; 0.5918674184921642 0.7330388792430105 … 1.0939538815008434 0.6184914634311238; 0.6065633632824808 0.7515654605219255 … 1.7924207131101162 0.8051124953385974; … ; 0.634604386850437 0.7811890145392605 … 0.8863106585850027 1.076109791801104; 0.5775749710299845 0.59099016646512 … 0.9818910878171854 0.6682060733334549]
- [0.09149721124217665 0.17969667899204914 … 0.5440581323601165 0.273146467561263; 0.18007004760373765 0.35872751117372387 … 1.074207884189963 0.5408301151153119; … ; 0.18546587851347732 0.36933194267843716 … 1.0689857180346085 0.535664908974506; 0.09282695289089189 0.1862406939416128 … 0.5338403733123852 0.2684580025377815;;; 1.4532903998420978 1.365936217500355 … 1.4954749009397657 1.2423049074611485; 1.2877036928791028 1.4200080803995732 … 1.5740006581253925 1.2706069102709774; … ; 1.3793548379521083 1.443938126342299 … 0.9968846413396514 1.0763495661667741; 1.4843115807335419 1.4734010943263298 … 1.1038574328951878 1.132308894439086;;; 0.5916419230470871 0.7311198829359915 … 1.095443625009697 0.6172594519977604; 0.6059859616449678 0.7525935652224851 … 1.7853188045157542 0.8046676662239477; … ; 0.63460370035745 0.7807376106671726 … 0.8875426479148713 1.0680369622906416; 0.5776370567827669 0.5903564449384013 … 0.9801079054840959 0.6680627299859772]
- [0.09128304215297972 0.17923631560856207 … 0.5402764559292867 0.2715244792521834; 0.17949952035427685 0.35586942576243014 … 1.0614721739667636 0.5401983945521921; … ; 0.18529248482140154 0.37018594642885794 … 1.0701257875950168 0.5341513841922286; 0.09345422442054493 0.18596002348371132 … 0.533756492060297 0.26817577687028155;;; 1.4534556535726337 1.3659166676168837 … 1.4974073398105652 1.2410000923622608; 1.2873914989696722 1.4184301921193745 … 1.5702908914131997 1.269989208070207; … ; 1.380283805999164 1.4445452349122867 … 1.000521993720594 1.0761246883518187; 1.4849377891585556 1.4733903501104981 … 1.1083802748512717 1.1325335874879374;;; 0.5920371573321748 0.7301537851439536 … 1.0973983734747719 0.6171277194124605; 0.607283784036059 0.7523440266616072 … 1.7856307154573248 0.807412184851573; … ; 0.6352734392358648 0.7828587411283375 … 0.8886637793435997 1.0593342389973721; 0.5781443425853834 0.5901962139823435 … 0.9819637683770973 0.6688807807084965]
- [0.09060821131726161 0.17978449112916198 … 0.5389805817159925 0.27302941981979806; 0.17860878635688884 0.3556219180105047 … 1.0604074917206074 0.542056733781131; … ; 0.18534519410026717 0.3696402283047962 … 1.0667160348347344 0.5343048781856167; 0.0939087082229901 0.18629320337489486 … 0.5334706265870327 0.2687730644944258;;; 1.4533842570364348 1.3653877209323524 … 1.4961227774414931 1.2399522904135671; 1.287620923036903 1.4193813279021021 … 1.5753596470449391 1.2706536863734677; … ; 1.3795380268265682 1.4423521753267528 … 0.9980179677445676 1.0786152978114751; 1.484277638539527 1.4719328640075395 … 1.1083009872259022 1.1341461635410761;;; 0.5921220946278042 0.7307483829906724 … 1.094373841288203 0.6186907299036514; 0.607852611481658 0.7524297507600238 … 1.7867587670040015 0.8088354813213183; … ; 0.6333862249882062 0.783071505862178 … 0.8945792908034496 1.0565179424665088; 0.5791935688771453 0.5903475694919273 … 0.9809879269516795 0.6692450177600525]
- [0.09063074027694036 0.17973134177324004 … 0.5413053918848315 0.27267562219258634; 0.1803307656667396 0.3561564180295167 … 1.064256526357397 0.5406460670394471; … ; 0.18638082675223788 0.36956714185490225 … 1.0641938605578312 0.5338407237543249; 0.0934616653047752 0.1864059102575423 … 0.536330785339371 0.2698548248715441;;; 1.453586452022282 1.3650852945837992 … 1.498159448784648 1.2412559736507482; 1.2883831126905345 1.419136252465309 … 1.5733269125963847 1.2706675643417609; … ; 1.3774247089597502 1.4466021950519308 … 1.0003606273242585 1.0727168054771894; 1.4842348186499836 1.4725151086685349 … 1.1108954380985927 1.1329120143923266;;; 0.5927024315290926 0.7308455084360383 … 1.0937691973386097 0.617802109422098; 0.6075093458958136 0.7489046880008572 … 1.7865163315760217 0.8108535253672958; … ; 0.6347831306876377 0.7821247249537027 … 0.8923666174469665 1.0531555055331574; 0.5790694513194155 0.5901108092232161 … 0.9799698516440524 0.670046782738477]
- [0.0905173530809036 0.18112342381784466 … 0.5395065343221155 0.2711122551047152; 0.18113451284867396 0.35545004583999457 … 1.06388317654188 0.5366834529312523; … ; 0.18638487823670863 0.36942780884542986 … 1.0700920634484048 0.5373129584546558; 0.09322643469588845 0.18653183539075746 … 0.5404244529292889 0.271116545575328;;; 1.453185771731066 1.36691539402099 … 1.496673753337769 1.2423645114198307; 1.287682807662386 1.4190760258824602 … 1.5748303766457497 1.2671718076594316; … ; 1.37918071339081 1.4447312615298844 … 0.993853798641061 1.0730864390699104; 1.4833315403647933 1.4701741425087658 … 1.1160467780807513 1.1335071860826271;;; 0.5932428883898664 0.7295774118430483 … 1.0890172920136334 0.6200237140781364; 0.6058133131291972 0.7501069060153126 … 1.7771395145876374 0.8095089426104722; … ; 0.6347324241856038 0.7787198500031186 … 0.8962279747768082 1.0493927514579136; 0.5792977128483533 0.5901078182493888 … 0.9789384512923862 0.6681553565974637]
- [0.0908348631118142 0.17972129529514427 … 0.5374790047258095 0.27128869884285034; 0.18044637443117567 0.35701623134979577 … 1.0660995956663213 0.5370648216405647; … ; 0.18596845603286305 0.36861170429064194 … 1.0695409593069785 0.5373366496440398; 0.09355009836749328 0.18608372553136893 … 0.5404474871161584 0.2715654157327684;;; 1.453361397957258 1.366558219803113 … 1.4953192096575796 1.2428307754261636; 1.2872251733777713 1.4208993132203118 … 1.573936550013771 1.268828443980049; … ; 1.3785515745980443 1.4455693640257428 … 0.9920132820329296 1.0740566880568851; 1.4832635936929328 1.4698990215522427 … 1.1168237053622807 1.1343288774584601;;; 0.5932068971602933 0.7282979822585267 … 1.0872427967251814 0.6205181362309911; 0.6049817472421246 0.7507542656695455 … 1.779161045889365 0.8065877258621807; … ; 0.6342515442886443 0.7791341575420562 … 0.9002618922593746 1.0462699671269888; 0.5791247385749666 0.5913569247411082 … 0.9777053932632851 0.6667504983411452]
using Plots;
+ [0.04936757763627451 0.23266587726889762 … 0.6955842205749666 0.11802059240758436; 0.2171032764533455 0.22839086116106433 … 0.616349255689188 0.659711868734099; … ; 0.1839474278985819 0.425594976753274 … 0.34071440555634097 1.0666647998120793; 0.09965414025578008 0.17297074052461078 … 0.9041225505564564 0.04486623047731094;;; 1.445465212450279 1.386463207006722 … 0.8378171477225499 1.4365763530794553; 1.296326394256788 1.3326189915298092 … 1.1229208097224839 1.1614287294786843; … ; 1.4186698734197087 1.6331043922860897 … 1.2216039101427882 1.25132699743229; 1.444898251218819 1.2856471343036429 … 1.2872649021796563 1.564025479404412;;; 0.5013844034586422 0.6347044332628422 … 0.943053270797234 0.5250673950019829; 0.6443363598385258 0.6264255399010453 … 0.4857697096957459 0.6797689313844539; … ; 0.6544736494083704 0.5127836410869208 … 1.171020884756757 0.6962390796383141; 0.6092266697131014 0.5847324153607868 … 0.8478127480281219 0.6654110141065476]
+ [0.054368506962519735 0.22270823220062178 … 0.664663052585209 0.13542172691915771; 0.20903934635919572 0.2459555869526689 … 0.6724751717974085 0.6320496011793023; … ; 0.1854836353610449 0.42001464680029027 … 0.458132513040156 0.9922761401762071; 0.09893490970042643 0.17609099905246792 … 0.8497224989979024 0.07985415713687168;;; 1.445495428348024 1.3866800034123055 … 0.8379573482540912 1.4366427601286271; 1.2964195791175235 1.3325669366458572 … 1.1228214945011834 1.1619302038483479; … ; 1.4185692948733233 1.6327352228636511 … 1.2215465074783014 1.2516639009573916; 1.4448894900508884 1.2854370318305297 … 1.2872847896029476 1.5640004785353867;;; 0.5013850848598458 0.634775928499963 … 0.9431208226549456 0.5251084788882071; 0.6443726328930179 0.6265836020943872 … 0.4858796526646913 0.6799080097961808; … ; 0.6544551143160909 0.5131439074793444 … 1.171037960525954 0.6968881575769118; 0.6092028529312917 0.5847015365803809 … 0.8479451511662699 0.6653440204997242]
+ [0.05906890224309878 0.21308954817490666 … 0.6359363309225959 0.1520715057561393; 0.20131955961833625 0.26246036892203517 … 0.726390104418558 0.60681171017007; … ; 0.1868511007745645 0.4149168641498036 … 0.569060758545492 0.9205639566049569; 0.09830789083806465 0.1789988562864745 … 0.7973834168368417 0.11298290238427548;;; 1.4454712184904859 1.386268743070451 … 0.8375945017647392 1.436340287505737; 1.2963671627675712 1.332639821309466 … 1.1232666068881598 1.160930070799613; … ; 1.4186320705703808 1.632838101608601 … 1.2217338064280279 1.251363148059953; 1.4449453359030415 1.2855620657425444 … 1.28693752059329 1.5640267745740128;;; 0.501378337528065 0.634515513293739 … 0.9429723104444032 0.5252372942816662; 0.6442287380625306 0.6262504514487814 … 0.4866472922293828 0.6809642093629991; … ; 0.6545512176002268 0.513715730523481 … 1.1704453908727237 0.6969545083229562; 0.609381051912636 0.5849661454942741 … 0.8487547468772078 0.6653350551406275]
+ [0.0633398950764975 0.20458477228744823 … 0.6104013241616513 0.16794638230189232; 0.19445221223702833 0.27719528772707963 … 0.7749170337703967 0.5833452827696398; … ; 0.18824065724906525 0.4097446164908087 … 0.6713910910266604 0.8544127672528096; 0.09760900269949713 0.1815493448576134 … 0.7522435136507305 0.14370439762735007;;; 1.445490705405726 1.3862236131204813 … 0.838767494715516 1.4364606321391273; 1.2960763601563698 1.3328593604331322 … 1.122408931237781 1.1616836153368562; … ; 1.4186193245980259 1.632279736717917 … 1.222711741723427 1.2514319109905718; 1.444914470171011 1.2858309578600513 … 1.2874709038025056 1.5640807989961016;;; 0.5015067190315834 0.6347176629614187 … 0.9428202775797869 0.5250678478729391; 0.6443653715987201 0.6262664865180607 … 0.4868305350862123 0.680232381191995; … ; 0.6543318808782985 0.5134922735193885 … 1.171037790073535 0.6975202458771683; 0.6091454750184669 0.5846922061638232 … 0.8491268270868971 0.665345098622704]
+ [0.06750557761273032 0.19693246779937446 … 0.5866427600504145 0.18185839187766747; 0.18862492831896427 0.29042510419370343 … 0.8194553274515467 0.5633280817137128; … ; 0.18933457284414354 0.4058184985221624 … 0.7618842471054537 0.7964562315327953; 0.09746016639287626 0.1840160387822561 … 0.7093009873545759 0.17125316155262563;;; 1.4454066076337022 1.3866670154311538 … 0.8376230358651287 1.4361030193399642; 1.2962932617383074 1.332898913875757 … 1.1243398781773661 1.1602117167276353; … ; 1.4184381429540465 1.631899817266112 … 1.2230279631371925 1.2518364824057788; 1.444823817629162 1.2855230001518185 … 1.2884272184033732 1.5640170040452597;;; 0.5013600288394156 0.6347037089917055 … 0.9426736945398153 0.5252070453498285; 0.6443099743919897 0.626505611437884 … 0.48677189965561907 0.6801841625064672; … ; 0.6545517355843076 0.5136063895205776 … 1.1699518784012068 0.6946040642830477; 0.6090689709403756 0.5850322457496201 … 0.8497157368066706 0.6654592382508349]
+ [0.0708658450090456 0.19011577033166263 … 0.5655431572424164 0.1947728267801722; 0.1831081797214739 0.30122981347182004 … 0.8596866057865217 0.5431040702890506; … ; 0.1902292483404371 0.40042124060060896 … 0.8435057218941675 0.7413376858048049; 0.09700544125538291 0.186434567676755 … 0.6727673648965036 0.1959933416379321;;; 1.445408951587974 1.3864697279728235 … 0.8384608057075106 1.4364121272373773; 1.296441187231234 1.3325802695221385 … 1.1218272572179326 1.1611771749845896; … ; 1.4186664713649546 1.6318393662514883 … 1.2215996157669018 1.2502697824926388; 1.4449338688368836 1.2855397147266805 … 1.2867950833303696 1.5637537549001814;;; 0.5015100197569827 0.6348217739637622 … 0.9420643725476003 0.5253451626479823; 0.6444173692150013 0.6265992041674117 … 0.4866728198949933 0.6801140029674406; … ; 0.6543460674285045 0.5134819342654507 … 1.1712879767276019 0.6971775507617535; 0.6091631270266883 0.5849158380397651 … 0.8485621944921579 0.6652979121939284]
+ [0.07390859193311859 0.18442982074422562 … 0.5475735260196121 0.20632307203951042; 0.17872426935327312 0.3104099820607644 … 0.893554598185358 0.5283964840913167; … ; 0.1906708585690636 0.3965245654569959 … 0.9111491468139413 0.6940740218388275; 0.09685086791373251 0.18831070996317636 … 0.6412041432405285 0.2165615454480941;;; 1.4452487564582666 1.3862624917724886 … 0.839026916544648 1.4362750877103225; 1.2963427216462369 1.3325484317926914 … 1.1207247708684762 1.1623272930569408; … ; 1.418813045518773 1.6316745643376573 … 1.2212493488317013 1.2503180177636974; 1.4448633366500996 1.2858305733830044 … 1.2867522519208807 1.5634797203768698;;; 0.50159225013395 0.6349474582927791 … 0.9422800618860873 0.5252789721482122; 0.6444677061732001 0.6260494260238151 … 0.4850339787670855 0.6809985964435782; … ; 0.6541917814720214 0.5146002860193426 … 1.1716578864472942 0.6981742700199417; 0.6090821298652084 0.58482976321335 … 0.8483858302700443 0.6651257729227334]
+ [0.07660092022798318 0.1794899332887928 … 0.5316223061141693 0.21531505358413136; 0.17549525165812446 0.31826485742708116 … 0.9197331561454327 0.5160718322921398; … ; 0.1912125424397472 0.3929181835799982 … 0.9675484539404692 0.6561899958674818; 0.09684817285675801 0.18958605654802824 … 0.6174122895414892 0.23394461202498137;;; 1.445325426828273 1.3864917853563714 … 0.8398191397413842 1.4363212401680492; 1.2965531908772383 1.3328175260609314 … 1.1193721231417284 1.1614770485958887; … ; 1.4184569129363878 1.6320997071255365 … 1.2196944127985443 1.2504902858469693; 1.4449260181216415 1.2859182468448354 … 1.2874130720827464 1.5632474260281293;;; 0.5018299783850175 0.6348864882013897 … 0.9416862960947767 0.5252301309170393; 0.6443522363583161 0.6269568449219476 … 0.4856407026332664 0.6801350021505514; … ; 0.6548698435453106 0.5139910159649437 … 1.1715046401148923 0.6977089949388138; 0.6087628389972707 0.5847403192040908 … 0.8499204001371019 0.6656434382633799]
+ [0.07734042236771868 0.1778160197846436 … 0.5268481260163858 0.2184666610959897; 0.17463720367552327 0.32037460783079713 … 0.9281432250786394 0.5133970469818814; … ; 0.19147250368462182 0.392413624985508 … 0.9850509272736807 0.6432068519474117; 0.09694400626539933 0.1898780870533635 … 0.6102153202207176 0.23891719364410113;;; 1.4453736980455487 1.3864459193084147 … 0.8406954673179369 1.436107465916368; 1.2964506543349643 1.3330079455606876 … 1.1181556500009762 1.1612831928907215; … ; 1.418117883696646 1.6316293789845058 … 1.219894150481218 1.250804932097243; 1.4450267552901257 1.285734568233742 … 1.287455087680251 1.5635443631832149;;; 0.5019317164998374 0.6344200300042473 … 0.940317093133458 0.5250780316711634; 0.6443592543890294 0.6263960514238355 … 0.48592335648080404 0.6804439656453433; … ; 0.65477229792902 0.5141027011088776 … 1.1733313209053287 0.6970185973566286; 0.6087105608924873 0.58508523092305 … 0.8489834357509181 0.6653140124374793]
using Plots;
 gr();
 
 # Use `Array` to transform the result back into a CPU-based `Array` for plotting
 p1 = surface(X, Y, Array(sol[end][:, :, 1]), title = "[A]")
 p2 = surface(X, Y, Array(sol[end][:, :, 2]), title = "[B]")
 p3 = surface(X, Y, Array(sol[end][:, :, 3]), title = "[C]")
-plot(p1, p2, p3, layout = grid(3, 1))
Example block output

We can see the cool effect that diffusion dampens the noise in [A] but is unable to dampen the noise in [B] which results in a very noisy [C]. The stiff SPDE takes much longer to solve even using high order plus adaptivity because stochastic problems are just that much more difficult (current research topic is to make new algorithms for this!). It gets GPU'd just by using CuArray like before. But there we go: solving systems of stochastic PDEs using high order adaptive algorithms with within-method GPU parallelism. That's gotta be a first? The cool thing is that nobody ever had to implement the GPU-parallelism either, it just exists by virtue of the Julia type system.

(Note: We can also use one of the SROCK methods for better performance here, but they will require a choice of dt. This is left to the reader to try.)

Note

This can take a while to solve! An explicit Runge-Kutta algorithm isn't necessarily great here, though to use a stiff solver on a problem of this size requires once again smartly choosing sparse linear solvers. The high order adaptive method is pretty much necessary though, since something like Euler-Maruyama is simply not stable enough to solve this at a reasonable dt. Also, the current algorithms are not so great at handling this problem. Good thing there's a publication coming along with some new stuff...

+plot(p1, p2, p3, layout = grid(3, 1))Example block output

We can see the cool effect that diffusion dampens the noise in [A] but is unable to dampen the noise in [B] which results in a very noisy [C]. The stiff SPDE takes much longer to solve even using high order plus adaptivity because stochastic problems are just that much more difficult (current research topic is to make new algorithms for this!). It gets GPU'd just by using CuArray like before. But there we go: solving systems of stochastic PDEs using high order adaptive algorithms with within-method GPU parallelism. That's gotta be a first? The cool thing is that nobody ever had to implement the GPU-parallelism either, it just exists by virtue of the Julia type system.

(Note: We can also use one of the SROCK methods for better performance here, but they will require a choice of dt. This is left to the reader to try.)

Note

This can take a while to solve! An explicit Runge-Kutta algorithm isn't necessarily great here, though to use a stiff solver on a problem of this size requires once again smartly choosing sparse linear solvers. The high order adaptive method is pretty much necessary though, since something like Euler-Maruyama is simply not stable enough to solve this at a reasonable dt. Also, the current algorithms are not so great at handling this problem. Good thing there's a publication coming along with some new stuff...

diff --git a/dev/showcase/massively_parallel_gpu/index.html b/dev/showcase/massively_parallel_gpu/index.html index cdc1de5d4a0..ac46fef4969 100644 --- a/dev/showcase/massively_parallel_gpu/index.html +++ b/dev/showcase/massively_parallel_gpu/index.html @@ -23,4 +23,4 @@ sol = solve(monteprob, Tsit5(), EnsembleThreads(), trajectories = 10_000, saveat = 1.0f0)
EnsembleSolution Solution of length 10000 with uType:
 SciMLBase.ODESolution{Float32, 2, Vector{StaticArraysCore.SVector{3, Float32}}, Nothing, Nothing, Vector{Float32}, Vector{Vector{StaticArraysCore.SVector{3, Float32}}}, SciMLBase.ODEProblem{StaticArraysCore.SVector{3, Float32}, Tuple{Float32, Float32}, false, StaticArraysCore.SVector{3, Float32}, SciMLBase.ODEFunction{false, SciMLBase.AutoSpecialize, typeof(Main.lorenz), LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, OrdinaryDiffEq.Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.InterpolationData{SciMLBase.ODEFunction{false, SciMLBase.AutoSpecialize, typeof(Main.lorenz), LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Vector{StaticArraysCore.SVector{3, Float32}}, Vector{Float32}, Vector{Vector{StaticArraysCore.SVector{3, Float32}}}, OrdinaryDiffEq.Tsit5ConstantCache}, DiffEqBase.Stats, Nothing}

Taking the Ensemble to the GPU

Now uhh, we just change EnsembleThreads() to EnsembleGPUArray()

sol = solve(monteprob, Tsit5(), EnsembleGPUArray(CUDA.CUDABackend()), trajectories = 10_000, saveat = 1.0f0)
EnsembleSolution Solution of length 10000 with uType:
 SciMLBase.ODESolution{Float32, 2, uType, Nothing, Nothing, Vector{Float32}, rateType, SciMLBase.ODEProblem{StaticArraysCore.SVector{3, Float32}, Tuple{Float32, Float32}, false, StaticArraysCore.SVector{3, Float32}, SciMLBase.ODEFunction{false, SciMLBase.AutoSpecialize, typeof(Main.lorenz), LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, OrdinaryDiffEq.Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, IType, DiffEqBase.Stats, Nothing} where {uType, rateType, IType}

Or for a more efficient version, EnsembleGPUKernel(). But that requires special solvers, so we also change to GPUTsit5().

sol = solve(monteprob, GPUTsit5(), EnsembleGPUKernel(CUDA.CUDABackend()), trajectories = 10_000)
EnsembleSolution Solution of length 10000 with uType:
-SciMLBase.ODESolution{Float32, 2, uType, Nothing, Nothing, tType, Nothing, P, A, IType, Nothing, Nothing} where {uType, tType, P, A, IType}

Okay, so that was anticlimactic, but that's the point: if it were harder than that, it wouldn't be automatic! Now go check out DiffEqGPU.jl's documentation for more details, that's the end of our show.

+SciMLBase.ODESolution{Float32, 2, uType, Nothing, Nothing, tType, Nothing, P, A, IType, Nothing, Nothing} where {uType, tType, P, A, IType}

Okay, so that was anticlimactic, but that's the point: if it were harder than that, it wouldn't be automatic! Now go check out DiffEqGPU.jl's documentation for more details, that's the end of our show.

diff --git a/dev/showcase/missing_physics/534623c5.svg b/dev/showcase/missing_physics/2c4c4ce0.svg similarity index 84% rename from dev/showcase/missing_physics/534623c5.svg rename to dev/showcase/missing_physics/2c4c4ce0.svg index 59f577e7310..b96da7f81de 100644 --- a/dev/showcase/missing_physics/534623c5.svg +++ b/dev/showcase/missing_physics/2c4c4ce0.svg @@ -1,52 +1,52 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/missing_physics/2b5bbf8a.svg b/dev/showcase/missing_physics/489ac7be.svg similarity index 97% rename from dev/showcase/missing_physics/2b5bbf8a.svg rename to dev/showcase/missing_physics/489ac7be.svg index c7f4a555895..b32d22d2bed 100644 --- a/dev/showcase/missing_physics/2b5bbf8a.svg +++ b/dev/showcase/missing_physics/489ac7be.svg @@ -1,36 +1,36 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/missing_physics/5fe4f215.svg b/dev/showcase/missing_physics/86c321bc.svg similarity index 85% rename from dev/showcase/missing_physics/5fe4f215.svg rename to dev/showcase/missing_physics/86c321bc.svg index 53fe9cc41a1..6667f1c7be0 100644 --- a/dev/showcase/missing_physics/5fe4f215.svg +++ b/dev/showcase/missing_physics/86c321bc.svg @@ -1,213 +1,213 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + Timeseries of UODE Error - - - - - - + + + + + + x(t) - - + + y(t) - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Neural Network Fit of U2(t) - - - - - - + + + + + + Neural Network - - + + True Missing Term - + - + - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + Extrapolated Fit From Short Training Data - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + x data - - + + y data - - + + True x(t) - - + + True y(t) - - + + Estimated x(t) - - + + Estimated y(t) - + Training - + Data diff --git a/dev/showcase/missing_physics/8ebb7bff.svg b/dev/showcase/missing_physics/a2c2395e.svg similarity index 88% rename from dev/showcase/missing_physics/8ebb7bff.svg rename to dev/showcase/missing_physics/a2c2395e.svg index 498531737a7..64773ccb5e6 100644 --- a/dev/showcase/missing_physics/8ebb7bff.svg +++ b/dev/showcase/missing_physics/a2c2395e.svg @@ -1,48 +1,48 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/missing_physics/c97f6698.svg b/dev/showcase/missing_physics/a583b620.svg similarity index 80% rename from dev/showcase/missing_physics/c97f6698.svg rename to dev/showcase/missing_physics/a583b620.svg index b7a615658bb..a5b4e43f3f1 100644 --- a/dev/showcase/missing_physics/c97f6698.svg +++ b/dev/showcase/missing_physics/a583b620.svg @@ -1,92 +1,92 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/missing_physics/604b577a.svg b/dev/showcase/missing_physics/b189172d.svg similarity index 79% rename from dev/showcase/missing_physics/604b577a.svg rename to dev/showcase/missing_physics/b189172d.svg index 6c03664f099..c24d2ba2101 100644 --- a/dev/showcase/missing_physics/604b577a.svg +++ b/dev/showcase/missing_physics/b189172d.svg @@ -1,92 +1,92 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/missing_physics/3d3d4a16.svg b/dev/showcase/missing_physics/bd202645.svg similarity index 83% rename from dev/showcase/missing_physics/3d3d4a16.svg rename to dev/showcase/missing_physics/bd202645.svg index c8426595b4b..5411f63bcff 100644 --- a/dev/showcase/missing_physics/3d3d4a16.svg +++ b/dev/showcase/missing_physics/bd202645.svg @@ -1,54 +1,54 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/missing_physics/f774f765.svg b/dev/showcase/missing_physics/ca401a68.svg similarity index 82% rename from dev/showcase/missing_physics/f774f765.svg rename to dev/showcase/missing_physics/ca401a68.svg index 8a689453679..af282835274 100644 --- a/dev/showcase/missing_physics/f774f765.svg +++ b/dev/showcase/missing_physics/ca401a68.svg @@ -1,155 +1,155 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/missing_physics/8b443a7a.svg b/dev/showcase/missing_physics/d1fa5e21.svg similarity index 90% rename from dev/showcase/missing_physics/8b443a7a.svg rename to dev/showcase/missing_physics/d1fa5e21.svg index c8ca64ddd6f..9136870db5f 100644 --- a/dev/showcase/missing_physics/8b443a7a.svg +++ b/dev/showcase/missing_physics/d1fa5e21.svg @@ -1,52 +1,52 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/missing_physics/index.html b/dev/showcase/missing_physics/index.html index 4e42a9fcdf9..f72b585f8f2 100644 --- a/dev/showcase/missing_physics/index.html +++ b/dev/showcase/missing_physics/index.html @@ -36,7 +36,7 @@ Xₙ = X .+ (noise_magnitude * x̄) .* randn(rng, eltype(X), size(X)) plot(solution, alpha = 0.75, color = :black, label = ["True Data" nothing]) -scatter!(t, transpose(Xₙ), color = :red, label = ["Noisy Data" nothing])Example block output

Definition of the Universal Differential Equation

Now let's define our UDE. We will use Lux.jl to define the neural network as follows:

rbf(x) = exp.(-(x .^ 2))
+scatter!(t, transpose(Xₙ), color = :red, label = ["Noisy Data" nothing])
Example block output

Definition of the Universal Differential Equation

Now let's define our UDE. We will use Lux.jl to define the neural network as follows:

rbf(x) = exp.(-(x .^ 2))
 
 # Multilayer FeedForward
 const U = Lux.Chain(Lux.Dense(2, 5, rbf), Lux.Dense(5, 5, rbf), Lux.Dense(5, 5, rbf),
@@ -186,26 +186,26 @@
 pl_losses = plot(1:5000, losses[1:5000], yaxis = :log10, xaxis = :log10,
                  xlabel = "Iterations", ylabel = "Loss", label = "ADAM", color = :blue)
 plot!(5001:length(losses), losses[5001:end], yaxis = :log10, xaxis = :log10,
-      xlabel = "Iterations", ylabel = "Loss", label = "BFGS", color = :red)
Example block output

Next, we compare the original data to the output of the UDE predictor. Note that we can even create more samples from the underlying model by simply adjusting the time steps!

## Analysis of the trained network
+      xlabel = "Iterations", ylabel = "Loss", label = "BFGS", color = :red)
Example block output

Next, we compare the original data to the output of the UDE predictor. Note that we can even create more samples from the underlying model by simply adjusting the time steps!

## Analysis of the trained network
 # Plot the data and the approximation
 ts = first(solution.t):(mean(diff(solution.t)) / 2):last(solution.t)
 X̂ = predict(p_trained, Xₙ[:, 1], ts)
 # Trained on noisy data vs real solution
 pl_trajectory = plot(ts, transpose(X̂), xlabel = "t", ylabel = "x(t), y(t)", color = :red,
                      label = ["UDE Approximation" nothing])
-scatter!(solution.t, transpose(Xₙ), color = :black, label = ["Measurements" nothing])
Example block output

Let's see how well the unknown term has been approximated:

# Ideal unknown interactions of the predictor
+scatter!(solution.t, transpose(Xₙ), color = :black, label = ["Measurements" nothing])
Example block output

Let's see how well the unknown term has been approximated:

# Ideal unknown interactions of the predictor
 Ȳ = [-p_[2] * (X̂[1, :] .* X̂[2, :])'; p_[3] * (X̂[1, :] .* X̂[2, :])']
 # Neural network guess
 Ŷ = U(X̂, p_trained, st)[1]
 
 pl_reconstruction = plot(ts, transpose(Ŷ), xlabel = "t", ylabel = "U(x,y)", color = :red,
                          label = ["UDE Approximation" nothing])
-plot!(ts, transpose(Ȳ), color = :black, label = ["True Interaction" nothing])
Example block output

And have a nice look at all the information:

# Plot the error
+plot!(ts, transpose(Ȳ), color = :black, label = ["True Interaction" nothing])
Example block output

And have a nice look at all the information:

# Plot the error
 pl_reconstruction_error = plot(ts, norm.(eachcol(Ȳ - Ŷ)), yaxis = :log, xlabel = "t",
                                ylabel = "L2-Error", label = nothing, color = :red)
 pl_missing = plot(pl_reconstruction, pl_reconstruction_error, layout = (2, 1))
 
-pl_overall = plot(pl_trajectory, pl_missing)
Example block output

That looks pretty good. And if we are happy with deep learning, we can leave it at that: we have trained a neural network to capture our missing dynamics.

But...

Can we also make it print out the LaTeX for what the missing equations were? Find out more after the break!

Symbolic regression via sparse regression (SINDy based)

This part of the showcase is still a work in progress... shame on us. But be back in a jiffy and we'll have it done.

Okay, that was a quick break, and that's good because this next part is pretty cool. Let's use DataDrivenDiffEq.jl to transform our trained neural network from machine learning mumbo jumbo into predictions of missing mechanistic equations. To do this, we first generate a symbolic basis that represents the space of mechanistic functions we believe this neural network should map to. Let's choose a bunch of polynomial functions:

@variables u[1:2]
+pl_overall = plot(pl_trajectory, pl_missing)
Example block output

That looks pretty good. And if we are happy with deep learning, we can leave it at that: we have trained a neural network to capture our missing dynamics.

But...

Can we also make it print out the LaTeX for what the missing equations were? Find out more after the break!

Symbolic regression via sparse regression (SINDy based)

This part of the showcase is still a work in progress... shame on us. But be back in a jiffy and we'll have it done.

Okay, that was a quick break, and that's good because this next part is pretty cool. Let's use DataDrivenDiffEq.jl to transform our trained neural network from machine learning mumbo jumbo into predictions of missing mechanistic equations. To do this, we first generate a symbolic basis that represents the space of mechanistic functions we believe this neural network should map to. Let's choose a bunch of polynomial functions:

@variables u[1:2]
 b = polynomial_basis(u, 4)
 basis = Basis(b, u);

\[ \begin{align} \varphi_1 =& 1 \\ @@ -224,8 +224,8 @@ \varphi_{1 4} =& u_2^{3} u_1 \\ \varphi_{1 5} =& u_2^{4} \end{align} - \]

Now let's define our DataDrivenProblems for the sparse regressions. To assess the capability of the sparse regression, we will look at 3 cases:

  • What if we trained no neural network and tried to automatically uncover the equations from the original noisy data? This is the approach in the literature known as structural identification of dynamical systems (SINDy). We will call this the full problem. This will assess whether this incorporation of prior information was helpful.
  • What if we trained the neural network using the ideal right-hand side missing derivative functions? This is the value computed in the plots above as Ȳ. This will tell us whether the symbolic discovery could work in ideal situations.
  • Do the symbolic regression directly on the function y = NN(x), i.e. the trained learned neural network. This is what we really want, and will tell us how to extend our known equations.

To define the full problem, we need to define a DataDrivenProblem that has the time series of the solution X, the time points of the solution t, and the derivative at each time point of the solution, obtained by the ODE solution's interpolation. We can just use an interpolation to get the derivative:

full_problem = ContinuousDataDrivenProblem(Xₙ, t)
Continuous DataDrivenProblem{Float64} ##DDProblem#192776 in 2 dimensions and 21 samples

Now for the other two symbolic regressions, we are regressing input/outputs of the missing terms, and thus we directly define the datasets as the input/output mappings like:

ideal_problem = DirectDataDrivenProblem(X̂, Ȳ)
-nn_problem = DirectDataDrivenProblem(X̂, Ŷ)
Direct DataDrivenProblem{Float64} ##DDProblem#192778 in 2 dimensions and 41 samples

Let's solve the data-driven problems using sparse regression. We will use the ADMM method, which requires we define a set of shrinking cutoff values λ, and we do this like:

λ = exp10.(-3:0.01:3)
+ \]
Now let's define our DataDrivenProblems for the sparse regressions. To assess the capability of the sparse regression, we will look at 3 cases:
  • What if we trained no neural network and tried to automatically uncover the equations from the original noisy data? This is the approach in the literature known as structural identification of dynamical systems (SINDy). We will call this the full problem. This will assess whether this incorporation of prior information was helpful.
  • What if we trained the neural network using the ideal right-hand side missing derivative functions? This is the value computed in the plots above as Ȳ. This will tell us whether the symbolic discovery could work in ideal situations.
  • Do the symbolic regression directly on the function y = NN(x), i.e. the trained learned neural network. This is what we really want, and will tell us how to extend our known equations.
To define the full problem, we need to define a DataDrivenProblem that has the time series of the solution X, the time points of the solution t, and the derivative at each time point of the solution, obtained by the ODE solution's interpolation. We can just use an interpolation to get the derivative:
full_problem = ContinuousDataDrivenProblem(Xₙ, t)
Continuous DataDrivenProblem{Float64} ##DDProblem#195380 in 2 dimensions and 21 samples
Now for the other two symbolic regressions, we are regressing input/outputs of the missing terms, and thus we directly define the datasets as the input/output mappings like:
ideal_problem = DirectDataDrivenProblem(X̂, Ȳ)
+nn_problem = DirectDataDrivenProblem(X̂, Ŷ)
Direct DataDrivenProblem{Float64} ##DDProblem#195382 in 2 dimensions and 41 samples
Let's solve the data-driven problems using sparse regression. We will use the ADMM method, which requires we define a set of shrinking cutoff values λ, and we do this like:
λ = exp10.(-3:0.01:3)
 opt = ADMM(λ)
DataDrivenSparse.ADMM{Vector{Float64}, Float64}([0.001, 0.0010232929922807535, 0.0010471285480508996, 0.001071519305237606, 0.0010964781961431851, 0.001122018454301963, 0.0011481536214968829, 0.001174897554939529, 0.001202264434617413, 0.0012302687708123812  …  812.8305161640995, 831.7637711026708, 851.1380382023768, 870.9635899560806, 891.2509381337459, 912.0108393559096, 933.2543007969915, 954.992586021436, 977.2372209558112, 1000.0], 1.0)
This is one of many methods for sparse regression, consult the DataDrivenDiffEq.jl documentation for more information on the algorithm choices. Taking this, let's solve each of the sparse regressions:
options = DataDrivenCommonOptions(maxiters = 10_000,
                                   normalize = DataNormalization(ZScoreTransform),
                                   selector = bic, digits = 1,
@@ -270,7 +270,7 @@
     println(eqs)
     println(get_parameter_map(eqs))
     println()
-end
Model ##Basis#192779 with 2 equations
+end
Model ##Basis#195383 with 2 equations
 States : u[1] u[2]
 Parameters : 6
 Independent variable: t
@@ -280,7 +280,7 @@
 
 Pair{SymbolicUtils.BasicSymbolic{Real}, Float64}[p₁ => 0.4, p₂ => -0.3, p₃ => 0.1, p₄ => -0.1, p₅ => 0.1, p₆ => -1.0]
 
-Model ##Basis#192783 with 2 equations
+Model ##Basis#195387 with 2 equations
 States : u[1] u[2]
 Parameters : p₁ p₂
 Independent variable: t
@@ -290,7 +290,7 @@
 
 Pair{SymbolicUtils.BasicSymbolic{Real}, Float64}[p₁ => -0.8, p₂ => 0.7]
 
-Model ##Basis#192787 with 2 equations
+Model ##Basis#195391 with 2 equations
 States : u[1] u[2]
 Parameters : p₁ p₂ p₃ p₄
 Independent variable: t
@@ -310,7 +310,7 @@
 
 # Plot
 plot(solution)
-plot!(estimate)
Example block output
We are still a bit off, so we fine tune the parameters by simply minimizing the residuals between the UDE predictor and our recovered parametrized equations:
function parameter_loss(p)
+plot!(estimate)
Example block output
We are still a bit off, so we fine tune the parameters by simply minimizing the residuals between the UDE predictor and our recovered parametrized equations:
function parameter_loss(p)
     Y = reduce(hcat, map(Base.Fix2(nn_eqs, p), eachcol(X̂)))
     sum(abs2, Ŷ .- Y)
 end
@@ -325,9 +325,9 @@
 t_long = (0.0, 50.0)
 estimation_prob = ODEProblem(recovered_dynamics!, u0, t_long, parameter_res)
 estimate_long = solve(estimation_prob, Tsit5(), saveat = 0.1) # Using higher tolerances here results in exit of julia
-plot(estimate_long)
Example block output
true_prob = ODEProblem(lotka!, u0, t_long, p_)
+plot(estimate_long)
Example block output
true_prob = ODEProblem(lotka!, u0, t_long, p_)
 true_solution_long = solve(true_prob, Tsit5(), saveat = estimate_long.t)
-plot!(true_solution_long)
Example block output
Post Processing and Plots
c1 = 3 # RGBA(174/255,192/255,201/255,1) # Maroon
+plot!(true_solution_long)
Example block output
Post Processing and Plots
c1 = 3 # RGBA(174/255,192/255,201/255,1) # Maroon
 c2 = :orange # RGBA(132/255,159/255,173/255,1) # Red
 c3 = :blue # RGBA(255/255,90/255,0,1) # Orange
 c4 = :purple # RGBA(153/255,50/255,204/255,1) # Purple
@@ -360,4 +360,4 @@
 annotate!([(1.5, 13, text("Training \nData", 10, :center, :top, :black, "Helvetica"))])
 l = @layout [grid(1, 2)
              grid(1, 1)]
-plot(p1, p2, p3, layout = l)
Example block output
+plot(p1, p2, p3, layout = l)
Example block output diff --git a/dev/showcase/ode_types/5a7e9cc0.svg b/dev/showcase/ode_types/36f8da94.svg similarity index 79% rename from dev/showcase/ode_types/5a7e9cc0.svg rename to dev/showcase/ode_types/36f8da94.svg index 595fcee8dcc..ee3c4b9a2b5 100644 --- a/dev/showcase/ode_types/5a7e9cc0.svg +++ b/dev/showcase/ode_types/36f8da94.svg @@ -1,136 +1,136 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/ode_types/174c148e.svg b/dev/showcase/ode_types/4ccd3ad4.svg similarity index 69% rename from dev/showcase/ode_types/174c148e.svg rename to dev/showcase/ode_types/4ccd3ad4.svg index 5d68f9a4711..57f73163035 100644 --- a/dev/showcase/ode_types/174c148e.svg +++ b/dev/showcase/ode_types/4ccd3ad4.svg @@ -1,251 +1,251 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/ode_types/dea91176.svg b/dev/showcase/ode_types/7a5651df.svg similarity index 67% rename from dev/showcase/ode_types/dea91176.svg rename to dev/showcase/ode_types/7a5651df.svg index f5eb4b4068e..b59b418ff60 100644 --- a/dev/showcase/ode_types/dea91176.svg +++ b/dev/showcase/ode_types/7a5651df.svg @@ -1,374 +1,374 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/ode_types/234de620.svg b/dev/showcase/ode_types/93f3d2e1.svg similarity index 88% rename from dev/showcase/ode_types/234de620.svg rename to dev/showcase/ode_types/93f3d2e1.svg index 56d771a9403..9c97202a43e 100644 --- a/dev/showcase/ode_types/234de620.svg +++ b/dev/showcase/ode_types/93f3d2e1.svg @@ -1,50 +1,50 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/ode_types/9149ebbe.svg b/dev/showcase/ode_types/cd59a14e.svg similarity index 69% rename from dev/showcase/ode_types/9149ebbe.svg rename to dev/showcase/ode_types/cd59a14e.svg index ff78e120508..91b04b18171 100644 --- a/dev/showcase/ode_types/9149ebbe.svg +++ b/dev/showcase/ode_types/cd59a14e.svg @@ -1,460 +1,460 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/ode_types/index.html b/dev/showcase/ode_types/index.html index 19616781dc8..5aaf4af5390 100644 --- a/dev/showcase/ode_types/index.html +++ b/dev/showcase/ode_types/index.html @@ -153,7 +153,7 @@ 1.5362091208988309 N 1.7720406194871123 N

This gives a normal solution object. Notice that the values are all with the correct units:

print(sol[:])
Unitful.Quantity{Float64, 𝐋 𝐌 𝐓^-2, Unitful.FreeUnits{(N,), 𝐋 𝐌 𝐓^-2, nothing}}[1.5 N, 1.5362091208988309 N, 1.7720406194871123 N]

And when we plot the solution, it automatically adds the units:

using Plots
 gr()
-plot(sol, lw = 3)
Example block output

Measurements.jl: Numbers with Linear Uncertainty Propagation

The result of a measurement should be given as a number with an attached uncertainty, besides the physical unit, and all operations performed involving the result of the measurement should propagate the uncertainty, taking care of correlation between quantities.

There is a Julia package for dealing with numbers with uncertainties: Measurements.jl. Thanks to Julia's features, DifferentialEquations.jl easily works together with Measurements.jl out-of-the-box.

Let's try to automate uncertainty propagation through number types on some classical physics examples!

Warning about Measurement type

Before going on with the tutorial, we must point up a subtlety of Measurements.jl that you should be aware of:

using Measurements
+plot(sol, lw = 3)
Example block output

Measurements.jl: Numbers with Linear Uncertainty Propagation

The result of a measurement should be given as a number with an attached uncertainty, besides the physical unit, and all operations performed involving the result of the measurement should propagate the uncertainty, taking care of correlation between quantities.

There is a Julia package for dealing with numbers with uncertainties: Measurements.jl. Thanks to Julia's features, DifferentialEquations.jl easily works together with Measurements.jl out-of-the-box.

Let's try to automate uncertainty propagation through number types on some classical physics examples!

Warning about Measurement type

Before going on with the tutorial, we must point up a subtlety of Measurements.jl that you should be aware of:

using Measurements
 5.23 ± 0.14 === 5.23 ± 0.14
false
(5.23 ± 0.14) - (5.23 ± 0.14)

\[0.0 \pm 0.2\]

(5.23 ± 0.14) / (5.23 ± 0.14)

\[1.0 \pm 0.038\]

The two numbers above, even though have the same nominal value and the same uncertainties, are actually two different measurements that only by chance share the same figures and their difference and their ratio have a non-zero uncertainty. It is common in physics to get very similar, or even equal, results for a repeated measurement, but the two measurements are not the same thing.

Instead, if you have one measurement and want to perform some operations involving it, you have to assign it to a variable:

x = 5.23 ± 0.14
 x === x
true
x - x

\[0.0 \pm 0.0\]

x / x

\[1.0 \pm 0.0\]

With that in mind, let's start using Measurements.jl for realsies.

Automated UQ on an ODE: Radioactive Decay of Carbon-14

The rate of decay of carbon-14 is governed by a first order linear ordinary differential equation:

\[\frac{\mathrm{d}u(t)}{\mathrm{d}t} = -\frac{u(t)}{\tau}\]

where $\tau$ is the mean lifetime of carbon-14, which is related to the half-life $t_{1/2} = (5730 \pm 40)$ years by the relation $\tau = t_{1/2}/\ln(2)$. Writing this in DifferentialEquations.jl syntax, this looks like:

# Half-life and mean lifetime of radiocarbon, in years
 t_12 = 5730 ± 40
@@ -217,7 +217,7 @@
      0.3817 ± 0.0026
      0.3163 ± 0.0025
      0.2983 ± 0.0025

How do the two solutions compare?

plot(sol.t, sol.u, label = "Numerical", xlabel = "Years", ylabel = "Fraction of Carbon-14")
-plot!(sol.t, u, label = "Analytic")
Example block output

The two curves are perfectly superimposed, indicating that the numerical solution matches the analytic one. We can check that also the uncertainties are correctly propagated in the numerical solution:

println("Quantity of carbon-14 after ", sol.t[11], " years:")
+plot!(sol.t, u, label = "Analytic")
Example block output

The two curves are perfectly superimposed, indicating that the numerical solution matches the analytic one. We can check that also the uncertainties are correctly propagated in the numerical solution:

println("Quantity of carbon-14 after ", sol.t[11], " years:")
 println("Numerical: ", sol[11])
 println("Analytic:  ", u[11])
Quantity of carbon-14 after 5207.541628507064 years:
 Numerical: 0.5326 ± 0.0023
@@ -286,7 +286,7 @@
  [0.0127 ± 0.0026, 0.0341 ± 0.0076]

And that's it! What about comparing it this time to the analytical solution?

u = u₀[2] .* cos.(sqrt(g / L) .* sol.t)
 
 plot(sol.t, getindex.(sol.u, 2), label = "Numerical")
-plot!(sol.t, u, label = "Analytic")
Example block output

Bingo. Also in this case there is a perfect superimposition between the two curves, including their uncertainties.

We can also have a look at the difference between the two solutions:

plot(sol.t, getindex.(sol.u, 2) .- u, label = "")
Example block output

Tiny difference on the order of the chosen 1e-6 tolerance.

Simple pendulum: Arbitrary amplitude

Now that we know how to solve differential equations involving numbers with uncertainties, we can solve the simple pendulum problem without any approximation. This time, the differential equation to solve is the following:

\[\ddot{\theta} + \frac{g}{L} \sin(\theta) = 0\]

That would be done via:

g = 9.79 ± 0.02; # Gravitational constants
+plot!(sol.t, u, label = "Analytic")
Example block output

Bingo. Also in this case there is a perfect superimposition between the two curves, including their uncertainties.

We can also have a look at the difference between the two solutions:

plot(sol.t, getindex.(sol.u, 2) .- u, label = "")
Example block output

Tiny difference on the order of the chosen 1e-6 tolerance.

Simple pendulum: Arbitrary amplitude

Now that we know how to solve differential equations involving numbers with uncertainties, we can solve the simple pendulum problem without any approximation. This time, the differential equation to solve is the following:

\[\ddot{\theta} + \frac{g}{L} \sin(\theta) = 0\]

That would be done via:

g = 9.79 ± 0.02; # Gravitational constants
 L = 1.00 ± 0.01; # Length of the pendulum
 
 #Initial Conditions
@@ -305,4 +305,4 @@
 prob = ODEProblem(simplependulum, u₀, tspan)
 sol = solve(prob, Tsit5(), reltol = 1e-6)
 
-plot(sol.t, getindex.(sol.u, 2), label = "Numerical")
Example block output

Warning about Linear Uncertainty Propagation

Measurements.jl uses linear uncertainty propagation, which has an error associated with it. MonteCarloMeasurements.jl has a page which showcases where this method can lead to incorrect uncertainty measurements. Thus for more nonlinear use cases, it's suggested that one uses one of the more powerful UQ methods, such as:

Basically, types can make the algorithm you want to run exceedingly simple to do, but make sure it's the correct algorithm!

+plot(sol.t, getindex.(sol.u, 2), label = "Numerical")Example block output

Warning about Linear Uncertainty Propagation

Measurements.jl uses linear uncertainty propagation, which has an error associated with it. MonteCarloMeasurements.jl has a page which showcases where this method can lead to incorrect uncertainty measurements. Thus for more nonlinear use cases, it's suggested that one uses one of the more powerful UQ methods, such as:

Basically, types can make the algorithm you want to run exceedingly simple to do, but make sure it's the correct algorithm!

diff --git a/dev/showcase/optimization_under_uncertainty/c47d2005.svg b/dev/showcase/optimization_under_uncertainty/0a47808e.svg similarity index 89% rename from dev/showcase/optimization_under_uncertainty/c47d2005.svg rename to dev/showcase/optimization_under_uncertainty/0a47808e.svg index d9a8f472533..9340540d374 100644 --- a/dev/showcase/optimization_under_uncertainty/c47d2005.svg +++ b/dev/showcase/optimization_under_uncertainty/0a47808e.svg @@ -1,50 +1,50 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/optimization_under_uncertainty/3a730523.svg b/dev/showcase/optimization_under_uncertainty/10a893d2.svg similarity index 58% rename from dev/showcase/optimization_under_uncertainty/3a730523.svg rename to dev/showcase/optimization_under_uncertainty/10a893d2.svg index ee98828d83c..9ced97e006a 100644 --- a/dev/showcase/optimization_under_uncertainty/3a730523.svg +++ b/dev/showcase/optimization_under_uncertainty/10a893d2.svg @@ -1,551 +1,551 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/optimization_under_uncertainty/200aeb10.svg b/dev/showcase/optimization_under_uncertainty/200aeb10.svg deleted file mode 100644 index 7dbaa2e4beb..00000000000 --- a/dev/showcase/optimization_under_uncertainty/200aeb10.svg +++ /dev/null @@ -1,150 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dev/showcase/optimization_under_uncertainty/39eff794.svg b/dev/showcase/optimization_under_uncertainty/39eff794.svg deleted file mode 100644 index 933d5aa7a0f..00000000000 --- a/dev/showcase/optimization_under_uncertainty/39eff794.svg +++ /dev/null @@ -1,150 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dev/showcase/optimization_under_uncertainty/847d49d1.svg b/dev/showcase/optimization_under_uncertainty/3b5a696d.svg similarity index 90% rename from dev/showcase/optimization_under_uncertainty/847d49d1.svg rename to dev/showcase/optimization_under_uncertainty/3b5a696d.svg index e303aeaff35..23bf0ea67cf 100644 --- a/dev/showcase/optimization_under_uncertainty/847d49d1.svg +++ b/dev/showcase/optimization_under_uncertainty/3b5a696d.svg @@ -1,47 +1,47 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/optimization_under_uncertainty/24fe6a59.svg b/dev/showcase/optimization_under_uncertainty/4b301c1c.svg similarity index 82% rename from dev/showcase/optimization_under_uncertainty/24fe6a59.svg rename to dev/showcase/optimization_under_uncertainty/4b301c1c.svg index dcc9e81bb82..ac0b84418ad 100644 --- a/dev/showcase/optimization_under_uncertainty/24fe6a59.svg +++ b/dev/showcase/optimization_under_uncertainty/4b301c1c.svg @@ -1,50 +1,50 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/optimization_under_uncertainty/251a8272.svg b/dev/showcase/optimization_under_uncertainty/9136ceba.svg similarity index 90% rename from dev/showcase/optimization_under_uncertainty/251a8272.svg rename to dev/showcase/optimization_under_uncertainty/9136ceba.svg index 3a5eac88c6c..a8c54cbe615 100644 --- a/dev/showcase/optimization_under_uncertainty/251a8272.svg +++ b/dev/showcase/optimization_under_uncertainty/9136ceba.svg @@ -1,45 +1,45 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/optimization_under_uncertainty/ac1faa55.svg b/dev/showcase/optimization_under_uncertainty/ac1faa55.svg new file mode 100644 index 00000000000..c2c854b11f4 --- /dev/null +++ b/dev/showcase/optimization_under_uncertainty/ac1faa55.svg @@ -0,0 +1,150 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/optimization_under_uncertainty/c16032af.svg b/dev/showcase/optimization_under_uncertainty/c16032af.svg new file mode 100644 index 00000000000..e77cd03904e --- /dev/null +++ b/dev/showcase/optimization_under_uncertainty/c16032af.svg @@ -0,0 +1,150 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/optimization_under_uncertainty/3f13945b.svg b/dev/showcase/optimization_under_uncertainty/d739c0f0.svg similarity index 62% rename from dev/showcase/optimization_under_uncertainty/3f13945b.svg rename to dev/showcase/optimization_under_uncertainty/d739c0f0.svg index 6f6bf72831b..9c98ec77f53 100644 --- a/dev/showcase/optimization_under_uncertainty/3f13945b.svg +++ b/dev/showcase/optimization_under_uncertainty/d739c0f0.svg @@ -1,551 +1,551 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/optimization_under_uncertainty/index.html b/dev/showcase/optimization_under_uncertainty/index.html index c0aac918330..d6462cba1d4 100644 --- a/dev/showcase/optimization_under_uncertainty/index.html +++ b/dev/showcase/optimization_under_uncertainty/index.html @@ -18,14 +18,14 @@ prob = ODEProblem(ball!, u0, tspan, p) sol = solve(prob, Tsit5(), callback = ground_cb) -plot(sol, vars = (1, 3), label = nothing, xlabel = "x", ylabel = "y")Example block output

For this particular problem, we wish to measure the impact distance from a point $y=25$ on a wall at $x=25$. So, we introduce an additional callback that terminates the simulation on wall impact.

stop_condition(u, t, integrator) = u[1] - 25.0
+plot(sol, vars = (1, 3), label = nothing, xlabel = "x", ylabel = "y")
Example block output

For this particular problem, we wish to measure the impact distance from a point $y=25$ on a wall at $x=25$. So, we introduce an additional callback that terminates the simulation on wall impact.

stop_condition(u, t, integrator) = u[1] - 25.0
 stop_cb = ContinuousCallback(stop_condition, terminate!)
 cbs = CallbackSet(ground_cb, stop_cb)
 
 tspan = (0.0, 1500.0)
 prob = ODEProblem(ball!, u0, tspan, p)
 sol = solve(prob, Tsit5(), callback = cbs)
-plot(sol, vars = (1, 3), label = nothing, xlabel = "x", ylabel = "y")
Example block output

To help visualize this problem, we plot as follows, where the star indicates a desired impact location

rectangle(xc, yc, w, h) = Shape(xc .+ [-w, w, w, -w] ./ 2.0, yc .+ [-h, -h, h, h] ./ 2.0)
+plot(sol, vars = (1, 3), label = nothing, xlabel = "x", ylabel = "y")
Example block output

To help visualize this problem, we plot as follows, where the star indicates a desired impact location

rectangle(xc, yc, w, h) = Shape(xc .+ [-w, w, w, -w] ./ 2.0, yc .+ [-h, -h, h, h] ./ 2.0)
 
 begin
     plot(sol, vars = (1, 3), label = nothing, lw = 3, c = :black)
@@ -34,7 +34,7 @@
     plot!(rectangle(27.5, 25, 5, 50), c = :red, label = nothing)
     scatter!([25], [25], marker = :star, ms = 10, label = nothing, c = :green)
     ylims!(0.0, 50.0)
-end
Example block output

Considering Uncertainty

We now wish to introduce uncertainty in p[2], the coefficient of restitution. This is defined via a continuous univariate distribution from Distributions.jl. We can then run a Monte Carlo simulation of 100 trajectories via the EnsembleProblem interface.

using Distributions
+end
Example block output

Considering Uncertainty

We now wish to introduce uncertainty in p[2], the coefficient of restitution. This is defined via a continuous univariate distribution from Distributions.jl. We can then run a Monte Carlo simulation of 100 trajectories via the EnsembleProblem interface.

using Distributions
 
 cor_dist = truncated(Normal(0.9, 0.02), 0.9 - 3 * 0.02, 1.0)
 trajectories = 100
@@ -52,7 +52,7 @@
     scatter!([25], [25], marker = :star, ms = 10, label = nothing, c = :green)
     plot!(sol, vars = (1, 3), label = nothing, lw = 3, c = :black, ls = :dash)
     xlims!(0.0, 27.5)
-end
Example block output

Here, we plot the first 350 Monte Carlo simulations along with the trajectory corresponding to the mean of the distribution (dashed line).

We now wish to compute the expected squared impact distance from the star. This is called an “observation” of our system or an “observable” of interest.

We define this observable as

obs(sol, p) = abs2(sol[3, end] - 25)
obs (generic function with 1 method)

With the observable defined, we can compute the expected squared miss distance from our Monte Carlo simulation results as

mean_ensemble = mean([obs(sol, p) for sol in ensemblesol])
35.28838879898037

Alternatively, we can use the Koopman() algorithm in SciMLExpectations.jl to compute this expectation much more efficiently as

using SciMLExpectations
+end
Example block output

Here, we plot the first 350 Monte Carlo simulations along with the trajectory corresponding to the mean of the distribution (dashed line).

We now wish to compute the expected squared impact distance from the star. This is called an “observation” of our system or an “observable” of interest.

We define this observable as

obs(sol, p) = abs2(sol[3, end] - 25)
obs (generic function with 1 method)

With the observable defined, we can compute the expected squared miss distance from our Monte Carlo simulation results as

mean_ensemble = mean([obs(sol, p) for sol in ensemblesol])
35.82033157028692

Alternatively, we can use the Koopman() algorithm in SciMLExpectations.jl to compute this expectation much more efficiently as

using SciMLExpectations
 gd = GenericDistribution(cor_dist)
 h(x, u, p) = u, [p[1]; x[1]]
 sm = SystemMap(prob, Tsit5(), callback = cbs)
@@ -92,7 +92,7 @@
     scatter!([25], [25], marker = :star, ms = 10, label = nothing, c = :green)
     ylims!(0.0, 50.0)
     xlims!(minx[1], 27.5)
-end
Example block output

Looks pretty good! But, how long did it take? Let's benchmark.

@time solve(opt_prob, optimizer)
u: 3-element Vector{Float64}:
+end
Example block output

Looks pretty good! But, how long did it take? Let's benchmark.

@time solve(opt_prob, optimizer)
u: 3-element Vector{Float64}:
   0.0
   2.4428947026478425
  49.20927899180528

Not bad for bound constrained optimization under uncertainty of a hybrid system!

Probabilistic Constraints

With this approach, we can also consider probabilistic constraints. Let us now consider a wall at $x=20$ with height 25.

constraint = [20.0, 25.0]
@@ -105,7 +105,7 @@
     scatter!([25], [25], marker = :star, ms = 10, label = nothing, c = :green)
     ylims!(0.0, 50.0)
     xlims!(minx[1], 27.5)
-end
Example block output

We now wish to minimize the same loss function as before, but introduce an inequality constraint such that the solution must have less than a 1% chance of colliding with the wall at $x=20$. This class of probabilistic constraints is called a chance constraint.

To do this, we first introduce a new callback and solve the system using the previous optimal solution

constraint_condition(u, t, integrator) = u[1] - constraint[1]
+end
Example block output

We now wish to minimize the same loss function as before, but introduce an inequality constraint such that the solution must have less than a 1% chance of colliding with the wall at $x=20$. This class of probabilistic constraints is called a chance constraint.

To do this, we first introduce a new callback and solve the system using the previous optimal solution

constraint_condition(u, t, integrator) = u[1] - constraint[1]
 function constraint_affect!(integrator)
     integrator.u[3] < constraint[2] ? terminate!(integrator) : nothing
 end
@@ -128,7 +128,7 @@
     scatter!([25], [25], marker = :star, ms = 10, label = nothing, c = :green)
     ylims!(0.0, 50.0)
     xlims!(minx[1], 27.5)
-end
Example block output

That doesn't look good!

We now need a second observable for the system. To compute a probability of impact, we use an indicator function for if a trajectory impacts the wall. In other words, this functions returns 1 if the trajectory hits the wall and 0 otherwise.

function constraint_obs(sol, p)
+end
Example block output

That doesn't look good!

We now need a second observable for the system. To compute a probability of impact, we use an indicator function for if a trajectory impacts the wall. In other words, this functions returns 1 if the trajectory hits the wall and 0 otherwise.

function constraint_obs(sol, p)
     sol((constraint[1] - sol[1, 1]) / sol[2, 1])[3] <= constraint[2] ? one(sol[1, end]) :
     zero(sol[1, end])
 end
constraint_obs (generic function with 1 method)

Using the previously computed optimal initial conditions, let's compute the probability of hitting this wall

sm = SystemMap(remake(prob, u0 = make_u0(minx)), Tsit5(), callback = cbs)
@@ -170,4 +170,4 @@
     scatter!([25], [25], marker = :star, ms = 10, label = nothing, c = :green)
     ylims!(0.0, 50.0)
     xlims!(minx[1], 27.5)
-end
Example block output +endExample block output diff --git a/dev/showcase/pinngpu/index.html b/dev/showcase/pinngpu/index.html index a0af7a38179..1c51fd41772 100644 --- a/dev/showcase/pinngpu/index.html +++ b/dev/showcase/pinngpu/index.html @@ -47,18 +47,18 @@ Dense(inner, inner, Lux.σ), Dense(inner, 1)) ps = Lux.setup(Random.default_rng(), chain)[1] -ps = ps |> ComponentArray
ComponentVector{Float32}(layer_1 = (weight = Float32[-0.31805974 0.09139786 0.41777533; 0.33261198 0.355208 0.3475564; … ; 0.019887907 0.01794524 0.4467731; -0.29428428 0.070409596 0.29158002], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_2 = (weight = Float32[0.110148214 -0.039831307 … -0.18552797 0.20934285; 0.03130965 0.036098626 … -0.3416626 0.08824845; … ; -0.062975965 -0.20154667 … -0.14984877 0.2576948; 0.05251867 -0.21741349 … -0.18838292 0.24400955], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_3 = (weight = Float32[0.30770952 -0.28010142 … 0.22652175 -0.24487144; -0.2648385 0.007866096 … 0.24210696 0.14350207; … ; 0.02278097 -0.2385938 … -0.33504015 -0.2527506; 0.21141843 -0.33944008 … 0.1068327 0.025542965], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_4 = (weight = Float32[-0.11138902 0.16841143 … -0.28922743 0.23058686; -0.050441105 0.22698173 … 0.2642822 0.19362028; … ; 0.21506791 0.05432761 … -0.13553715 0.10650118; 0.089259274 0.27197495 … 0.20043933 -0.2861113], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_5 = (weight = Float32[0.43197757 -0.19581653 … 0.3141886 0.21478513], bias = Float32[0.0;;]))

Step 4: Place it on the GPU.

Just plop it on that sucker. We must ensure that our initial parameters for the neural network are on the GPU. If that is done, then the internal computations will all take place on the GPU. This is done by using the gpu function on the initial parameters, like:

ps = ps |> gpu .|> Float64
ComponentVector{Float64}(layer_1 = (weight = [-0.3180597424507141 0.09139785915613174 0.41777533292770386; 0.33261197805404663 0.35520800948143005 0.3475564122200012; … ; 0.019887907430529594 0.017945239320397377 0.44677311182022095; -0.29428428411483765 0.07040959596633911 0.2915800213813782], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_2 = (weight = [0.11014821380376816 -0.03983130678534508 … -0.185527965426445 0.2093428522348404; 0.03130964934825897 0.036098625510931015 … -0.34166258573532104 0.08824844658374786; … ; -0.06297596544027328 -0.20154666900634766 … -0.14984877407550812 0.257694810628891; 0.05251866951584816 -0.21741348505020142 … -0.18838292360305786 0.24400955438613892], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_3 = (weight = [0.3077095150947571 -0.2801014184951782 … 0.22652174532413483 -0.24487143754959106; -0.26483848690986633 0.007866095751523972 … 0.24210695922374725 0.14350207149982452; … ; 0.022780969738960266 -0.23859380185604095 … -0.3350401520729065 -0.25275060534477234; 0.21141843497753143 -0.33944007754325867 … 0.10683269798755646 0.025542965158820152], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_4 = (weight = [-0.11138901859521866 0.16841143369674683 … -0.2892274260520935 0.23058685660362244; -0.05044110491871834 0.2269817292690277 … 0.2642821967601776 0.19362027943134308; … ; 0.21506790816783905 0.05432761088013649 … -0.13553714752197266 0.10650117695331573; 0.08925927430391312 0.27197495102882385 … 0.20043933391571045 -0.2861112952232361], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_5 = (weight = [0.4319775700569153 -0.1958165317773819 … 0.3141885995864868 0.2147851288318634], bias = [0.0;;]))

Step 5: Discretize the PDE via a PINN Training Strategy

strategy = GridTraining(0.05)
+ps = ps |> ComponentArray
ComponentVector{Float32}(layer_1 = (weight = Float32[-0.04209649 0.03640633 -0.42233258; -0.27310106 0.11635699 -0.00016003122; … ; -0.3885179 0.36692217 0.14476876; 0.40472105 -0.4149998 0.12267596], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_2 = (weight = Float32[0.008163133 0.0072792894 … 0.20061062 -0.04669632; -0.07312441 0.27643892 … -0.2523947 -0.24058221; … ; 0.2636631 0.0053683077 … 0.3085236 0.28209713; 0.28872296 0.31498188 … -0.16516055 0.1793684], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_3 = (weight = Float32[0.28254935 0.22810632 … -0.013064762 0.10094312; 0.28235918 -0.12003873 … -0.20721164 0.19388023; … ; -0.23300982 -0.04227186 … 0.07577099 -0.105062164; 0.24137248 -0.24671064 … -0.23774576 -0.29888633], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_4 = (weight = Float32[0.32447055 -0.24739334 … 0.0961311 0.18743312; 0.10031225 0.12210903 … 0.043552842 -0.01940066; … ; -0.13252878 -0.085322425 … -0.082256004 0.022710107; -0.30401477 -0.06619704 … 0.08506681 0.06184563], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_5 = (weight = Float32[-0.31794032 0.062833145 … 0.08427943 0.17099291], bias = Float32[0.0;;]))

Step 4: Place it on the GPU.

Just plop it on that sucker. We must ensure that our initial parameters for the neural network are on the GPU. If that is done, then the internal computations will all take place on the GPU. This is done by using the gpu function on the initial parameters, like:

ps = ps |> gpu .|> Float64
ComponentVector{Float64}(layer_1 = (weight = [-0.04209648817777634 0.03640633076429367 -0.4223325848579407; -0.2731010615825653 0.1163569912314415 -0.0001600312243681401; … ; -0.3885178864002228 0.36692216992378235 0.14476875960826874; 0.40472105145454407 -0.4149998128414154 0.12267596274614334], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_2 = (weight = [0.008163132704794407 0.007279289420694113 … 0.20061062276363373 -0.04669632017612457; -0.07312440872192383 0.2764389216899872 … -0.2523947060108185 -0.24058221280574799; … ; 0.263663113117218 0.005368307698518038 … 0.30852359533309937 0.28209713101387024; 0.288722962141037 0.3149818778038025 … -0.16516055166721344 0.17936840653419495], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_3 = (weight = [0.28254935145378113 0.2281063199043274 … -0.013064761646091938 0.10094311833381653; 0.28235918283462524 -0.12003873288631439 … -0.20721164345741272 0.19388023018836975; … ; -0.23300981521606445 -0.04227185994386673 … 0.07577098906040192 -0.10506216436624527; 0.2413724809885025 -0.2467106431722641 … -0.2377457618713379 -0.29888632893562317], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_4 = (weight = [0.32447054982185364 -0.24739333987236023 … 0.0961311012506485 0.187433123588562; 0.1003122478723526 0.12210903316736221 … 0.04355284199118614 -0.019400659948587418; … ; -0.13252878189086914 -0.08532242476940155 … -0.0822560042142868 0.022710107266902924; -0.30401477217674255 -0.06619703769683838 … 0.08506681025028229 0.061845630407333374], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_5 = (weight = [-0.3179403245449066 0.06283314526081085 … 0.08427943289279938 0.170992910861969], bias = [0.0;;]))

Step 5: Discretize the PDE via a PINN Training Strategy

strategy = GridTraining(0.05)
 discretization = PhysicsInformedNN(chain,
                                    strategy,
                                    init_params = ps)
 prob = discretize(pde_system, discretization)
OptimizationProblem. In-place: true
-u0: ComponentVector{Float64}(layer_1 = (weight = [-0.3180597424507141 0.09139785915613174 0.41777533292770386; 0.33261197805404663 0.35520800948143005 0.3475564122200012; … ; 0.019887907430529594 0.017945239320397377 0.44677311182022095; -0.29428428411483765 0.07040959596633911 0.2915800213813782], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_2 = (weight = [0.11014821380376816 -0.03983130678534508 … -0.185527965426445 0.2093428522348404; 0.03130964934825897 0.036098625510931015 … -0.34166258573532104 0.08824844658374786; … ; -0.06297596544027328 -0.20154666900634766 … -0.14984877407550812 0.257694810628891; 0.05251866951584816 -0.21741348505020142 … -0.18838292360305786 0.24400955438613892], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_3 = (weight = [0.3077095150947571 -0.2801014184951782 … 0.22652174532413483 -0.24487143754959106; -0.26483848690986633 0.007866095751523972 … 0.24210695922374725 0.14350207149982452; … ; 0.022780969738960266 -0.23859380185604095 … -0.3350401520729065 -0.25275060534477234; 0.21141843497753143 -0.33944007754325867 … 0.10683269798755646 0.025542965158820152], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_4 = (weight = [-0.11138901859521866 0.16841143369674683 … -0.2892274260520935 0.23058685660362244; -0.05044110491871834 0.2269817292690277 … 0.2642821967601776 0.19362027943134308; … ; 0.21506790816783905 0.05432761088013649 … -0.13553714752197266 0.10650117695331573; 0.08925927430391312 0.27197495102882385 … 0.20043933391571045 -0.2861112952232361], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_5 = (weight = [0.4319775700569153 -0.1958165317773819 … 0.3141885995864868 0.2147851288318634], bias = [0.0;;]))

Step 6: Solve the Optimization Problem

callback = function (p, l)
+u0: ComponentVector{Float64}(layer_1 = (weight = [-0.04209648817777634 0.03640633076429367 -0.4223325848579407; -0.2731010615825653 0.1163569912314415 -0.0001600312243681401; … ; -0.3885178864002228 0.36692216992378235 0.14476875960826874; 0.40472105145454407 -0.4149998128414154 0.12267596274614334], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_2 = (weight = [0.008163132704794407 0.007279289420694113 … 0.20061062276363373 -0.04669632017612457; -0.07312440872192383 0.2764389216899872 … -0.2523947060108185 -0.24058221280574799; … ; 0.263663113117218 0.005368307698518038 … 0.30852359533309937 0.28209713101387024; 0.288722962141037 0.3149818778038025 … -0.16516055166721344 0.17936840653419495], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_3 = (weight = [0.28254935145378113 0.2281063199043274 … -0.013064761646091938 0.10094311833381653; 0.28235918283462524 -0.12003873288631439 … -0.20721164345741272 0.19388023018836975; … ; -0.23300981521606445 -0.04227185994386673 … 0.07577098906040192 -0.10506216436624527; 0.2413724809885025 -0.2467106431722641 … -0.2377457618713379 -0.29888632893562317], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_4 = (weight = [0.32447054982185364 -0.24739333987236023 … 0.0961311012506485 0.187433123588562; 0.1003122478723526 0.12210903316736221 … 0.04355284199118614 -0.019400659948587418; … ; -0.13252878189086914 -0.08532242476940155 … -0.0822560042142868 0.022710107266902924; -0.30401477217674255 -0.06619703769683838 … 0.08506681025028229 0.061845630407333374], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_5 = (weight = [-0.3179403245449066 0.06283314526081085 … 0.08427943289279938 0.170992910861969], bias = [0.0;;]))

Step 6: Solve the Optimization Problem

callback = function (p, l)
     println("Current loss is: $l")
     return false
 end
 
-res = Optimization.solve(prob, Adam(0.01); callback = callback, maxiters = 2500);
u: ComponentVector{Float64}(layer_1 = (weight = [-2.5220170881159354 0.29797818384476094 0.2670176888668424; -2.2918721717641084 0.2785461489181596 0.23426295691637433; … ; 1.1091474519738806 0.4626718849258362 0.49840882243876183; 1.123137018962559 0.9146674937302515 1.0332178537856234], bias = [-0.23519256552968168; -0.09247989489310907; … ; -2.953408089339703; -1.2540572060553212;;]), layer_2 = (weight = [-0.04871252139309415 -0.2407933704128366 … -1.052197820343992 0.6229933507326167; 0.37388332243910954 0.23997189154688894 … -2.0304206483954923 -0.3756721126462302; … ; -1.840156550704733 -1.807506707245994 … 0.7833496693140999 0.3689494374190989; 1.4203590477536803 0.9233808833653475 … 0.5938864265803286 0.39373060681470534], bias = [0.04349818891653107; 0.08016971145930621; … ; -0.23276456089165562; -0.04723306891700018;;]), layer_3 = (weight = [1.5507512790740807 -2.5338859611709834 … 1.340848761718156 -1.0139371266453179; -0.06519062941850728 0.33243342431502076 … 0.9940028157506032 -1.00036503588841; … ; 0.3464526628296136 0.37035952177075365 … -0.5271059281081063 -2.078534723170858; 1.694278293417124 -2.9437086854558254 … 1.4731916078350094 -0.5820688501652579], bias = [-0.25949163032341493; 0.043192532125077315; … ; -0.30490692694266536; -0.4246887698187586;;]), layer_4 = (weight = [-1.7520584545001983 -0.19129967467720885 … -0.9990324207596009 -1.9111518257803437; 1.6305680455781715 0.4960374492631608 … 1.5388013096746918 1.9542028287740196; … ; -1.020012350849501 -0.3677596268847341 … -0.6793730487107279 -1.2828689551328996; -1.6043775233215527 0.5204930598901248 … 0.7687967547362488 -2.3015615364019615], bias = [1.035049793659374; -1.3889876571502242; … ; 1.0168264053244231; 1.0709607334331297;;]), layer_5 = (weight = [7.496439508792172 -3.811943821418891 … 5.612700144756875 5.616142301597866], bias = [1.0852238014458466;;]))

We then use the remake function to rebuild the PDE problem to start a new optimization at the optimized parameters, and continue with a lower learning rate:

prob = remake(prob, u0 = res.u)
-res = Optimization.solve(prob, Adam(0.001); callback = callback, maxiters = 2500);
u: ComponentVector{Float64}(layer_1 = (weight = [-2.513021154770015 0.28015594902345087 0.24975956052005763; -2.278358720055486 0.25813376834262985 0.2267717304737042; … ; 0.9941086677763132 0.47223780507477536 0.5032013110168644; 1.1523736529224782 0.9464661701311868 0.9999684244995865], bias = [-0.23455428328311928; -0.08623426022862822; … ; -2.981283811098214; -1.2502208728386908;;]), layer_2 = (weight = [-0.05515653717535878 -0.24804033394291625 … -1.0504469842145687 0.6167179388079069; 0.34982264687622056 0.215739116951086 … -2.0307881179288234 -0.3829450708715097; … ; -1.8601456122958073 -1.8284839487587072 … 0.7958624798194162 0.36824707638724813; 1.3946457878971026 0.9000003667334605 … 0.6150841237588397 0.39598492425368925], bias = [0.03574777713408737; 0.0764492218767386; … ; -0.23249053351968207; -0.0453512607112972;;]), layer_3 = (weight = [1.545638424905824 -2.5246453282453323 … 1.370995807551898 -0.998907031619789; -0.1915267769207282 0.146104103139531 … 1.0226994413710215 -1.122561799100511; … ; 0.19793270204963623 0.46791824735082 … -0.5349413927967688 -2.169066188568371; 1.6928157508691473 -2.988141256973519 … 1.4858626588083335 -0.5783478310481953], bias = [-0.24956459882227394; 0.0007744758867360564; … ; -0.2943155923529731; -0.4163952605784402;;]), layer_4 = (weight = [-1.7646900046215064 -0.1892605488019787 … -1.002583839670708 -1.877399353885816; 1.617242568202816 0.5464342656392136 … 1.686758858911586 1.9247738974972004; … ; -1.0178036777547372 -0.3327193235006818 … -0.6570522532160115 -1.32725652434867; -1.563408062094862 0.4616196073860088 … 0.7366552012886828 -2.1644385251239058], bias = [1.0488135194333021; -1.411917261629015; … ; 1.0502541819746465; 1.049295480685738;;]), layer_5 = (weight = [8.028621566661942 -3.911697022399661 … 5.728297057763678 5.746274150120693], bias = [1.071798801679538;;]))

Step 7: Inspect the PINN's Solution

Finally, we inspect the solution:

phi = discretization.phi
+res = Optimization.solve(prob, Adam(0.01); callback = callback, maxiters = 2500);
u: ComponentVector{Float64}(layer_1 = (weight = [-2.3529280487784394 -0.5237888909562819 -0.5619596533247538; -3.1737147371912156 0.592263907183205 0.5756038747971065; … ; -3.6942731743015567 -0.1737466497981103 -0.22011316467206446; -1.876320378644675 -0.06099939150796067 0.017936332750268544], bias = [1.431337711003383; -1.8270812880662954; … ; 0.5195345356593984; 0.5560070031972858;;]), layer_2 = (weight = [-0.6643559383321906 1.0004153103646407 … -2.5476532233561913 0.3531675736288698; 1.6100092772872596 0.3553268438494184 … 2.1918988107881505 0.7379396567230324; … ; 1.5706927146426801 0.00034077015345366113 … -2.437627655038228 0.958706182788082; -0.236077791278979 2.768085478804406 … 0.45414038352219493 0.6675262859368354], bias = [0.17053081280175406; 0.0575156259939366; … ; -0.07019716174955853; 0.10719201990099851;;]), layer_3 = (weight = [-0.08719477914568292 -0.37546011625935105 … -0.4673969725410324 -0.19601454032212604; -0.6812808519620843 1.2159348944258248 … -0.3850896246458142 -1.0566860386400423; … ; -1.1843820719082594 0.6984925888250586 … 0.03981057655868366 -1.1466814949694706; -0.7663678580348198 -1.2602042962405222 … -1.2874561359522854 -1.2323517987812422], bias = [0.027850447782629403; 0.32484378234756395; … ; 0.09088885449638129; 0.17952285527750056;;]), layer_4 = (weight = [-1.2045404187996724 0.7059572350410684 … 0.8715180346912945 1.7560202455215137; 1.6466102188399392 -1.7572512469804815 … -1.6628129037020742 -2.10832250270898; … ; 1.686295187854086 -1.42310415286527 … -1.286726510163923 -1.405908971302427; 0.24961962271121982 -0.2939864170863261 … 0.47285620776143467 -1.1978962364717642], bias = [-1.112846398084647; 1.138876711401969; … ; 1.3159248303900797; 0.5894416830011179;;]), layer_5 = (weight = [-2.7865503791140016 7.74266163909326 … 7.260540746448829 6.4105619673184275], bias = [1.4181614986764302;;]))

We then use the remake function to rebuild the PDE problem to start a new optimization at the optimized parameters, and continue with a lower learning rate:

prob = remake(prob, u0 = res.u)
+res = Optimization.solve(prob, Adam(0.001); callback = callback, maxiters = 2500);
u: ComponentVector{Float64}(layer_1 = (weight = [-1.7734865823232784 -0.38435194674107837 -0.4064862088690445; -3.2518182390194994 0.5907185259774006 0.5665374751137972; … ; -3.6753292771782884 -0.19615672687939983 -0.2416995140537983; -1.6441015675584179 0.030965687044392894 0.051702935612733515], bias = [1.699740752520274; -1.8512414717253791; … ; 0.5735287666490125; 0.7568646580396544;;]), layer_2 = (weight = [-0.6442921496545139 1.0835201955007427 … -2.580999351299536 0.4594616501099933; 1.296514829048729 0.4840626127012573 … 3.000620437254965 0.4665122934216573; … ; 1.5454656777769034 -0.015594584847440496 … -2.49458279869677 0.9344954676988106; -0.23685568073575314 2.989336406544804 … 0.5298598292812522 0.7405659309001622], bias = [0.15066185338842866; 0.01782008658922495; … ; -0.11341572067518914; 0.10883553077055874;;]), layer_3 = (weight = [-0.15011866048236203 -0.15003861565937648 … -0.5858740250583889 0.09987643925211472; -0.7579594727169912 1.5199541747642558 … -0.4351533937772534 -1.1159482911394398; … ; -1.2120259442035672 0.7109016753689595 … 0.050784930378121214 -1.2765395775423882; -0.7317437150130681 -1.292548718422605 … -1.257718088299417 -1.250397665368042], bias = [-0.09208176847450007; 0.313426950410258; … ; 0.12120281153579898; 0.27908286333507576;;]), layer_4 = (weight = [-1.3910786561064312 0.46640177519541204 … 0.4887796888999915 2.152906891427832; 2.000167086735794 -2.1787501271403626 … -2.0384119101781324 -2.308617788347178; … ; 1.8561773030450175 -1.4199851742935485 … -1.3562372709027184 -1.4895141956963533; 0.5118754139947149 -0.45440426879728724 … 0.29349389692729716 -1.6873215351854018], bias = [-1.1929036620665898; 1.4669427314739412; … ; 1.4577391062313125; 0.6446118400045179;;]), layer_5 = (weight = [-1.7927020978463681 8.931744826513707 … 7.933590321432513 8.785057046771685], bias = [2.2863574036043257;;]))

Step 7: Inspect the PINN's Solution

Finally, we inspect the solution:

phi = discretization.phi
 ts, xs, ys = [infimum(d.domain):0.1:supremum(d.domain) for d in domains]
 u_real = [analytic_sol_func(t, x, y) for t in ts for x in xs for y in ys]
 u_predict = [first(Array(phi(gpu([t, x, y]), res.u))) for t in ts for x in xs for y in ys]
@@ -83,4 +83,4 @@
     gif(anim, "3pde.gif", fps = 10)
 end
 
-plot_(res)

3pde

+plot_(res)

3pde

diff --git a/dev/showcase/showcase/index.html b/dev/showcase/showcase/index.html index d9b1edb3ff7..bb674674525 100644 --- a/dev/showcase/showcase/index.html +++ b/dev/showcase/showcase/index.html @@ -1,2 +1,2 @@ -The SciML Showcase · Overview of Julia's SciML
GitHub

The SciML Showcase

The SciML Showcase is a display of some cool things that can be done by connecting SciML software.

Note

The SciML Showcase is not meant to be training/tutorials, but inspirational demonstrations! If you're looking for simple examples to get started with, check out the getting started section.

Want to see some cool things that you can do with SciML? Check out the following:

+The SciML Showcase · Overview of Julia's SciML
GitHub

The SciML Showcase

The SciML Showcase is a display of some cool things that can be done by connecting SciML software.

Note

The SciML Showcase is not meant to be training/tutorials, but inspirational demonstrations! If you're looking for simple examples to get started with, check out the getting started section.

Want to see some cool things that you can do with SciML? Check out the following:

diff --git a/dev/showcase/symbolic_analysis/5e671764.svg b/dev/showcase/symbolic_analysis/1c89b5cd.svg similarity index 98% rename from dev/showcase/symbolic_analysis/5e671764.svg rename to dev/showcase/symbolic_analysis/1c89b5cd.svg index 5b422c6d7bf..2a11a08d39a 100644 --- a/dev/showcase/symbolic_analysis/5e671764.svg +++ b/dev/showcase/symbolic_analysis/1c89b5cd.svg @@ -1,54 +1,54 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/symbolic_analysis/257fb233.svg b/dev/showcase/symbolic_analysis/244e1c76.svg similarity index 96% rename from dev/showcase/symbolic_analysis/257fb233.svg rename to dev/showcase/symbolic_analysis/244e1c76.svg index de7d9a2df78..5b76e9c7626 100644 --- a/dev/showcase/symbolic_analysis/257fb233.svg +++ b/dev/showcase/symbolic_analysis/244e1c76.svg @@ -1,54 +1,54 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/symbolic_analysis/c175b5fa.svg b/dev/showcase/symbolic_analysis/c6d9db61.svg similarity index 98% rename from dev/showcase/symbolic_analysis/c175b5fa.svg rename to dev/showcase/symbolic_analysis/c6d9db61.svg index 14539114611..58d0dfec760 100644 --- a/dev/showcase/symbolic_analysis/c175b5fa.svg +++ b/dev/showcase/symbolic_analysis/c6d9db61.svg @@ -1,54 +1,54 @@ - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/symbolic_analysis/index.html b/dev/showcase/symbolic_analysis/index.html index d42bea4856f..adfe4e4eb2b 100644 --- a/dev/showcase/symbolic_analysis/index.html +++ b/dev/showcase/symbolic_analysis/index.html @@ -23,7 +23,7 @@ pendulum_sys = structural_simplify(dae_index_lowering(traced_sys)) prob = ODAEProblem(pendulum_sys, [], tspan) sol = solve(prob, Tsit5(), abstol = 1e-8, reltol = 1e-8) -plot(sol, vars = states(traced_sys))Example block output

Explanation

Attempting to Solve the Equation

In this tutorial, we will look at the pendulum system:

\[\begin{aligned} +plot(sol, vars = states(traced_sys))Example block output

Explanation

Attempting to Solve the Equation

In this tutorial, we will look at the pendulum system:

\[\begin{aligned} x^\prime &= v_x\\ v_x^\prime &= Tx\\ y^\prime &= v_y\\ @@ -74,11 +74,11 @@ sol = solve(prob, Rodas4()) using Plots -plot(sol, vars = states(traced_sys))Example block output

Note that plotting using states(traced_sys) is done so that any variables which are symbolically eliminated, or any variable reordering done for enhanced parallelism/performance, still show up in the resulting plot and the plot is shown in the same order as the original numerical code.

Note that we can even go a bit further. If we use the ODAEProblem constructor, we can remove the algebraic equations from the states of the system and fully transform the index-3 DAE into an index-0 ODE, which can be solved via an explicit Runge-Kutta method:

traced_sys = modelingtoolkitize(pendulum_prob)
+plot(sol, vars = states(traced_sys))
Example block output

Note that plotting using states(traced_sys) is done so that any variables which are symbolically eliminated, or any variable reordering done for enhanced parallelism/performance, still show up in the resulting plot and the plot is shown in the same order as the original numerical code.

Note that we can even go a bit further. If we use the ODAEProblem constructor, we can remove the algebraic equations from the states of the system and fully transform the index-3 DAE into an index-0 ODE, which can be solved via an explicit Runge-Kutta method:

traced_sys = modelingtoolkitize(pendulum_prob)
 pendulum_sys = structural_simplify(dae_index_lowering(traced_sys))
 prob = ODAEProblem(pendulum_sys, Pair[], tspan)
 sol = solve(prob, Tsit5(), abstol = 1e-8, reltol = 1e-8)
-plot(sol, vars = states(traced_sys))
Example block output

And there you go: this has transformed the model from being too hard to solve with implicit DAE solvers, to something that is easily solved with explicit Runge-Kutta methods for non-stiff equations.

Parameter Identifiability in ODE Models

Ordinary differential equations are commonly used for modeling real-world processes. The problem of parameter identifiability is one of the key design challenges for mathematical models. A parameter is said to be identifiable if one can recover its value from experimental data. Structural identifiability is a theoretical property of a model that answers this question. In this tutorial, we will show how to use StructuralIdentifiability.jl with ModelingToolkit.jl to assess identifiability of parameters in ODE models. The theory behind StructuralIdentifiability.jl is presented in paper [4].

We will start by illustrating local identifiability in which a parameter is known up to finitely many values, and then proceed to determining global identifiability, that is, which parameters can be identified uniquely.

To install StructuralIdentifiability.jl, simply run

using Pkg
+plot(sol, vars = states(traced_sys))
Example block output

And there you go: this has transformed the model from being too hard to solve with implicit DAE solvers, to something that is easily solved with explicit Runge-Kutta methods for non-stiff equations.

Parameter Identifiability in ODE Models

Ordinary differential equations are commonly used for modeling real-world processes. The problem of parameter identifiability is one of the key design challenges for mathematical models. A parameter is said to be identifiable if one can recover its value from experimental data. Structural identifiability is a theoretical property of a model that answers this question. In this tutorial, we will show how to use StructuralIdentifiability.jl with ModelingToolkit.jl to assess identifiability of parameters in ODE models. The theory behind StructuralIdentifiability.jl is presented in paper [4].

We will start by illustrating local identifiability in which a parameter is known up to finitely many values, and then proceed to determining global identifiability, that is, which parameters can be identified uniquely.

To install StructuralIdentifiability.jl, simply run

using Pkg
 Pkg.add("StructuralIdentifiability")

The package has a standalone data structure for ordinary differential equations, but is also compatible with ODESystem type from ModelingToolkit.jl.

Local Identifiability

Input System

We will consider the following model:

\[\begin{cases} \frac{d\,x_4}{d\,t} = - \frac{k_5 x_4}{k_6 + x_4},\\ \frac{d\,x_5}{d\,t} = \frac{k_5 x_4}{k_6 + x_4} - \frac{k_7 x_5}{(k_8 + x_5 + x_6)},\\ @@ -175,4 +175,4 @@ funcs_to_check = to_check, p = 0.9) # Dict{Num, Symbol} with 2 entries: # b => :globally -# c => :globally

Both parameters b, c are globally identifiable with probability 0.9 in this case.

+# c => :globally

Both parameters b, c are globally identifiable with probability 0.9 in this case.