From 91bfa177a38f99ce01feb37c894f476be75c3604 Mon Sep 17 00:00:00 2001 From: "Documenter.jl" Date: Wed, 20 Sep 2023 01:44:54 +0000 Subject: [PATCH] build based on 49e330bde --- dev/comparisons/cppfortran/index.html | 2 +- dev/comparisons/matlab/index.html | 2 +- dev/comparisons/python/index.html | 2 +- dev/comparisons/r/index.html | 2 +- dev/getting_started/find_root/index.html | 2 +- .../first_optimization/index.html | 2 +- .../first_simulation/index.html | 418 +- dev/getting_started/fit_simulation/index.html | 2 +- .../getting_started/index.html | 2 +- dev/getting_started/installation/index.html | 2 +- dev/highlevels/array_libraries/index.html | 2 +- .../developer_documentation/index.html | 2 +- dev/highlevels/equation_solvers/index.html | 2 +- .../function_approximation/index.html | 2 +- dev/highlevels/implicit_layers/index.html | 2 +- dev/highlevels/interfaces/index.html | 2 +- dev/highlevels/inverse_problems/index.html | 2 +- dev/highlevels/learning_resources/index.html | 2 +- .../model_libraries_and_importers/index.html | 2 +- dev/highlevels/modeling_languages/index.html | 2 +- dev/highlevels/numerical_utilities/index.html | 2 +- dev/highlevels/parameter_analysis/index.html | 2 +- .../index.html | 2 +- dev/highlevels/plots_visualization/index.html | 2 +- dev/highlevels/symbolic_learning/index.html | 2 +- dev/highlevels/symbolic_tools/index.html | 2 +- .../uncertainty_quantification/index.html | 2 +- dev/index.html | 2 +- dev/overview/index.html | 2 +- dev/search/index.html | 2 +- dev/showcase/bayesian_neural_ode/index.html | 3062 +++++++------- dev/showcase/blackhole/index.html | 868 ++-- dev/showcase/brusselator/index.html | 2 +- dev/showcase/gpu_spde/index.html | 3522 ++++++++--------- .../massively_parallel_gpu/index.html | 2 +- dev/showcase/missing_physics/index.html | 1282 +++--- dev/showcase/ode_types/index.html | 2402 +++++------ .../optimization_under_uncertainty/index.html | 2952 +++++++------- dev/showcase/pinngpu/index.html | 10 +- dev/showcase/showcase/index.html | 2 +- dev/showcase/symbolic_analysis/index.html | 242 +- 41 files changed, 7417 insertions(+), 7405 deletions(-) diff --git a/dev/comparisons/cppfortran/index.html b/dev/comparisons/cppfortran/index.html index 0b5a6beb193..0b724deeb6b 100644 --- a/dev/comparisons/cppfortran/index.html +++ b/dev/comparisons/cppfortran/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Getting Started with Julia's SciML for the C++/Fortran User

You don't need help if you're a Fortran guru. I'm just kidding, you're not a Lisp developer. If you're coming from C++ or Fortran, you may be familiar with high-performance computing environments similar to SciML, such as PETSc, Trilinos, or Sundials. The following are some points to help the transition.

Why SciML? High-Level Workflow Reasons

If you're coming from “hardcore” C++/Fortran computing environments, some things to check out with Julia's SciML are:

  • Interactivity - use the interactive REPL to easily investigate numerical details.
  • Metaprogramming performance tools - tools like LoopVectorization.jl can be used to generate faster code than even some of the most hand-optimized C++/Fortran code. Current benchmarks show this SIMD-optimized Julia code outperforming OpenBLAS and MKL BLAS implementations in many performance regimes.
  • Symbolic modeling languages - writing models by hand can leave a lot of performance on the table. Using high-level modeling tools like ModelingToolkit can automate symbolic simplifications, which improve the stability and performance of numerical solvers. On complex models, even the best handwritten C++/Fortran code is orders of magnitude behind the code that symbolic tearing algorithms can achieve!
  • Composable Library Components - In C++/Fortran environments, every package feels like a silo. Arrays made for PETSc cannot easily be used in Trilinos, and converting Sundials NVector outputs to DataFrames for post-simulation data processing is a process itself. The Julia SciML environment embraces interoperability. Don't wait for SciML to do it: by using generic coding with JIT compilation, these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • Wrappers to the Libraries You Know and Trust - Moving to SciML does not have to be a quick transition. SciML has extensive wrappers to many widely-used classical solver environments such as SUNDIALS and Hairer's classic Fortran ODE solvers (dopri5, dop853, etc.). Using these wrapped solvers is painless and can be swapped in for the Julia versions with one line of code. This gives you a way to incrementally adopt new features/methods while retaining the older pieces you know and trust.
  • Don't Start from Scratch - SciML builds on the extensive Base library of Julia, and thus grows and improves with every update to the language. With hundreds of monthly contributors to SciML and hundreds of monthly contributors to Julia, SciML is one of the most actively developed open-source scientific computing ecosystems out there!
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, Sundials/Hairer in purple/red represent C++/Fortrans most commonly used solvers:

Why SciML? Some Technical Details

Let's face the facts, in the open benchmarks the pure-Julia solvers tend to outperform the classic “best” C++ and Fortran solvers in almost every example (with a few notable exceptions). But why?

The answer is two-fold: Julia is as fast as C++/Fortran, and the algorithms are what matter.

Julia is as Fast as C++/Fortran

While Julia code looks high level like Python or MATLAB, its performance is on par with C++ and Fortran. At a technical level, when Julia code is type-stable, i.e. that the types that are returned from a function are deducible at compile-time from the types that go into a function, then Julia can optimize it as much as C++ or Fortran by automatically devirtualizing all dynamic behavior and compile-time optimizing the quasi-static code. This is not an empirical statement, it's a provable type-theoretic result. The resulting compiler used on the resulting quasi-static representation is LLVM, the same optimizing compiler used by clang and LFortran.

For more details on how Julia code is optimized and how to optimize your own Julia code, check out this chapter from the SciML Book.

SciML's Julia Algorithms Have Performance Advantages in Many Common Regimes

There are many ways which Julia's algorithms achieve performance advantages. Some facts to highlight include:

Let's Dig Deep Into One Case: Adjoints of ODEs for Solving Inverse Problems

To really highlight how JIT compilation and automatic differentiation integration can change algorithms, let's look at the problem of differentiating an ODE solver. As is derived and discussed in detail at a seminar with the American Statistical Association, there are many ways to implement well-known “adjoint” methods which are required for performance. Each has different stability and performance trade-offs, and Julia's SciML is the only system to systemically offer all of the trade-off options. In many cases, using analytical adjoints of a solver is not advised due to performance reasons, with the trade-off described in detail here. Likewise, even when analytical adjoints are used, it turns out that for general nonlinear equations there is a trick which uses automatic differentiation in the construction of the analytical adjoint to improve its performance. As demonstrated in this publication, this can lead to about 2-3 orders of magnitude performance improvements. These AD-enhanced adjoints are showcased as the seeding methods in this plot:

Unless one directly defines special “vjp” functions, this is how the Julia SciML methods achieve orders of magnitude performance advantages over CVODES's adjoints and PETSC's TS-adjoint.

Moral of the story, even there are many reasons to use automatic differentiation of a solver, and even if an analytical adjoint rule is used for some specific performance reason, that analytical expression can often times be accelerated by orders of magnitude itself by embedding some form of automatic differentiation into it. This is just one algorithm of many which are optimized in this fashion.

+
Edit on GitHub

Getting Started with Julia's SciML for the C++/Fortran User

You don't need help if you're a Fortran guru. I'm just kidding, you're not a Lisp developer. If you're coming from C++ or Fortran, you may be familiar with high-performance computing environments similar to SciML, such as PETSc, Trilinos, or Sundials. The following are some points to help the transition.

Why SciML? High-Level Workflow Reasons

If you're coming from “hardcore” C++/Fortran computing environments, some things to check out with Julia's SciML are:

  • Interactivity - use the interactive REPL to easily investigate numerical details.
  • Metaprogramming performance tools - tools like LoopVectorization.jl can be used to generate faster code than even some of the most hand-optimized C++/Fortran code. Current benchmarks show this SIMD-optimized Julia code outperforming OpenBLAS and MKL BLAS implementations in many performance regimes.
  • Symbolic modeling languages - writing models by hand can leave a lot of performance on the table. Using high-level modeling tools like ModelingToolkit can automate symbolic simplifications, which improve the stability and performance of numerical solvers. On complex models, even the best handwritten C++/Fortran code is orders of magnitude behind the code that symbolic tearing algorithms can achieve!
  • Composable Library Components - In C++/Fortran environments, every package feels like a silo. Arrays made for PETSc cannot easily be used in Trilinos, and converting Sundials NVector outputs to DataFrames for post-simulation data processing is a process itself. The Julia SciML environment embraces interoperability. Don't wait for SciML to do it: by using generic coding with JIT compilation, these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • Wrappers to the Libraries You Know and Trust - Moving to SciML does not have to be a quick transition. SciML has extensive wrappers to many widely-used classical solver environments such as SUNDIALS and Hairer's classic Fortran ODE solvers (dopri5, dop853, etc.). Using these wrapped solvers is painless and can be swapped in for the Julia versions with one line of code. This gives you a way to incrementally adopt new features/methods while retaining the older pieces you know and trust.
  • Don't Start from Scratch - SciML builds on the extensive Base library of Julia, and thus grows and improves with every update to the language. With hundreds of monthly contributors to SciML and hundreds of monthly contributors to Julia, SciML is one of the most actively developed open-source scientific computing ecosystems out there!
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, Sundials/Hairer in purple/red represent C++/Fortrans most commonly used solvers:

Why SciML? Some Technical Details

Let's face the facts, in the open benchmarks the pure-Julia solvers tend to outperform the classic “best” C++ and Fortran solvers in almost every example (with a few notable exceptions). But why?

The answer is two-fold: Julia is as fast as C++/Fortran, and the algorithms are what matter.

Julia is as Fast as C++/Fortran

While Julia code looks high level like Python or MATLAB, its performance is on par with C++ and Fortran. At a technical level, when Julia code is type-stable, i.e. that the types that are returned from a function are deducible at compile-time from the types that go into a function, then Julia can optimize it as much as C++ or Fortran by automatically devirtualizing all dynamic behavior and compile-time optimizing the quasi-static code. This is not an empirical statement, it's a provable type-theoretic result. The resulting compiler used on the resulting quasi-static representation is LLVM, the same optimizing compiler used by clang and LFortran.

For more details on how Julia code is optimized and how to optimize your own Julia code, check out this chapter from the SciML Book.

SciML's Julia Algorithms Have Performance Advantages in Many Common Regimes

There are many ways which Julia's algorithms achieve performance advantages. Some facts to highlight include:

Let's Dig Deep Into One Case: Adjoints of ODEs for Solving Inverse Problems

To really highlight how JIT compilation and automatic differentiation integration can change algorithms, let's look at the problem of differentiating an ODE solver. As is derived and discussed in detail at a seminar with the American Statistical Association, there are many ways to implement well-known “adjoint” methods which are required for performance. Each has different stability and performance trade-offs, and Julia's SciML is the only system to systemically offer all of the trade-off options. In many cases, using analytical adjoints of a solver is not advised due to performance reasons, with the trade-off described in detail here. Likewise, even when analytical adjoints are used, it turns out that for general nonlinear equations there is a trick which uses automatic differentiation in the construction of the analytical adjoint to improve its performance. As demonstrated in this publication, this can lead to about 2-3 orders of magnitude performance improvements. These AD-enhanced adjoints are showcased as the seeding methods in this plot:

Unless one directly defines special “vjp” functions, this is how the Julia SciML methods achieve orders of magnitude performance advantages over CVODES's adjoints and PETSC's TS-adjoint.

Moral of the story, even there are many reasons to use automatic differentiation of a solver, and even if an analytical adjoint rule is used for some specific performance reason, that analytical expression can often times be accelerated by orders of magnitude itself by embedding some form of automatic differentiation into it. This is just one algorithm of many which are optimized in this fashion.

diff --git a/dev/comparisons/matlab/index.html b/dev/comparisons/matlab/index.html index 20c5c96f318..41083561841 100644 --- a/dev/comparisons/matlab/index.html +++ b/dev/comparisons/matlab/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Getting Started with Julia's SciML for the MATLAB User

If you're a MATLAB user who has looked into Julia for some performance improvements, you may have noticed that the standard library does not have all of the “batteries” included with a base MATLAB installation. Where's the ODE solver? Where's fmincon and fsolve? Those scientific computing functionalities are the pieces provided by the Julia SciML ecosystem!

Why SciML? High-Level Workflow Reasons

  • Performance - The key reason people are moving from MATLAB to Julia's SciML in droves is performance. Even simple ODE solvers are much faster!, demonstrating orders of magnitude performance improvements for differential equations, nonlinear solving, optimization, and more. And the performance advantages continue to grow as more complex algorithms are required.
  • Julia is quick to learn from MATLAB - Most ODE codes can be translated in a few minutes. If you need help, check out the QuantEcon MATLAB-Python-Julia Cheat Sheet.
  • Package Management and Versioning - Julia's package manager takes care of dependency management, testing, and continuous delivery in order to make the installation and maintenance process smoother. For package users, this means it's easier to get packages with complex functionality in your hands.
  • Free and Open Source - If you want to know how things are being computed, just look at our GitHub organization. Lots of individuals use Julia's SciML to research how the algorithms actually work because of how accessible and tweakable the ecosystem is!
  • Composable Library Components - In MATLAB environments, every package feels like a silo. Functions made for one file exchange library cannot easily compose with another. SciML's generic coding with JIT compilation these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, MATLAB in orange represents MATLAB's most commonly used solvers:

Need a case study?

Check out this talk from NASA Scientists getting a 15,000x acceleration by switching from Simulink to Julia's ModelingToolkit!

Need Help Translating from MATLAB to Julia?

The following resources can be particularly helpful when adopting Julia for SciML for the first time:

MATLAB to Julia SciML Functionality Translations

The following chart will help you get quickly acquainted with Julia's SciML Tools:

MATLAB FunctionSciML-Supported Julia packages
plotPlots, Makie
sparseSparseArrays
interp1DataInterpolations
\, gmres, cgLinearSolve
fsolveNonlinearSolve
quadIntegrals
fminconOptimization
odeXXDifferentialEquations
ode45Tsit5
ode113VCABM
ode23sRosenbrock23
ode15sQNDF or FBDF
ode15iIDA
bvp4c and bvp5cDifferentialEquations
Simulink, SimscapeModelingToolkit
fftFFTW
chebfunApproxFun
+
Edit on GitHub

Getting Started with Julia's SciML for the MATLAB User

If you're a MATLAB user who has looked into Julia for some performance improvements, you may have noticed that the standard library does not have all of the “batteries” included with a base MATLAB installation. Where's the ODE solver? Where's fmincon and fsolve? Those scientific computing functionalities are the pieces provided by the Julia SciML ecosystem!

Why SciML? High-Level Workflow Reasons

  • Performance - The key reason people are moving from MATLAB to Julia's SciML in droves is performance. Even simple ODE solvers are much faster!, demonstrating orders of magnitude performance improvements for differential equations, nonlinear solving, optimization, and more. And the performance advantages continue to grow as more complex algorithms are required.
  • Julia is quick to learn from MATLAB - Most ODE codes can be translated in a few minutes. If you need help, check out the QuantEcon MATLAB-Python-Julia Cheat Sheet.
  • Package Management and Versioning - Julia's package manager takes care of dependency management, testing, and continuous delivery in order to make the installation and maintenance process smoother. For package users, this means it's easier to get packages with complex functionality in your hands.
  • Free and Open Source - If you want to know how things are being computed, just look at our GitHub organization. Lots of individuals use Julia's SciML to research how the algorithms actually work because of how accessible and tweakable the ecosystem is!
  • Composable Library Components - In MATLAB environments, every package feels like a silo. Functions made for one file exchange library cannot easily compose with another. SciML's generic coding with JIT compilation these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, MATLAB in orange represents MATLAB's most commonly used solvers:

Need a case study?

Check out this talk from NASA Scientists getting a 15,000x acceleration by switching from Simulink to Julia's ModelingToolkit!

Need Help Translating from MATLAB to Julia?

The following resources can be particularly helpful when adopting Julia for SciML for the first time:

MATLAB to Julia SciML Functionality Translations

The following chart will help you get quickly acquainted with Julia's SciML Tools:

MATLAB FunctionSciML-Supported Julia packages
plotPlots, Makie
sparseSparseArrays
interp1DataInterpolations
\, gmres, cgLinearSolve
fsolveNonlinearSolve
quadIntegrals
fminconOptimization
odeXXDifferentialEquations
ode45Tsit5
ode113VCABM
ode23sRosenbrock23
ode15sQNDF or FBDF
ode15iIDA
bvp4c and bvp5cDifferentialEquations
Simulink, SimscapeModelingToolkit
fftFFTW
chebfunApproxFun
diff --git a/dev/comparisons/python/index.html b/dev/comparisons/python/index.html index 9da59a5f5ba..3b0ac3bbefb 100644 --- a/dev/comparisons/python/index.html +++ b/dev/comparisons/python/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Getting Started with Julia's SciML for the Python User

If you're a Python user who has looked into Julia, you're probably wondering what is the equivalent to SciPy is. And you found it: it's the SciML ecosystem! To a Python developer, SciML is SciPy, but with the high-performance GPU, capabilities of PyTorch, and neural network capabilities, all baked right in. With SciML, there is no “separate world” of machine learning sublanguages: there is just one cohesive package ecosystem.

Why SciML? High-Level Workflow Reasons

  • Performance - The key reason people are moving from SciPy to Julia's SciML in droves is performance. Even simple ODE solvers are much faster!, demonstrating orders of magnitude performance improvements for differential equations, nonlinear solving, optimization, and more. And the performance advantages continue to grow as more complex algorithms are required.
  • Package Management and Versioning - Julia's package manager takes care of dependency management, testing, and continuous delivery in order to make the installation and maintenance process smoother. For package users, this means it's easier to get packages with complex functionality in your hands.
  • Composable Library Components - In Python environments, every package feels like a silo. Functions made for one file exchange library cannot easily compose with another. SciML's generic coding with JIT compilation these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, SciPy in yellow represents Python's most commonly used solvers:

Need Help Translating from Python to Julia?

The following resources can be particularly helpful when adopting Julia for SciML for the first time:

Python to Julia SciML Functionality Translations

The following chart will help you get quickly acquainted with Julia's SciML Tools:

Workflow ElementSciML-Supported Julia packages
MatplotlibPlots, Makie
scipy.specialSpecialFunctions
scipy.linalg.solveLinearSolve
scipy.integrateIntegrals
scipy.optimizeOptimization
scipy.optimize.fsolveNonlinearSolve
scipy.interpolateDataInterpolations
scipy.fftFFTW
scipy.linalgJulia's Built-In Linear Algebra
scipy.sparseSparseArrays, ARPACK
odeint/solve_ivpDifferentialEquations
scipy.integrate.solve_bvpBoundary-value problem
PyTorchFlux, Lux
gillespy2Catalyst, JumpProcesses
scipy.optimize.approx_fprimeFiniteDiff
autogradForwardDiff*, Enzyme*, DiffEqSensitivity
StanTuring
sympySymbolics

Why is Differentiable Programming Important for Scientific Computing?

Check out this blog post that goes into detail on how training neural networks in tandem with simulation improves performance by orders of magnitude. But can't you use analytical adjoint definitions? You can, but there are tricks to mix automatic differentiation into the adjoint definitions for a few orders of magnitude improvement too, as explained in this blog post.

These facts, along with many others, compose to algorithmic improvements with the implementation improvements, which leads to orders of magnitude improvements!

+
Edit on GitHub

Getting Started with Julia's SciML for the Python User

If you're a Python user who has looked into Julia, you're probably wondering what is the equivalent to SciPy is. And you found it: it's the SciML ecosystem! To a Python developer, SciML is SciPy, but with the high-performance GPU, capabilities of PyTorch, and neural network capabilities, all baked right in. With SciML, there is no “separate world” of machine learning sublanguages: there is just one cohesive package ecosystem.

Why SciML? High-Level Workflow Reasons

  • Performance - The key reason people are moving from SciPy to Julia's SciML in droves is performance. Even simple ODE solvers are much faster!, demonstrating orders of magnitude performance improvements for differential equations, nonlinear solving, optimization, and more. And the performance advantages continue to grow as more complex algorithms are required.
  • Package Management and Versioning - Julia's package manager takes care of dependency management, testing, and continuous delivery in order to make the installation and maintenance process smoother. For package users, this means it's easier to get packages with complex functionality in your hands.
  • Composable Library Components - In Python environments, every package feels like a silo. Functions made for one file exchange library cannot easily compose with another. SciML's generic coding with JIT compilation these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, SciPy in yellow represents Python's most commonly used solvers:

Need Help Translating from Python to Julia?

The following resources can be particularly helpful when adopting Julia for SciML for the first time:

Python to Julia SciML Functionality Translations

The following chart will help you get quickly acquainted with Julia's SciML Tools:

Workflow ElementSciML-Supported Julia packages
MatplotlibPlots, Makie
scipy.specialSpecialFunctions
scipy.linalg.solveLinearSolve
scipy.integrateIntegrals
scipy.optimizeOptimization
scipy.optimize.fsolveNonlinearSolve
scipy.interpolateDataInterpolations
scipy.fftFFTW
scipy.linalgJulia's Built-In Linear Algebra
scipy.sparseSparseArrays, ARPACK
odeint/solve_ivpDifferentialEquations
scipy.integrate.solve_bvpBoundary-value problem
PyTorchFlux, Lux
gillespy2Catalyst, JumpProcesses
scipy.optimize.approx_fprimeFiniteDiff
autogradForwardDiff*, Enzyme*, DiffEqSensitivity
StanTuring
sympySymbolics

Why is Differentiable Programming Important for Scientific Computing?

Check out this blog post that goes into detail on how training neural networks in tandem with simulation improves performance by orders of magnitude. But can't you use analytical adjoint definitions? You can, but there are tricks to mix automatic differentiation into the adjoint definitions for a few orders of magnitude improvement too, as explained in this blog post.

These facts, along with many others, compose to algorithmic improvements with the implementation improvements, which leads to orders of magnitude improvements!

diff --git a/dev/comparisons/r/index.html b/dev/comparisons/r/index.html index 4f216f7e411..73bd0fc5d84 100644 --- a/dev/comparisons/r/index.html +++ b/dev/comparisons/r/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Getting Started with Julia's SciML for the R User

If you're an R user who has looked into Julia, you're probably wondering where all of the scientific computing packages are. How do I solve ODEs? Solve f(x)=0 for x? Etc. SciML is the ecosystem for doing this with Julia.

Why SciML? High-Level Workflow Reasons

  • Performance - The key reason people are moving from R to Julia's SciML in droves is performance. Even simple ODE solvers are much faster!, demonstrating orders of magnitude performance improvements for differential equations, nonlinear solving, optimization, and more. And the performance advantages continue to grow as more complex algorithms are required.
  • Composable Library Components - In R environments, every package feels like a silo. Functions made for one file exchange library cannot easily compose with another. SciML's generic coding with JIT compilation these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • A Global Harmonious Documentation for Scientific Computing - R's documentation for scientific computing is scattered in a bunch of individual packages where the developers do not talk to each other! This not only leads to documentation differences, but also “style” differences: one package uses tol while the other uses atol. With Julia's SciML, the whole ecosystem is considered together, and inconsistencies are handled at the global level. The goal is to be working in one environment with one language.
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, deSolve in blue represents R's most commonly used solver:

Need Help Translating from R to Julia?

The following resources can be particularly helpful when adopting Julia for SciML for the first time:

R to Julia SciML Functionality Translations

The following chart will help you get quickly acquainted with Julia's SciML Tools:

R Function/PackageSciML-Supported Julia packages
data.frameDataFrames
plotPlots, Makie
ggplot2AlgebraOfGraphics
deSolveDifferentialEquations
StanTuring

Want to See the Power of Julia?

Check out this R-Bloggers blog post on diffeqr, a package which uses ModelingToolkit to translate R code to Julia, and achieves 350x acceleration over R's popular deSolve ODE solver package. But when the solve is done purely in Julia, it achieves 2777x acceleration over deSolve!

+
Edit on GitHub

Getting Started with Julia's SciML for the R User

If you're an R user who has looked into Julia, you're probably wondering where all of the scientific computing packages are. How do I solve ODEs? Solve f(x)=0 for x? Etc. SciML is the ecosystem for doing this with Julia.

Why SciML? High-Level Workflow Reasons

  • Performance - The key reason people are moving from R to Julia's SciML in droves is performance. Even simple ODE solvers are much faster!, demonstrating orders of magnitude performance improvements for differential equations, nonlinear solving, optimization, and more. And the performance advantages continue to grow as more complex algorithms are required.
  • Composable Library Components - In R environments, every package feels like a silo. Functions made for one file exchange library cannot easily compose with another. SciML's generic coding with JIT compilation these connections create new optimized code on the fly and allow for a more expansive feature set than can ever be documented. Take new high-precision number types from a package and stick them into a nonlinear solver. Take a package for Intel GPU arrays and stick it into the differential equation solver to use specialized hardware acceleration.
  • A Global Harmonious Documentation for Scientific Computing - R's documentation for scientific computing is scattered in a bunch of individual packages where the developers do not talk to each other! This not only leads to documentation differences, but also “style” differences: one package uses tol while the other uses atol. With Julia's SciML, the whole ecosystem is considered together, and inconsistencies are handled at the global level. The goal is to be working in one environment with one language.
  • Easier High-Performance and Parallel Computing - With Julia's ecosystem, CUDA will automatically install of the required binaries and cu(A)*cu(B) is then all that's required to GPU-accelerate large-scale linear algebra. MPI is easy to install and use. Distributed computing through password-less SSH. Multithreading is automatic and baked into many libraries, with a specialized algorithm to ensure hierarchical usage does not oversubscribe threads. Basically, libraries give you a lot of parallelism for free, and doing the rest is a piece of cake.
  • Mix Scientific Computing with Machine Learning - Want to automate the discovery of missing physical laws using neural networks embedded in differentiable simulations? Julia's SciML is the ecosystem with the tooling to integrate machine learning into the traditional high-performance scientific computing domains, from multiphysics simulations to partial differential equations.

In this plot, deSolve in blue represents R's most commonly used solver:

Need Help Translating from R to Julia?

The following resources can be particularly helpful when adopting Julia for SciML for the first time:

R to Julia SciML Functionality Translations

The following chart will help you get quickly acquainted with Julia's SciML Tools:

R Function/PackageSciML-Supported Julia packages
data.frameDataFrames
plotPlots, Makie
ggplot2AlgebraOfGraphics
deSolveDifferentialEquations
StanTuring

Want to See the Power of Julia?

Check out this R-Bloggers blog post on diffeqr, a package which uses ModelingToolkit to translate R code to Julia, and achieves 350x acceleration over R's popular deSolve ODE solver package. But when the solve is done purely in Julia, it achieves 2777x acceleration over deSolve!

diff --git a/dev/getting_started/find_root/index.html b/dev/getting_started/find_root/index.html index 75b273b6964..19a2480e478 100644 --- a/dev/getting_started/find_root/index.html +++ b/dev/getting_started/find_root/index.html @@ -73,4 +73,4 @@ 0.0 0.0 0.0

Step 5: Analyze the Solution

Now let's check out the solution. First of all, what kind of thing is the sol? We can see that by asking for its type:

typeof(sol)
SciMLBase.NonlinearSolution{Float64, 1, Vector{Float64}, Vector{Float64}, SciMLBase.NonlinearProblem{Vector{Float64}, true, Vector{Float64}, SciMLBase.NonlinearFunction{true, SciMLBase.FullSpecialize, ModelingToolkit.var"#f#733"{RuntimeGeneratedFunctions.RuntimeGeneratedFunction{(:ˍ₋arg1, :ˍ₋arg2), ModelingToolkit.var"#_RGF_ModTag", ModelingToolkit.var"#_RGF_ModTag", (0x31381c72, 0xc57f609d, 0x303e44f7, 0x142ed376, 0x9b81df74), Nothing}, RuntimeGeneratedFunctions.RuntimeGeneratedFunction{(:ˍ₋out, :ˍ₋arg1, :ˍ₋arg2), ModelingToolkit.var"#_RGF_ModTag", ModelingToolkit.var"#_RGF_ModTag", (0x071f1454, 0x9075ad55, 0xc8a6d474, 0xdb5593a1, 0xf697d5cb), Nothing}}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Vector{Symbol}, Vector{Symbol}, ModelingToolkit.var"#generated_observed#736"{ModelingToolkit.NonlinearSystem, Dict{Any, Any}}, Nothing, ModelingToolkit.NonlinearSystem, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardNonlinearProblem}, NonlinearSolve.NewtonRaphson{0, true, Val{:forward}, Nothing, typeof(NonlinearSolve.DEFAULT_PRECS), true, nothing}, Nothing, Nothing, SciMLBase.NLStats}

From this, we can see that it is an NonlinearSolution. We can see the documentation for how to use the NonlinearSolution by checking the NonlinearSolve.jl solution type page. For example, the solution is stored as .u. What is the solution to our nonlinear system, and what is the final residual value? We can check it as follows:

# Analyze the solution
-@show sol.u, sol.resid
([0.0, 0.0, 0.0], [0.0, 0.0, 0.0])
+@show sol.u, sol.resid
([0.0, 0.0, 0.0], [0.0, 0.0, 0.0])
diff --git a/dev/getting_started/first_optimization/index.html b/dev/getting_started/first_optimization/index.html index 0fe0d8d554c..54318c93686 100644 --- a/dev/getting_started/first_optimization/index.html +++ b/dev/getting_started/first_optimization/index.html @@ -32,4 +32,4 @@ sol = solve(prob, NLopt.LD_LBFGS())
u: 2-element Vector{Float64}:
  1.0
  1.0

Step 4: Analyze the Solution

Now let's check out the solution. First of all, what kind of thing is the sol? We can see that by asking for its type:

typeof(sol)
SciMLBase.OptimizationSolution{Float64, 1, Vector{Float64}, Optimization.OptimizationCache{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), OptimizationForwardDiffExt.var"#38#56"{ForwardDiff.GradientConfig{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 2, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 2}}}, OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}}, OptimizationForwardDiffExt.var"#41#59"{ForwardDiff.HessianConfig{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 2, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Float64}, ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 2}, 2}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Float64}, Float64, 2}}}, OptimizationForwardDiffExt.var"#37#55"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}}, OptimizationForwardDiffExt.var"#44#62", Nothing, OptimizationForwardDiffExt.var"#48#66"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, OptimizationForwardDiffExt.var"#53#71"{SciMLBase.OptimizationFunction{true, ADTypes.AutoForwardDiff{nothing, Nothing}, typeof(Main.L), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Optimization.ReInitCache{Vector{Float64}, Vector{Float64}}, Vector{Float64}, Vector{Float64}, Nothing, Nothing, Nothing, NLopt.Algorithm, Base.Iterators.Cycle{Tuple{Optimization.NullData}}, Bool, Optimization.var"#41#43"}, NLopt.Algorithm, Float64, NLopt.Opt, Float64, Nothing}

From this, we can see that it is an OptimizationSolution. We can see the documentation for how to use the OptimizationSolution by checking the Optimization.jl solution type page. For example, the solution is stored as .u. What is the solution to our optimization, and what is the final loss value? We can check it as follows:

# Analyze the solution
-@show sol.u, L(sol.u, p)
([1.0, 1.0], 0.0)
+@show sol.u, L(sol.u, p)
([1.0, 1.0], 0.0)
diff --git a/dev/getting_started/first_simulation/index.html b/dev/getting_started/first_simulation/index.html index f23faf9917c..67fc5afdad6 100644 --- a/dev/getting_started/first_simulation/index.html +++ b/dev/getting_started/first_simulation/index.html @@ -42,89 +42,89 @@ plot(p1, p2, layout = (2, 1)) - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + +

Step-by-Step Solution

Step 1: Install and Import the Required Packages

To do this tutorial, we will need a few components:

To start, let's add these packages as demonstrated in the installation tutorial:

using Pkg
 Pkg.add(["ModelingToolkit", "DifferentialEquations", "Plots"])

Now we're ready. Let's load in these packages:

using ModelingToolkit, DifferentialEquations, Plots

Step 2: Define our ODE Equations

Now let's define our ODEs. We use the ModelingToolkit.@variabes statement to declare our variables. We have the independent variable time t, and then define our 3 state variables:

# Define our state variables: state(t) = initial condition
 @variables t x(t)=1 y(t)=1 z(t)=2

\[ \begin{equation} @@ -245,189 +245,189 @@

- + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 
p2 = plot(sol, idxs = z, title = "Total Animals")
- + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + +

Finally, let's make a plot where we merge these two plot elements. To do so, we can take our two plot objects, p1 and p2, and make a plot with both of them. Then we tell Plots to do a layout of (2,1), or 2 rows and 1 columns. Let's see what happens when we bring these together:

plot(p1, p2, layout = (2, 1))
- + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + +

And tada, we have a full analysis of our ecosystem!

Bonus Step: Emoji Variables

If you made it this far, then congrats, you get to learn a fun fact! Since Julia code can use Unicode, emojis work for variable names. Here's the simulation using emojis of rabbits and wolves to define the system:

using ModelingToolkit, DifferentialEquations
 @parameters α=1.5 β=1.0 γ=3.0 δ=1.0
 @variables t 🐰(t)=1.0 🐺(t)=1.0
@@ -481,4 +481,4 @@ 
« Installing SciML SoftwareSolve your first optimization problem »
Powered by Documenter.jl and the Julia Programming Language.

+ [1.0337581256020607, 0.9063703842886133]

Now go make your professor mad that they have to grade a fully emojified code. I'll vouch for you: the documentation told you to do this.

diff --git a/dev/getting_started/fit_simulation/index.html b/dev/getting_started/fit_simulation/index.html index b59fd25588d..0fa38d3af4f 100644 --- a/dev/getting_started/fit_simulation/index.html +++ b/dev/getting_started/fit_simulation/index.html @@ -140,4 +140,4 @@ 1.499584010680642 0.9996974030659543 3.0020191064934587 - 1.0012321222850014

and the answer from the optimization is our desired parameters.

+ 1.0012321222850014

and the answer from the optimization is our desired parameters.

diff --git a/dev/getting_started/getting_started/index.html b/dev/getting_started/getting_started/index.html index f05e1561bbd..e59cc461087 100644 --- a/dev/getting_started/getting_started/index.html +++ b/dev/getting_started/getting_started/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Getting Started with Julia's SciML

Quickly: What is Julia's SciML Ecosystem?

Julia's SciML is:

  • SciPy or MATLAB's standard library but in Julia, but
  • Runs orders of magnitude faster, even outperforms C and Fortran libraries, and
  • Is fully compatible with machine learning and automatic differentiation,
  • All while having an easy-to-use high level interactive development environment.

Interested?

Introductory Tutorials

Note

Each of the SciML packages starts with its own introductory tutorial as well! Once you have started to get the hang of a few things, start checking out the introductory tutorials of the different packages. For example, the DifferentialEquations.jl getting started tutorial is a fun one!

Coming from...

Are you familiar with other scientific computing tools? Take a look at the guided introductions below.

+
Edit on GitHub

Getting Started with Julia's SciML

Quickly: What is Julia's SciML Ecosystem?

Julia's SciML is:

  • SciPy or MATLAB's standard library but in Julia, but
  • Runs orders of magnitude faster, even outperforms C and Fortran libraries, and
  • Is fully compatible with machine learning and automatic differentiation,
  • All while having an easy-to-use high level interactive development environment.

Interested?

Introductory Tutorials

Note

Each of the SciML packages starts with its own introductory tutorial as well! Once you have started to get the hang of a few things, start checking out the introductory tutorials of the different packages. For example, the DifferentialEquations.jl getting started tutorial is a fun one!

Coming from...

Are you familiar with other scientific computing tools? Take a look at the guided introductions below.

diff --git a/dev/getting_started/installation/index.html b/dev/getting_started/installation/index.html index e8abe4575cd..dbab79d034f 100644 --- a/dev/getting_started/installation/index.html +++ b/dev/getting_started/installation/index.html @@ -4,4 +4,4 @@ gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash});
Edit on GitHub

Installing SciML Software

Step 1: Install Julia

Download Julia using this website.

Note

Some Linux distributions do weird and incorrect things with Julia installations! Please install Julia using the binaries provided by the official JuliaLang website!

To ensure that you have installed Julia correctly, open it up and type versioninfo() in the REPL. It should look like the following:

(with the CPU/OS/etc. details matching your computer!)

If you got stuck in this installation process, ask for help on the Julia Discourse or in the Julia Zulip chatrooms

Optional Step 1.5: Get VS Code Setup with the Julia Extension

You can run SciML with Julia in any development environment you please, but our recommended environment is VS Code. For more information on using Julia with VS Code, check out the Julia VS Code Extension website. Let's install it!

First download VS Code from the official website.

Next, open Visual Studio Code and click Extensions.

Then, search for “Julia” in the search bar on the top of the extension tab, click on the “Julia” extension, and click the install button on the tab that opens up.

To make sure your installation is correct, try running some code. Open a new file by either going to the top left navigation bar File |> New Text File, or hitting Ctrl+n. Name your new file test.jl (important: the Julia VS Code functionality only turns on when using a .jl file!). Next, type 1+1 and hit Ctrl+Enter. A Julia REPL should pop up and the result 2 should be displayed. Your environment should look something like this:

For more help on using the VS Code editor with Julia, check out the VS Code in Julia documentation. Useful keyboard commands can be found here.

Once again, if you got stuck in this installation process, ask for help on the Julia Discourse or in the Julia Zulip chatrooms

Step 2: Install a SciML Package

SciML is over 130 Julia packages. That's too much stuff to give someone in a single download! Thus instead, the SciML organization divides its functionality into composable modules that can be mixed and matched as required. Installing SciML ecosystem functionality is equivalent to installation of such packages.

For example, do you need the differential equation solver? Then install DifferentialEquations via the command:

using Pkg;
-Pkg.add("DifferentialEquations");

in the Julia REPL. Or, for a more robust REPL experience, hit the ] command to make the blue pkg> REPL environment start, and type in add DifferentialEquations. The package REPL environment will have nice extras like auto-complete that will be useful in the future. This command should run an installation sequence and precompile all of the packages (precompile = "run a bunch of performance optimizations!"). Don't be surprised if this installation process takes ~10 minutes on older computers. During the installation, it should look like this:

And that's it!

How do I test that my installed correctly?

The best way is to build and run your first simulation!

+Pkg.add("DifferentialEquations");

in the Julia REPL. Or, for a more robust REPL experience, hit the ] command to make the blue pkg> REPL environment start, and type in add DifferentialEquations. The package REPL environment will have nice extras like auto-complete that will be useful in the future. This command should run an installation sequence and precompile all of the packages (precompile = "run a bunch of performance optimizations!"). Don't be surprised if this installation process takes ~10 minutes on older computers. During the installation, it should look like this:

And that's it!

How do I test that my installed correctly?

The best way is to build and run your first simulation!

diff --git a/dev/highlevels/array_libraries/index.html b/dev/highlevels/array_libraries/index.html index aeb11c38041..fb1c93f3ea3 100644 --- a/dev/highlevels/array_libraries/index.html +++ b/dev/highlevels/array_libraries/index.html @@ -26,4 +26,4 @@ lorenz_p = (σ = 10.0, ρ = 28.0, β = 8 / 3) lorenz_ic = ComponentArray(x = 0.0, y = 0.0, z = 0.0) -lorenz_prob = ODEProblem(lorenz!, lorenz_ic, tspan, lorenz_p)

Is that beautiful? Yes, it is.

StaticArrays.jl: Statically-Defined Arrays

StaticArrays.jl is a library for statically-defined arrays. Because these arrays have type-level information for size, they recompile the solvers for every new size. They can be dramatically faster for small sizes (up to approximately size 10), but for larger equations they increase compile time with little to no benefit.

CUDA.jl: NVIDIA CUDA-Based GPU Array Computations

CUDA.jl is the library for defining arrays which live on NVIDIA GPUs (CuArray). SciML's libraries will respect the GPU-ness of the inputs, i.e., if the input arrays live on the GPU then the operations will all take place on the GPU or else the libraries will error if it's unable to do so. Thus, using CUDA.jl's CuArray is how one GPU-accelerates any computation with the SciML organization's libraries. Simply use a CuArray as the initial condition to an ODE solve or as the initial guess for a nonlinear solve, and the whole solve will recompile to take place on the GPU.

AMDGPU.jl: AMD-Based GPU Array Computations

AMDGPU.jl is the library for defining arrays which live on AMD GPUs (ROCArray). SciML's libraries will respect the GPU-ness of the inputs, i.e., if the input arrays live on the GPU then the operations will all take place on the GPU or else the libraries will error if it's unable to do so. Thus using AMDGPU.jl's ROCArray is how one GPU-accelerates any computation with the SciML organization's libraries. Simply use a ROCArray as the initial condition to an ODE solve or as the initial guess for a nonlinear solve, and the whole solve will recompile to take place on the GPU.

FillArrays.jl: Lazy Arrays

FillArrays.jl is a library for defining arrays with lazy values. For example, an O(1) representation of the identity matrix is given by Eye{Int}(5). FillArrays.jl is used extensively throughout the ecosystem to improve runtime and memory performance.

BandedMatrices.jl: Fast Banded Matrices

Banded matrices show up in many equation solver contexts, such as the Jacobians of many partial differential equations. While the base SparseMatrixCSC sparse matrix type can represent such matrices, BandedMatrices.jl is a specialized format specifically for BandedMatrices which can be used to greatly improve performance of operations on a banded matrix.

BlockBandedMatrices.jl: Fast Block-Banded Matrices

Block banded matrices show up in many equation solver contexts, such as the Jacobians of many systems of partial differential equations. While the base SparseMatrixCSC sparse matrix type can represent such matrices, BlockBandedMatrices.jl is a specialized format specifically for BlockBandedMatrices which can be used to greatly improve performance of operations on a block-banded matrix.

+lorenz_prob = ODEProblem(lorenz!, lorenz_ic, tspan, lorenz_p)

Is that beautiful? Yes, it is.

StaticArrays.jl: Statically-Defined Arrays

StaticArrays.jl is a library for statically-defined arrays. Because these arrays have type-level information for size, they recompile the solvers for every new size. They can be dramatically faster for small sizes (up to approximately size 10), but for larger equations they increase compile time with little to no benefit.

CUDA.jl: NVIDIA CUDA-Based GPU Array Computations

CUDA.jl is the library for defining arrays which live on NVIDIA GPUs (CuArray). SciML's libraries will respect the GPU-ness of the inputs, i.e., if the input arrays live on the GPU then the operations will all take place on the GPU or else the libraries will error if it's unable to do so. Thus, using CUDA.jl's CuArray is how one GPU-accelerates any computation with the SciML organization's libraries. Simply use a CuArray as the initial condition to an ODE solve or as the initial guess for a nonlinear solve, and the whole solve will recompile to take place on the GPU.

AMDGPU.jl: AMD-Based GPU Array Computations

AMDGPU.jl is the library for defining arrays which live on AMD GPUs (ROCArray). SciML's libraries will respect the GPU-ness of the inputs, i.e., if the input arrays live on the GPU then the operations will all take place on the GPU or else the libraries will error if it's unable to do so. Thus using AMDGPU.jl's ROCArray is how one GPU-accelerates any computation with the SciML organization's libraries. Simply use a ROCArray as the initial condition to an ODE solve or as the initial guess for a nonlinear solve, and the whole solve will recompile to take place on the GPU.

FillArrays.jl: Lazy Arrays

FillArrays.jl is a library for defining arrays with lazy values. For example, an O(1) representation of the identity matrix is given by Eye{Int}(5). FillArrays.jl is used extensively throughout the ecosystem to improve runtime and memory performance.

BandedMatrices.jl: Fast Banded Matrices

Banded matrices show up in many equation solver contexts, such as the Jacobians of many partial differential equations. While the base SparseMatrixCSC sparse matrix type can represent such matrices, BandedMatrices.jl is a specialized format specifically for BandedMatrices which can be used to greatly improve performance of operations on a banded matrix.

BlockBandedMatrices.jl: Fast Block-Banded Matrices

Block banded matrices show up in many equation solver contexts, such as the Jacobians of many systems of partial differential equations. While the base SparseMatrixCSC sparse matrix type can represent such matrices, BlockBandedMatrices.jl is a specialized format specifically for BlockBandedMatrices which can be used to greatly improve performance of operations on a block-banded matrix.

diff --git a/dev/highlevels/developer_documentation/index.html b/dev/highlevels/developer_documentation/index.html index b0dcd77c3c4..fe184993bcc 100644 --- a/dev/highlevels/developer_documentation/index.html +++ b/dev/highlevels/developer_documentation/index.html @@ -4,4 +4,4 @@ gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash});
Edit on GitHub

Developer Documentation

For uniformity and clarity, the SciML Open-Source Software Organization has many well-defined rules and practices for its development. However, we stress one important principle:

Do not be deterred from contributing if you think you do not know everything. No one knows everything. These rules and styles are designed for iterative contributions. Open pull requests and contribute what you can with what you know, and the maintainers will help you learn and do the rest!

If you need any help contributing, please feel welcome joining our community channels.

We welcome everybody.

Getting Started With Contributing to SciML

To get started contributing to SciML, check out the following resources:

SciMLStyle: The SciML Style Guide for Julia

SciML Code Style

This is a style guide for how to program in Julia for SciML contributions. It describes everything one needs to know, from preferred naming schemes of functions to fundamental dogmas for designing traits. We stress that this style guide is meant to be comprehensive for the sake of designing automatic formatters and teaching desired rules, but complete knowledge and adherence to the style guide is not required for contributions!

COLPRAC: Contributor's Guide on Collaborative Practices for Community Packages

ColPrac: Contributor's Guide on Collaborative Practices for Community Packages

What are the rules for when PRs should be merged? What are the rules for whether to tag a major, minor, or patch release? All of these development rules are defined in COLPRAC.

DiffEq Developer Documentation

There are many solver libraries which share similar internals, such as OrdinaryDiffEq.jl, StochasticDiffEq.jl, and DelayDiffEq.jl. This section of the documentation describes the internal systems of these packages and how they are used to quickly write efficient solvers.

Third-Party Libraries to Note

Documenter.jl

Documenter.jl is the documentation generation library that the SciML organization uses, and thus its documentation is the documentation of the documentation.

JuliaFormatter.jl

JuliaFormatter.jl is the formatter used by the SciML organization to enforce the SciML Style. Setting style = "sciml" in a .JuliaFormatter.toml file of a repo and using the standard FormatCheck.yml as part of continuous integration makes JuliaFormatter check for SciML Style compliance on pull requests.

To run JuliaFormatter in a SciML repository, do:

using JuliaFomatter, DevedPackage
-JuliaFormatter.format(pkgdir(DevedPackage))

which will reformat the code according to the SciML Style.

GitHub Actions Continuous Integrations

The SciML Organization uses continuous integration testing to always ensure tests are passing when merging pull requests. The organization uses the GitHub Actions supplied by Julia Actions to accomplish this. Common continuous integration scripts are:

  • CI.yml, the standard CI script
  • Downstream.yml, used to specify packages for downstream testing. This will make packages which depend on the current package also be tested to ensure that “non-breaking changes” do not actually break other packages.
  • Documentation.yml, used to run the documentation automatic generation with Documenter.jl
  • FormatCheck.yml, used to check JuliaFormatter SciML Style compliance

CompatHelper

CompatHelper is used to automatically create pull requests whenever a dependent package is upper bounded. The results of CompatHelper PRs should be checked to ensure that the latest version of the dependencies are grabbed for the test process. After successful CompatHelper PRs, i.e. if the increase of the upper bound did not cause a break to the tests, a new version tag should follow. It is set up by adding the CompatHelper.yml GitHub action.

TagBot

TagBot automatically creates tags in the GitHub repository whenever a package is registered to the Julia General repository. It is set up by adding the TagBot.yml GitHub action.

+JuliaFormatter.format(pkgdir(DevedPackage))

which will reformat the code according to the SciML Style.

GitHub Actions Continuous Integrations

The SciML Organization uses continuous integration testing to always ensure tests are passing when merging pull requests. The organization uses the GitHub Actions supplied by Julia Actions to accomplish this. Common continuous integration scripts are:

  • CI.yml, the standard CI script
  • Downstream.yml, used to specify packages for downstream testing. This will make packages which depend on the current package also be tested to ensure that “non-breaking changes” do not actually break other packages.
  • Documentation.yml, used to run the documentation automatic generation with Documenter.jl
  • FormatCheck.yml, used to check JuliaFormatter SciML Style compliance

CompatHelper

CompatHelper is used to automatically create pull requests whenever a dependent package is upper bounded. The results of CompatHelper PRs should be checked to ensure that the latest version of the dependencies are grabbed for the test process. After successful CompatHelper PRs, i.e. if the increase of the upper bound did not cause a break to the tests, a new version tag should follow. It is set up by adding the CompatHelper.yml GitHub action.

TagBot

TagBot automatically creates tags in the GitHub repository whenever a package is registered to the Julia General repository. It is set up by adding the TagBot.yml GitHub action.

diff --git a/dev/highlevels/equation_solvers/index.html b/dev/highlevels/equation_solvers/index.html index 47bcb860573..64c8628e93e 100644 --- a/dev/highlevels/equation_solvers/index.html +++ b/dev/highlevels/equation_solvers/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Equation Solvers

The SciML Equation Solvers cover a large set of SciMLProblems with SciMLAlgorithms that are efficient, numerically stable, and flexible. These methods tie into libraries like SciMLSensitivity.jl to be fully differentiable and compatible with machine learning pipelines, and are designed for integration with applications like parameter estimation, global sensitivity analysis, and more.

LinearSolve.jl: Unified Interface for Linear Solvers

LinearSolve.jl is the canonical library for solving LinearProblems. It includes:

  • Fast pure Julia LU factorizations which outperform standard BLAS
  • KLU for faster sparse LU factorization on unstructured matrices
  • UMFPACK for faster sparse LU factorization on matrices with some repeated structure
  • MKLPardiso wrappers for handling many sparse matrices faster than SuiteSparse (KLU, UMFPACK) methods
  • GPU-offloading for large dense matrices
  • Wrappers to all of the Krylov implementations (Krylov.jl, IterativeSolvers.jl, KrylovKit.jl) for easy testing of all of them. LinearSolve.jl handles the API differences, especially with the preconditioner definitions
  • A polyalgorithm that smartly chooses between these methods
  • A caching interface which automates caching of symbolic factorizations and numerical factorizations as optimally as possible
  • Compatible with arbitrary AbstractArray and Number types, such as GPU-based arrays, uncertainty quantification number types, and more.

NonlinearSolve.jl: Unified Interface for Nonlinear Solvers

NonlinearSolve.jl is the canonical library for solving NonlinearProblems. It includes:

  • Fast non-allocating implementations on static arrays of common methods (Newton-Rhapson)
  • Bracketing methods (Bisection, Falsi) for methods with known upper and lower bounds (IntervalNonlinearProblem)
  • Wrappers to common other solvers (NLsolve.jl, MINPACK, KINSOL from Sundials) for trust region methods, line search-based approaches, etc.
  • Built over the LinearSolve.jl API for maximum flexibility and performance in the solving approach
  • Compatible with arbitrary AbstractArray and Number types, such as GPU-based arrays, uncertainty quantification number types, and more.

DifferentialEquations.jl: Unified Interface for Differential Equation Solvers

DifferentialEquations.jl is the canonical library for solving DEProblems. This includes:

  • Discrete equations (function maps, discrete stochastic (Gillespie/Markov) simulations) (DiscreteProblem)
  • Ordinary differential equations (ODEs) (ODEProblem)
  • Split and Partitioned ODEs (Symplectic integrators, IMEX Methods) (SplitODEProblem)
  • Stochastic ordinary differential equations (SODEs or SDEs) (SDEProblem)
  • Stochastic differential-algebraic equations (SDAEs) (SDEProblem with mass matrices)
  • Random differential equations (RODEs or RDEs) (RODEProblem)
  • Differential algebraic equations (DAEs) (DAEProblem and ODEProblem with mass matrices)
  • Delay differential equations (DDEs) (DDEProblem)
  • Neutral, retarded, and algebraic delay differential equations (NDDEs, RDDEs, and DDAEs)
  • Stochastic delay differential equations (SDDEs) (SDDEProblem)
  • Experimental support for stochastic neutral, retarded, and algebraic delay differential equations (SNDDEs, SRDDEs, and SDDAEs)
  • Mixed discrete and continuous equations (Hybrid Equations, Jump Diffusions) (DEProblems with callbacks and JumpProblem)

The well-optimized DifferentialEquations solvers benchmark as some of the fastest implementations of classic algorithms. It also includes algorithms from recent research which routinely outperform the “standard” C/Fortran methods, and algorithms optimized for high-precision and HPC applications. Simultaneously, it wraps the classic C/Fortran methods, making it easy to switch over to them whenever necessary. Solving differential equations with different methods from different languages and packages can be done by changing one line of code, allowing for easy benchmarking to ensure you are using the fastest method possible.

DifferentialEquations.jl integrates with the Julia package sphere. Examples are:

  • GPU acceleration through CUDAnative.jl and CuArrays.jl
  • Automated sparsity detection with Symbolics.jl
  • Automatic Jacobian coloring with SparseDiffTools.jl, allowing for fast solutions to problems with sparse or structured (Tridiagonal, Banded, BlockBanded, etc.) Jacobians
  • Allowing the specification of linear solvers for maximal efficiency
  • Progress meter integration with the Juno IDE for estimated time to solution
  • Automatic plotting of time series and phase plots
  • Built-in interpolations
  • Wraps for common C/Fortran methods, like Sundials and Hairer's radau
  • Arbitrary precision with BigFloats and Arbfloats
  • Arbitrary array types, allowing the definition of differential equations on matrices and distributed arrays
  • Unit-checked arithmetic with Unitful

Optimization.jl: Unified Interface for Optimization

Optimization.jl is the canonical library for solving OptimizationProblems. It includes wrappers of most of the Julia nonlinear optimization ecosystem, allowing one syntax to use all packages in a uniform manner. This covers:

Integrals.jl: Unified Interface for Numerical Integration

Integrals.jl is the canonical library for solving IntegralsProblems. It includes wrappers of most of the Julia quadrature ecosystem, allowing one syntax to use all packages in a uniform manner. This covers:

  • Gauss-Kronrod quadrature
  • Cubature methods (both h and p cubature)
  • Adaptive Monte Carlo methods

JumpProcesses.jl: Stochastic Simulation Algorithms for Jump Processes, Jump-ODEs, and Jump-Diffusions

JumpProcesses.jl is the library for Poisson jump processes, also known as chemical master equations or Gillespie simulations, for simulating chemical reaction networks and other applications. It allows for solving with many methods, including:

  • Direct: the Gillespie Direct method SSA.
  • RDirect: A variant of Gillespie's Direct method that uses rejection to sample the next reaction.
  • DirectCR: The Composition-Rejection Direct method of Slepoy et al. For large networks and linear chain-type networks, it will often give better performance than Direct. (Requires dependency graph, see below.)
  • DirectFW: the Gillespie Direct method SSA with FunctionWrappers. This aggregator uses a different internal storage format for collections of ConstantRateJumps.
  • FRM: the Gillespie first reaction method SSA. Direct should generally offer better performance and be preferred to FRM.
  • FRMFW: the Gillespie first reaction method SSA with FunctionWrappers.
  • NRM: The Gibson-Bruck Next Reaction Method. For some reaction network structures, this may offer better performance than Direct (for example, large, linear chains of reactions). (Requires dependency graph, see below.)
  • RSSA: The Rejection SSA (RSSA) method of Thanh et al. With RSSACR, for very large reaction networks, it often offers the best performance of all methods. (Requires dependency graph, see below.)
  • RSSACR: The Rejection SSA (RSSA) with Composition-Rejection method of Thanh et al. With RSSA, for very large reaction networks, it often offers the best performance of all methods. (Requires dependency graph, see below.)
  • SortingDirect: The Sorting Direct Method of McCollum et al. It will usually offer performance as good as Direct, and for some systems can offer substantially better performance. (Requires dependency graph, see below.)

The design of JumpProcesses.jl composes with DifferentialEquations.jl, allowing for discrete stochastic chemical reactions to be easily mixed with differential equation models, allowing for simulation of hybrid systems, jump diffusions, and differential equations driven by Levy processes.

In addition, JumpProcesses's interfaces allow for solving with regular jump methods, such as adaptive Tau-Leaping.

Third-Party Libraries to Note

JuMP.jl: Julia for Mathematical Programming

While Optimization.jl is the preferred library for nonlinear optimization, for all other forms of optimization Julia for Mathematical Programming (JuMP) is the star. JuMP is the leading choice in Julia for doing:

  • Linear Programming
  • Quadratic Programming
  • Convex Programming
  • Conic Programming
  • Semidefinite Programming
  • Mixed-Complementarity Programming
  • Integer Programming
  • Mixed Integer (nonlinear/linear) Programming
  • (Mixed Integer) Second Order Conic Programming

JuMP can also be used for some nonlinear programming, though the Optimization.jl bindings to the JuMP solvers (via MathOptInterface.jl) is generally preferred.

FractionalDiffEq.jl: Fractional Differential Equation Solvers

FractionalDiffEq.jl is a set of high-performance solvers for fractional differential equations.

ManifoldDiffEq.jl: Solvers for Differential Equations on Manifolds

ManifoldDiffEq.jl is a set of high-performance solvers for differential equations on manifolds using methods such as Lie Group actions and frozen coefficients (Crouch-Grossman methods). These solvers can in many cases out-perform the OrdinaryDiffEq.jl nonautonomous operator ODE solvers by using methods specialized on manifold definitions of ManifoldsBase.

Manopt.jl: Optimization on Manifolds

ManOpt.jl allows for easy and efficient solving of nonlinear optimization problems on manifolds.

+
Edit on GitHub

Equation Solvers

The SciML Equation Solvers cover a large set of SciMLProblems with SciMLAlgorithms that are efficient, numerically stable, and flexible. These methods tie into libraries like SciMLSensitivity.jl to be fully differentiable and compatible with machine learning pipelines, and are designed for integration with applications like parameter estimation, global sensitivity analysis, and more.

LinearSolve.jl: Unified Interface for Linear Solvers

LinearSolve.jl is the canonical library for solving LinearProblems. It includes:

  • Fast pure Julia LU factorizations which outperform standard BLAS
  • KLU for faster sparse LU factorization on unstructured matrices
  • UMFPACK for faster sparse LU factorization on matrices with some repeated structure
  • MKLPardiso wrappers for handling many sparse matrices faster than SuiteSparse (KLU, UMFPACK) methods
  • GPU-offloading for large dense matrices
  • Wrappers to all of the Krylov implementations (Krylov.jl, IterativeSolvers.jl, KrylovKit.jl) for easy testing of all of them. LinearSolve.jl handles the API differences, especially with the preconditioner definitions
  • A polyalgorithm that smartly chooses between these methods
  • A caching interface which automates caching of symbolic factorizations and numerical factorizations as optimally as possible
  • Compatible with arbitrary AbstractArray and Number types, such as GPU-based arrays, uncertainty quantification number types, and more.

NonlinearSolve.jl: Unified Interface for Nonlinear Solvers

NonlinearSolve.jl is the canonical library for solving NonlinearProblems. It includes:

  • Fast non-allocating implementations on static arrays of common methods (Newton-Rhapson)
  • Bracketing methods (Bisection, Falsi) for methods with known upper and lower bounds (IntervalNonlinearProblem)
  • Wrappers to common other solvers (NLsolve.jl, MINPACK, KINSOL from Sundials) for trust region methods, line search-based approaches, etc.
  • Built over the LinearSolve.jl API for maximum flexibility and performance in the solving approach
  • Compatible with arbitrary AbstractArray and Number types, such as GPU-based arrays, uncertainty quantification number types, and more.

DifferentialEquations.jl: Unified Interface for Differential Equation Solvers

DifferentialEquations.jl is the canonical library for solving DEProblems. This includes:

  • Discrete equations (function maps, discrete stochastic (Gillespie/Markov) simulations) (DiscreteProblem)
  • Ordinary differential equations (ODEs) (ODEProblem)
  • Split and Partitioned ODEs (Symplectic integrators, IMEX Methods) (SplitODEProblem)
  • Stochastic ordinary differential equations (SODEs or SDEs) (SDEProblem)
  • Stochastic differential-algebraic equations (SDAEs) (SDEProblem with mass matrices)
  • Random differential equations (RODEs or RDEs) (RODEProblem)
  • Differential algebraic equations (DAEs) (DAEProblem and ODEProblem with mass matrices)
  • Delay differential equations (DDEs) (DDEProblem)
  • Neutral, retarded, and algebraic delay differential equations (NDDEs, RDDEs, and DDAEs)
  • Stochastic delay differential equations (SDDEs) (SDDEProblem)
  • Experimental support for stochastic neutral, retarded, and algebraic delay differential equations (SNDDEs, SRDDEs, and SDDAEs)
  • Mixed discrete and continuous equations (Hybrid Equations, Jump Diffusions) (DEProblems with callbacks and JumpProblem)

The well-optimized DifferentialEquations solvers benchmark as some of the fastest implementations of classic algorithms. It also includes algorithms from recent research which routinely outperform the “standard” C/Fortran methods, and algorithms optimized for high-precision and HPC applications. Simultaneously, it wraps the classic C/Fortran methods, making it easy to switch over to them whenever necessary. Solving differential equations with different methods from different languages and packages can be done by changing one line of code, allowing for easy benchmarking to ensure you are using the fastest method possible.

DifferentialEquations.jl integrates with the Julia package sphere. Examples are:

  • GPU acceleration through CUDAnative.jl and CuArrays.jl
  • Automated sparsity detection with Symbolics.jl
  • Automatic Jacobian coloring with SparseDiffTools.jl, allowing for fast solutions to problems with sparse or structured (Tridiagonal, Banded, BlockBanded, etc.) Jacobians
  • Allowing the specification of linear solvers for maximal efficiency
  • Progress meter integration with the Juno IDE for estimated time to solution
  • Automatic plotting of time series and phase plots
  • Built-in interpolations
  • Wraps for common C/Fortran methods, like Sundials and Hairer's radau
  • Arbitrary precision with BigFloats and Arbfloats
  • Arbitrary array types, allowing the definition of differential equations on matrices and distributed arrays
  • Unit-checked arithmetic with Unitful

Optimization.jl: Unified Interface for Optimization

Optimization.jl is the canonical library for solving OptimizationProblems. It includes wrappers of most of the Julia nonlinear optimization ecosystem, allowing one syntax to use all packages in a uniform manner. This covers:

Integrals.jl: Unified Interface for Numerical Integration

Integrals.jl is the canonical library for solving IntegralsProblems. It includes wrappers of most of the Julia quadrature ecosystem, allowing one syntax to use all packages in a uniform manner. This covers:

  • Gauss-Kronrod quadrature
  • Cubature methods (both h and p cubature)
  • Adaptive Monte Carlo methods

JumpProcesses.jl: Stochastic Simulation Algorithms for Jump Processes, Jump-ODEs, and Jump-Diffusions

JumpProcesses.jl is the library for Poisson jump processes, also known as chemical master equations or Gillespie simulations, for simulating chemical reaction networks and other applications. It allows for solving with many methods, including:

  • Direct: the Gillespie Direct method SSA.
  • RDirect: A variant of Gillespie's Direct method that uses rejection to sample the next reaction.
  • DirectCR: The Composition-Rejection Direct method of Slepoy et al. For large networks and linear chain-type networks, it will often give better performance than Direct. (Requires dependency graph, see below.)
  • DirectFW: the Gillespie Direct method SSA with FunctionWrappers. This aggregator uses a different internal storage format for collections of ConstantRateJumps.
  • FRM: the Gillespie first reaction method SSA. Direct should generally offer better performance and be preferred to FRM.
  • FRMFW: the Gillespie first reaction method SSA with FunctionWrappers.
  • NRM: The Gibson-Bruck Next Reaction Method. For some reaction network structures, this may offer better performance than Direct (for example, large, linear chains of reactions). (Requires dependency graph, see below.)
  • RSSA: The Rejection SSA (RSSA) method of Thanh et al. With RSSACR, for very large reaction networks, it often offers the best performance of all methods. (Requires dependency graph, see below.)
  • RSSACR: The Rejection SSA (RSSA) with Composition-Rejection method of Thanh et al. With RSSA, for very large reaction networks, it often offers the best performance of all methods. (Requires dependency graph, see below.)
  • SortingDirect: The Sorting Direct Method of McCollum et al. It will usually offer performance as good as Direct, and for some systems can offer substantially better performance. (Requires dependency graph, see below.)

The design of JumpProcesses.jl composes with DifferentialEquations.jl, allowing for discrete stochastic chemical reactions to be easily mixed with differential equation models, allowing for simulation of hybrid systems, jump diffusions, and differential equations driven by Levy processes.

In addition, JumpProcesses's interfaces allow for solving with regular jump methods, such as adaptive Tau-Leaping.

Third-Party Libraries to Note

JuMP.jl: Julia for Mathematical Programming

While Optimization.jl is the preferred library for nonlinear optimization, for all other forms of optimization Julia for Mathematical Programming (JuMP) is the star. JuMP is the leading choice in Julia for doing:

  • Linear Programming
  • Quadratic Programming
  • Convex Programming
  • Conic Programming
  • Semidefinite Programming
  • Mixed-Complementarity Programming
  • Integer Programming
  • Mixed Integer (nonlinear/linear) Programming
  • (Mixed Integer) Second Order Conic Programming

JuMP can also be used for some nonlinear programming, though the Optimization.jl bindings to the JuMP solvers (via MathOptInterface.jl) is generally preferred.

FractionalDiffEq.jl: Fractional Differential Equation Solvers

FractionalDiffEq.jl is a set of high-performance solvers for fractional differential equations.

ManifoldDiffEq.jl: Solvers for Differential Equations on Manifolds

ManifoldDiffEq.jl is a set of high-performance solvers for differential equations on manifolds using methods such as Lie Group actions and frozen coefficients (Crouch-Grossman methods). These solvers can in many cases out-perform the OrdinaryDiffEq.jl nonautonomous operator ODE solvers by using methods specialized on manifold definitions of ManifoldsBase.

Manopt.jl: Optimization on Manifolds

ManOpt.jl allows for easy and efficient solving of nonlinear optimization problems on manifolds.

diff --git a/dev/highlevels/function_approximation/index.html b/dev/highlevels/function_approximation/index.html index f2b4ee62056..3d5b7863d6d 100644 --- a/dev/highlevels/function_approximation/index.html +++ b/dev/highlevels/function_approximation/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Function Approximation

While SciML is not an ecosystem for machine learning, SciML has many libraries for doing machine learning with its equation solver libraries and machine learning libraries which are integrated into the equation solvers.

Surrogates.jl: Easy Generation of Differentiable Surrogate Models

Surrogates.jl is a library for generating surrogate approximations to computationally expensive simulations. It has the following high-dimensional function approximators:

  • Kriging
  • Kriging using Stheno
  • Radial Basis
  • Wendland
  • Linear
  • Second Order Polynomial
  • Support Vector Machines (Wait for LIBSVM resolution)
  • Neural Networks
  • Random Forests
  • Lobachevsky splines
  • Inverse-distance
  • Polynomial expansions
  • Variable fidelity
  • Mixture of experts (Waiting GaussianMixtures package to work on v1.5)
  • Earth
  • Gradient Enhanced Kriging

ReservoirComputing.jl: Fast and Flexible Reservoir Computing Methods

ReservoirComputing.jl is a library for doing machine learning using reservoir computing techniques, such as with methods like Echo State Networks (ESNs). Its reservoir computing methods make it stabilized for usage with difficult equations like stiff dynamics, chaotic equations, and more.

Third-Party Libraries to Note

Flux.jl: the ML library that doesn't make you tensor

Flux.jl is the most popular machine learning library in the Julia programming language. SciML's libraries are heavily tested with it and its automatic differentiation engine Zygote.jl for composability and compatibility.

Lux.jl: Explicitly Parameterized Neural Networks in Julia

Lux.jl is a library for fully explicitly parameterized neural networks. Thus, while alternative interfaces are required to use Flux with many equation solvers (i.e. Flux.destructure), Lux.jl's explicit design marries effortlessly with the SciML equation solver libraries. For this reason, SciML's library are also heavily tested with Lux to ensure compatibility with neural network definitions from here.

SimpleChains.jl: Fast Small-Scale Machine Learning

SimpleChains.jl is a library specialized for small-scale machine learning. It uses non-allocating mutating forms to be highly efficient for the cases where matrix multiplication kernels cannot overcome the common overheads of machine learning libraries. Thus for SciML cases with small neural networks (<100 node layers) and non-batched usage (many/most use cases), SimpleChains.jl can be the fastest choice for the neural network definitions.

NNLib.jl: Neural Network Primitives with Multiple Backends

NNLib.jl is the core library which defines the handling of common functions, like conv and how they map to device accelerators such as the NVIDIA cudnn. This library can thus be used to directly grab many of the core functions used in machine learning, such as common activation functions and gather/scatter operations, without depending on the given style of any machine learning library.

GeometricFlux.jl: Geometric Deep Learning and Graph Neural Networks

GeometricFlux.jl is a library for graph neural networks and geometric deep learning. It is the one that is used and tested by the SciML developers for mixing with equation solver applications.

AbstractGPs.jl: Fast and Flexible Gaussian Processes

AbstractGPs.jl is the fast and flexible Gaussian Process library that is used by the SciML packages and recommended for downstream usage.

MLDatasets.jl: Common Machine Learning Datasets

MLDatasets.jl is a common interface for accessing common machine learning datasets. For example, if you want to run a test on MNIST data, MLDatasets is the quickest way to obtain it.

MLUtils.jl: Utility Functions for Machine Learning Pipelines

MLUtils.jl is a library of utility functions for making writing common machine learning pipelines easier. This includes functionality for:

  • An extensible dataset interface (numobs and getobs).
  • Data iteration and data loaders (eachobs and DataLoader).
  • Lazy data views (obsview).
  • Resampling procedures (undersample and oversample).
  • Train/test splits (splitobs)
  • Data partitioning and aggregation tools (batch, unbatch, chunk, group_counts, group_indices).
  • Folds for cross-validation (kfolds, leavepout).
  • Datasets lazy transformations (mapobs, filterobs, groupobs, joinobs, shuffleobs).
  • Toy datasets for demonstration purpose.
  • Other data handling utilities (flatten, normalise, unsqueeze, stack, unstack).
+
Edit on GitHub

Function Approximation

While SciML is not an ecosystem for machine learning, SciML has many libraries for doing machine learning with its equation solver libraries and machine learning libraries which are integrated into the equation solvers.

Surrogates.jl: Easy Generation of Differentiable Surrogate Models

Surrogates.jl is a library for generating surrogate approximations to computationally expensive simulations. It has the following high-dimensional function approximators:

  • Kriging
  • Kriging using Stheno
  • Radial Basis
  • Wendland
  • Linear
  • Second Order Polynomial
  • Support Vector Machines (Wait for LIBSVM resolution)
  • Neural Networks
  • Random Forests
  • Lobachevsky splines
  • Inverse-distance
  • Polynomial expansions
  • Variable fidelity
  • Mixture of experts (Waiting GaussianMixtures package to work on v1.5)
  • Earth
  • Gradient Enhanced Kriging

ReservoirComputing.jl: Fast and Flexible Reservoir Computing Methods

ReservoirComputing.jl is a library for doing machine learning using reservoir computing techniques, such as with methods like Echo State Networks (ESNs). Its reservoir computing methods make it stabilized for usage with difficult equations like stiff dynamics, chaotic equations, and more.

Third-Party Libraries to Note

Flux.jl: the ML library that doesn't make you tensor

Flux.jl is the most popular machine learning library in the Julia programming language. SciML's libraries are heavily tested with it and its automatic differentiation engine Zygote.jl for composability and compatibility.

Lux.jl: Explicitly Parameterized Neural Networks in Julia

Lux.jl is a library for fully explicitly parameterized neural networks. Thus, while alternative interfaces are required to use Flux with many equation solvers (i.e. Flux.destructure), Lux.jl's explicit design marries effortlessly with the SciML equation solver libraries. For this reason, SciML's library are also heavily tested with Lux to ensure compatibility with neural network definitions from here.

SimpleChains.jl: Fast Small-Scale Machine Learning

SimpleChains.jl is a library specialized for small-scale machine learning. It uses non-allocating mutating forms to be highly efficient for the cases where matrix multiplication kernels cannot overcome the common overheads of machine learning libraries. Thus for SciML cases with small neural networks (<100 node layers) and non-batched usage (many/most use cases), SimpleChains.jl can be the fastest choice for the neural network definitions.

NNLib.jl: Neural Network Primitives with Multiple Backends

NNLib.jl is the core library which defines the handling of common functions, like conv and how they map to device accelerators such as the NVIDIA cudnn. This library can thus be used to directly grab many of the core functions used in machine learning, such as common activation functions and gather/scatter operations, without depending on the given style of any machine learning library.

GeometricFlux.jl: Geometric Deep Learning and Graph Neural Networks

GeometricFlux.jl is a library for graph neural networks and geometric deep learning. It is the one that is used and tested by the SciML developers for mixing with equation solver applications.

AbstractGPs.jl: Fast and Flexible Gaussian Processes

AbstractGPs.jl is the fast and flexible Gaussian Process library that is used by the SciML packages and recommended for downstream usage.

MLDatasets.jl: Common Machine Learning Datasets

MLDatasets.jl is a common interface for accessing common machine learning datasets. For example, if you want to run a test on MNIST data, MLDatasets is the quickest way to obtain it.

MLUtils.jl: Utility Functions for Machine Learning Pipelines

MLUtils.jl is a library of utility functions for making writing common machine learning pipelines easier. This includes functionality for:

  • An extensible dataset interface (numobs and getobs).
  • Data iteration and data loaders (eachobs and DataLoader).
  • Lazy data views (obsview).
  • Resampling procedures (undersample and oversample).
  • Train/test splits (splitobs)
  • Data partitioning and aggregation tools (batch, unbatch, chunk, group_counts, group_indices).
  • Folds for cross-validation (kfolds, leavepout).
  • Datasets lazy transformations (mapobs, filterobs, groupobs, joinobs, shuffleobs).
  • Toy datasets for demonstration purpose.
  • Other data handling utilities (flatten, normalise, unsqueeze, stack, unstack).
diff --git a/dev/highlevels/implicit_layers/index.html b/dev/highlevels/implicit_layers/index.html index 3a70326c4f0..d3b119b3af8 100644 --- a/dev/highlevels/implicit_layers/index.html +++ b/dev/highlevels/implicit_layers/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Implicit Layer Deep Learning

Implicit layer deep learning is a field which uses implicit rules, such as differential equations and nonlinear solvers, to define the layers of neural networks. This field has brought the potential to automatically optimize network depth and improve training performance. SciML's differentiable solver ecosystem is specifically designed to accommodate implicit layer methodologies, and provides libraries with pre-built layers for common methods.

DiffEqFlux.jl: High Level Pre-Built Architectures for Implicit Deep Learning

DiffEqFlux.jl is a library of pre-built architectures for implicit deep learning, including layer definitions for methods like:

DeepEquilibriumNetworks.jl: Deep Equilibrium Models Made Fast

DeepEquilibriumNetworks.jl is a library of optimized layer implementations for Deep Equilibrium Models (DEQs). It uses special training techniques such as implicit-explicit regularization in order to accelerate the convergence over traditional implementations, all while using the optimized and flexible SciML libraries under the hood.

+
Edit on GitHub

Implicit Layer Deep Learning

Implicit layer deep learning is a field which uses implicit rules, such as differential equations and nonlinear solvers, to define the layers of neural networks. This field has brought the potential to automatically optimize network depth and improve training performance. SciML's differentiable solver ecosystem is specifically designed to accommodate implicit layer methodologies, and provides libraries with pre-built layers for common methods.

DiffEqFlux.jl: High Level Pre-Built Architectures for Implicit Deep Learning

DiffEqFlux.jl is a library of pre-built architectures for implicit deep learning, including layer definitions for methods like:

DeepEquilibriumNetworks.jl: Deep Equilibrium Models Made Fast

DeepEquilibriumNetworks.jl is a library of optimized layer implementations for Deep Equilibrium Models (DEQs). It uses special training techniques such as implicit-explicit regularization in order to accelerate the convergence over traditional implementations, all while using the optimized and flexible SciML libraries under the hood.

diff --git a/dev/highlevels/interfaces/index.html b/dev/highlevels/interfaces/index.html index dba6860134b..f40178843c1 100644 --- a/dev/highlevels/interfaces/index.html +++ b/dev/highlevels/interfaces/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

The SciML Interface Libraries

SciMLBase.jl: The SciML Common Interface

SciMLBase.jl defines the core interfaces of the SciML libraries, such as the definitions of abstract types like SciMLProblem, along with their instantiations like ODEProblem. While SciMLBase.jl is insufficient to solve any equations, it holds all the equation definitions, and thus downstream libraries which wish to allow for using SciML solvers without depending on any solvers can directly depend on SciMLBase.jl.

SciMLOperators.jl: The AbstractSciMLOperator Interface

SciMLOperators.jl defines the interface for how matrix-free linear and affine operators are defined and used throughout the SciML ecosystem.

DiffEqNoiseProcess.jl: The SciML Common Noise Interface

DiffEqNoiseProcess.jl defines the common interface for stochastic noise processes used by the equation solvers of the SciML ecosystem.

CommonSolve.jl: The Common Definition of Solve

CommonSolve.jl is the library that defines the solve, solve!, and init interfaces which are used throughout all the SciML equation solvers. It's defined as an extremely lightweight library so that other ecosystems can build on the same solve definition without clashing with SciML when both export.

Static.jl: A Shared Interface for Static Compile-Time Computation

Static.jl is a set of statically parameterized types for performing operations in a statically-defined (compiler-optimized) way with respect to values.

DiffEqBase.jl: A Library of Shared Components for Differential Equation Solvers

DiffEqBase.jl is the core shared component of the DifferentialEquations.jl ecosystem. It's not intended for non-developer users to interface directly with, instead it's used for the common functionality for uniformity of implementation between the solver libraries.

Third-Party Libraries to Note

ArrayInterface.jl: Extensions to the Julia AbstractArray Interface

ArrayInterface.jl are traits and functions which extend the Julia Base AbstractArray interface, giving a much larger set of queries to allow for writing high-performance generic code over all array types. For example, functions include can_change_size to know if an AbstractArray type is compatible with resize!, fast_scalar_indexing to know whether direct scalar indexing A[i] is optimized, and functions like findstructralnz to get the structural non-zeros of arbitrary sparse and structured matrices.

Adapt.jl: Conversion to Allow Chip-Generic Programs

Adapt.jl makes it possible to write code that is generic to the compute devices, i.e. code that works on both CPUs and GPUs. It defines the adapt function which acts like convert(T, x), but without the restriction of returning a T. This allows you to “convert” wrapper types, like Adjoint to be GPU compatible (for example) without throwing away the wrapper.

Example usage:

adapt(CuArray, ::Adjoint{Array})::Adjoint{CuArray}

AbstractFFTs.jl: High Level Shared Interface for Fast Fourier Transformation Libraries

AbstractFFTs.jl defines the common interface for Fast Fourier Transformations (FFTs) in Julia. Similar to SciMLBase.jl, AbstractFFTs.jl is not a solver library but instead a shared API which is extended by solver libraries such as FFTW.jl. Code written using AbstractFFTs.jl can be made compatible with FFT libraries without having an explicit dependency on a solver.

GPUArrays.jl: Common Interface for GPU-Based Array Types

GPUArrays.jl defines the shared higher-level operations for GPU-based array types, like CUDA.jl's CuArray and AMDGPU.jl's ROCmArray. Packages in SciML use the designation x isa AbstractGPUArray in order to find out if a user's operation is on the GPU and specialize computations.

RecipesBase.jl: Standard Plotting Recipe Interface

RecipesBase.jl defines the common interface for plotting recipes, composable transformations of Julia data types into simpler data types for visualization with libraries such as Plots.jl and Makie.jl. SciML libraries attempt to always include plot recipes wherever possible for ease of visualization.

Tables.jl: Common Interface for Tabular Data Types

Tables.jl is a common interface for defining tabular data structures, such as DataFrames.jl. SciML's libraries extend the Tables.jl interface to allow for automated conversions into data frame libraries without explicit dependence on any singular implementation.

+
Edit on GitHub

The SciML Interface Libraries

SciMLBase.jl: The SciML Common Interface

SciMLBase.jl defines the core interfaces of the SciML libraries, such as the definitions of abstract types like SciMLProblem, along with their instantiations like ODEProblem. While SciMLBase.jl is insufficient to solve any equations, it holds all the equation definitions, and thus downstream libraries which wish to allow for using SciML solvers without depending on any solvers can directly depend on SciMLBase.jl.

SciMLOperators.jl: The AbstractSciMLOperator Interface

SciMLOperators.jl defines the interface for how matrix-free linear and affine operators are defined and used throughout the SciML ecosystem.

DiffEqNoiseProcess.jl: The SciML Common Noise Interface

DiffEqNoiseProcess.jl defines the common interface for stochastic noise processes used by the equation solvers of the SciML ecosystem.

CommonSolve.jl: The Common Definition of Solve

CommonSolve.jl is the library that defines the solve, solve!, and init interfaces which are used throughout all the SciML equation solvers. It's defined as an extremely lightweight library so that other ecosystems can build on the same solve definition without clashing with SciML when both export.

Static.jl: A Shared Interface for Static Compile-Time Computation

Static.jl is a set of statically parameterized types for performing operations in a statically-defined (compiler-optimized) way with respect to values.

DiffEqBase.jl: A Library of Shared Components for Differential Equation Solvers

DiffEqBase.jl is the core shared component of the DifferentialEquations.jl ecosystem. It's not intended for non-developer users to interface directly with, instead it's used for the common functionality for uniformity of implementation between the solver libraries.

Third-Party Libraries to Note

ArrayInterface.jl: Extensions to the Julia AbstractArray Interface

ArrayInterface.jl are traits and functions which extend the Julia Base AbstractArray interface, giving a much larger set of queries to allow for writing high-performance generic code over all array types. For example, functions include can_change_size to know if an AbstractArray type is compatible with resize!, fast_scalar_indexing to know whether direct scalar indexing A[i] is optimized, and functions like findstructralnz to get the structural non-zeros of arbitrary sparse and structured matrices.

Adapt.jl: Conversion to Allow Chip-Generic Programs

Adapt.jl makes it possible to write code that is generic to the compute devices, i.e. code that works on both CPUs and GPUs. It defines the adapt function which acts like convert(T, x), but without the restriction of returning a T. This allows you to “convert” wrapper types, like Adjoint to be GPU compatible (for example) without throwing away the wrapper.

Example usage:

adapt(CuArray, ::Adjoint{Array})::Adjoint{CuArray}

AbstractFFTs.jl: High Level Shared Interface for Fast Fourier Transformation Libraries

AbstractFFTs.jl defines the common interface for Fast Fourier Transformations (FFTs) in Julia. Similar to SciMLBase.jl, AbstractFFTs.jl is not a solver library but instead a shared API which is extended by solver libraries such as FFTW.jl. Code written using AbstractFFTs.jl can be made compatible with FFT libraries without having an explicit dependency on a solver.

GPUArrays.jl: Common Interface for GPU-Based Array Types

GPUArrays.jl defines the shared higher-level operations for GPU-based array types, like CUDA.jl's CuArray and AMDGPU.jl's ROCmArray. Packages in SciML use the designation x isa AbstractGPUArray in order to find out if a user's operation is on the GPU and specialize computations.

RecipesBase.jl: Standard Plotting Recipe Interface

RecipesBase.jl defines the common interface for plotting recipes, composable transformations of Julia data types into simpler data types for visualization with libraries such as Plots.jl and Makie.jl. SciML libraries attempt to always include plot recipes wherever possible for ease of visualization.

Tables.jl: Common Interface for Tabular Data Types

Tables.jl is a common interface for defining tabular data structures, such as DataFrames.jl. SciML's libraries extend the Tables.jl interface to allow for automated conversions into data frame libraries without explicit dependence on any singular implementation.

diff --git a/dev/highlevels/inverse_problems/index.html b/dev/highlevels/inverse_problems/index.html index 0f3dab8b5ad..56ba7f3e8ea 100644 --- a/dev/highlevels/inverse_problems/index.html +++ b/dev/highlevels/inverse_problems/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Parameter Estimation, Bayesian Analysis, and Inverse Problems

Parameter estimation for models and equations, also known as dynamic data analysis, solving the inverse problem, or Bayesian posterior estimation (when done probabilistically), is provided by the SciML tools for the equations in its set. In this introduction, we briefly present the relevant packages that facilitate parameter estimation, namely:

We also provide information regarding the respective strengths of these packages so that you can easily decide which one suits your needs best.

SciMLSensitivity.jl: Local Sensitivity Analysis and Automatic Differentiation Support for Solvers

SciMLSensitivity.jl is the system for local sensitivity, which all other inverse problem methods rely on. This package defines the interactions between the equation solvers and automatic differentiation, defining fast overloads for forward and adjoint (reverse) sensitivity analysis for fast gradient and Jacobian calculations with respect to model inputs. Its documentation covers how to use direct differentiation of equation solvers in conjunction with tools like Optimization.jl to perform model calibration of ODEs against data, PDE-constrained optimization, nonlinear optimal controls analysis, and much more. As a lower level tool, this library is very versatile, feature-rich, and high-performance, giving all the tools required but not directly providing a higher level interface.

Note

Sensitivity analysis is kept in a separate library from the solvers (SciMLSensitivity.jl), in order to not require all equation solvers to have a dependency on all automatic differentiation libraries. If automatic differentiation is applied to a solver library without importing SciMLSensitivity.jl, an error is thrown letting the user know to import SciMLSensitivity.jl for the functionality to exist.

DataDrivenDiffEq.jl: Data-Driven Modeling and Equation Discovery

The distinguishing feature of this package is that its ultimate goal is to identify the differential equation model that generated the input data. Depending on the user's needs, the package can provide structural identification of a given differential equation (output in a symbolic form) or structural estimation (output as a function for prediction purposes).

DiffEqParamEstim.jl: Simplified Parameter Estimation Interface

This package is for simplified parameter estimation. While not as flexible of a system like DiffEqFlux.jl, it provides ready-made functions for doing standard optimization procedures like L2 fitting and MAP estimates. Among other features, it allows for the optimization of parameters in ODEs, stochastic problems, and delay differential equations.

DiffEqBayes.jl: Simplified Bayesian Estimation Interface

As the name suggests, this package has been designed to provide the estimation of differential equations parameters by Bayesian methods. It works in conjunction with Turing.jl, CmdStan.jl, DynamicHMC.jl, and ApproxBayes.jl. While not as flexible as direct usage of DiffEqFlux.jl or Turing.jl, DiffEqBayes.jl can be an approachable interface for those not familiar with Bayesian estimation, and provides a nice way to use Stan from pure Julia.

Third-Party Tools of Note

Turing.jl: A Flexible Probabilistic Programming Language for Bayesian Analysis

In the context of differential equations and parameter estimation, Turing.jl allows for a Bayesian estimation of differential equations (used in conjunction with the high-level package DiffEqBayes.jl). For more examples on combining Turing.jl with DiffEqBayes.jl, see the documentation below. It is important to note that Turing.jl can also perform Bayesian estimation without relying on DiffEqBayes.jl (for an example, consult this tutorial).

Topopt.jl: Topology Optimization in Julia

Topopt.jl solves topology optimization problems which are inverse problems on partial differential equations, solving for an optimal domain.

Recommended Automatic Differentiation Libraries

Solving inverse problems commonly requires using automatic differentiation (AD). SciML includes extensive support for automatic differentiation throughout its solvers, though some AD libraries are more tested than others. The following libraries are the current recommendations of the SciML developers.

ForwardDiff.jl: Operator-Overloading Forward Mode Automatic Differentiation

ForwardDiff.jl is a library for operator-overloading based forward-mode automatic differentiation. It's commonly used as the default method for generating Jacobians throughout the SciML solver libraries.

Note

Because ForwardDiff.jl uses an operator overloading approach, uses of ForwardDiff.jl require that any caches for non-allocating mutating code allows for Dual numbers. To allow such code to be ForwardDiff.jl-compatible, see PreallocationTools.jl.

Enzyme.jl: LLVM-Level Forward and Reverse Mode Automatic Differentiation

Enzyme.jl is an LLVM-level AD library for forward and reverse automatic differentiation. It supports many features required for high performance, such as being able to differentiate mutating and interleave compiler optimization with the AD passes. However, it does not support all of the Julia runtime, and thus some code with many dynamic behaviors and garbage collection (GC) invocations can be incompatible with Enzyme. Enzyme.jl is quickly becoming the new standard AD for SciML.

Zygote.jl: Julia-Level Source-to-Source Reverse Mode Automatic Differentiation

Zygote.jl is the current standard user-level reverse-mode automatic differentiation library for the SciML solvers. User-level means that many library tutorials, like in SciMLSensitivity.jl and DiffEqFlux.jl, showcase user code using Zygote.jl. This is because Zygote.jl is the AD engine associated with the Flux machine learning library. However, Zygote.jl has many limitations which limits its performance in equation solver contexts, such as an inability to handle mutation and introducing many small allocations and type-instabilities. For this reason, the SciML equation solvers define differentiation overloads using ChainRules.jl, meaning that the equation solvers tend not to use Zygote.jl internally even if the user code uses Zygote.gradient. In this manner, the speed and performance of more advanced techniques can be preserved while using the Julia standard.

FiniteDiff.jl: Fast Finite Difference Approximations

FiniteDiff.jl is the preferred fallback library for numerical differentiation and is commonly used by SciML solver libraries when automatic differentiation is disabled.

SparseDiffTools.jl: Tools for Fast Automatic Differentiation with Sparse Operators

SparseDiffTools.jl is a library for sparse automatic differentiation. It's used internally by many of the SciML equation solver libraries, which explicitly expose interfaces for colorvec color vectors generated by SparseDiffTools.jl's methods. SparseDiffTools.jl also includes many features useful to users, such as operators for matrix-free Jacobian-vector and Hessian-vector products.

+
Edit on GitHub

Parameter Estimation, Bayesian Analysis, and Inverse Problems

Parameter estimation for models and equations, also known as dynamic data analysis, solving the inverse problem, or Bayesian posterior estimation (when done probabilistically), is provided by the SciML tools for the equations in its set. In this introduction, we briefly present the relevant packages that facilitate parameter estimation, namely:

We also provide information regarding the respective strengths of these packages so that you can easily decide which one suits your needs best.

SciMLSensitivity.jl: Local Sensitivity Analysis and Automatic Differentiation Support for Solvers

SciMLSensitivity.jl is the system for local sensitivity, which all other inverse problem methods rely on. This package defines the interactions between the equation solvers and automatic differentiation, defining fast overloads for forward and adjoint (reverse) sensitivity analysis for fast gradient and Jacobian calculations with respect to model inputs. Its documentation covers how to use direct differentiation of equation solvers in conjunction with tools like Optimization.jl to perform model calibration of ODEs against data, PDE-constrained optimization, nonlinear optimal controls analysis, and much more. As a lower level tool, this library is very versatile, feature-rich, and high-performance, giving all the tools required but not directly providing a higher level interface.

Note

Sensitivity analysis is kept in a separate library from the solvers (SciMLSensitivity.jl), in order to not require all equation solvers to have a dependency on all automatic differentiation libraries. If automatic differentiation is applied to a solver library without importing SciMLSensitivity.jl, an error is thrown letting the user know to import SciMLSensitivity.jl for the functionality to exist.

DataDrivenDiffEq.jl: Data-Driven Modeling and Equation Discovery

The distinguishing feature of this package is that its ultimate goal is to identify the differential equation model that generated the input data. Depending on the user's needs, the package can provide structural identification of a given differential equation (output in a symbolic form) or structural estimation (output as a function for prediction purposes).

DiffEqParamEstim.jl: Simplified Parameter Estimation Interface

This package is for simplified parameter estimation. While not as flexible of a system like DiffEqFlux.jl, it provides ready-made functions for doing standard optimization procedures like L2 fitting and MAP estimates. Among other features, it allows for the optimization of parameters in ODEs, stochastic problems, and delay differential equations.

DiffEqBayes.jl: Simplified Bayesian Estimation Interface

As the name suggests, this package has been designed to provide the estimation of differential equations parameters by Bayesian methods. It works in conjunction with Turing.jl, CmdStan.jl, DynamicHMC.jl, and ApproxBayes.jl. While not as flexible as direct usage of DiffEqFlux.jl or Turing.jl, DiffEqBayes.jl can be an approachable interface for those not familiar with Bayesian estimation, and provides a nice way to use Stan from pure Julia.

Third-Party Tools of Note

Turing.jl: A Flexible Probabilistic Programming Language for Bayesian Analysis

In the context of differential equations and parameter estimation, Turing.jl allows for a Bayesian estimation of differential equations (used in conjunction with the high-level package DiffEqBayes.jl). For more examples on combining Turing.jl with DiffEqBayes.jl, see the documentation below. It is important to note that Turing.jl can also perform Bayesian estimation without relying on DiffEqBayes.jl (for an example, consult this tutorial).

Topopt.jl: Topology Optimization in Julia

Topopt.jl solves topology optimization problems which are inverse problems on partial differential equations, solving for an optimal domain.

Recommended Automatic Differentiation Libraries

Solving inverse problems commonly requires using automatic differentiation (AD). SciML includes extensive support for automatic differentiation throughout its solvers, though some AD libraries are more tested than others. The following libraries are the current recommendations of the SciML developers.

ForwardDiff.jl: Operator-Overloading Forward Mode Automatic Differentiation

ForwardDiff.jl is a library for operator-overloading based forward-mode automatic differentiation. It's commonly used as the default method for generating Jacobians throughout the SciML solver libraries.

Note

Because ForwardDiff.jl uses an operator overloading approach, uses of ForwardDiff.jl require that any caches for non-allocating mutating code allows for Dual numbers. To allow such code to be ForwardDiff.jl-compatible, see PreallocationTools.jl.

Enzyme.jl: LLVM-Level Forward and Reverse Mode Automatic Differentiation

Enzyme.jl is an LLVM-level AD library for forward and reverse automatic differentiation. It supports many features required for high performance, such as being able to differentiate mutating and interleave compiler optimization with the AD passes. However, it does not support all of the Julia runtime, and thus some code with many dynamic behaviors and garbage collection (GC) invocations can be incompatible with Enzyme. Enzyme.jl is quickly becoming the new standard AD for SciML.

Zygote.jl: Julia-Level Source-to-Source Reverse Mode Automatic Differentiation

Zygote.jl is the current standard user-level reverse-mode automatic differentiation library for the SciML solvers. User-level means that many library tutorials, like in SciMLSensitivity.jl and DiffEqFlux.jl, showcase user code using Zygote.jl. This is because Zygote.jl is the AD engine associated with the Flux machine learning library. However, Zygote.jl has many limitations which limits its performance in equation solver contexts, such as an inability to handle mutation and introducing many small allocations and type-instabilities. For this reason, the SciML equation solvers define differentiation overloads using ChainRules.jl, meaning that the equation solvers tend not to use Zygote.jl internally even if the user code uses Zygote.gradient. In this manner, the speed and performance of more advanced techniques can be preserved while using the Julia standard.

FiniteDiff.jl: Fast Finite Difference Approximations

FiniteDiff.jl is the preferred fallback library for numerical differentiation and is commonly used by SciML solver libraries when automatic differentiation is disabled.

SparseDiffTools.jl: Tools for Fast Automatic Differentiation with Sparse Operators

SparseDiffTools.jl is a library for sparse automatic differentiation. It's used internally by many of the SciML equation solver libraries, which explicitly expose interfaces for colorvec color vectors generated by SparseDiffTools.jl's methods. SparseDiffTools.jl also includes many features useful to users, such as operators for matrix-free Jacobian-vector and Hessian-vector products.

diff --git a/dev/highlevels/learning_resources/index.html b/dev/highlevels/learning_resources/index.html index 377c5b47952..9ecde6936de 100644 --- a/dev/highlevels/learning_resources/index.html +++ b/dev/highlevels/learning_resources/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Curated Learning, Teaching, and Training Resources

While the SciML documentation is made to be comprehensive, there will always be good alternative resources. The purpose of this section of the documentation is to highlight the alternative resources which can be helpful for learning how to use the SciML Open-Source Software libraries.

JuliaCon and SciMLCon Videos

Many tutorials and introductions to packages have been taught through previous JuliaCon/SciMLCon workshops and talks. The following is a curated list of such training videos:

SciML Book: Parallel Computing and Scientific Machine Learning (SciML): Methods and Applications

The book Parallel Computing and Scientific Machine Learning (SciML): Methods and Applications is a compilation of the lecture notes from the MIT Course 18.337J/6.338J: Parallel Computing and Scientific Machine Learning. It contains a walkthrough of many of the methods implemented in the SciML libraries, as well as how to understand much of the functionality at a deeper level. This course was intended for MIT graduate students in engineering, computer science, and mathematics and thus may have a high prerequisite requirement than many other resources.

sir-julia: Various implementations of the classical SIR model in Julia

For those who like to learn by example, the repository sir-julia is a great resource! It showcases how to use the SciML libraries in many different ways to simulate different variations of the classic SIR epidemic model.

Other Books Featuring SciML

+
Edit on GitHub

Curated Learning, Teaching, and Training Resources

While the SciML documentation is made to be comprehensive, there will always be good alternative resources. The purpose of this section of the documentation is to highlight the alternative resources which can be helpful for learning how to use the SciML Open-Source Software libraries.

JuliaCon and SciMLCon Videos

Many tutorials and introductions to packages have been taught through previous JuliaCon/SciMLCon workshops and talks. The following is a curated list of such training videos:

SciML Book: Parallel Computing and Scientific Machine Learning (SciML): Methods and Applications

The book Parallel Computing and Scientific Machine Learning (SciML): Methods and Applications is a compilation of the lecture notes from the MIT Course 18.337J/6.338J: Parallel Computing and Scientific Machine Learning. It contains a walkthrough of many of the methods implemented in the SciML libraries, as well as how to understand much of the functionality at a deeper level. This course was intended for MIT graduate students in engineering, computer science, and mathematics and thus may have a high prerequisite requirement than many other resources.

sir-julia: Various implementations of the classical SIR model in Julia

For those who like to learn by example, the repository sir-julia is a great resource! It showcases how to use the SciML libraries in many different ways to simulate different variations of the classic SIR epidemic model.

Other Books Featuring SciML

diff --git a/dev/highlevels/model_libraries_and_importers/index.html b/dev/highlevels/model_libraries_and_importers/index.html index 6c1f7d2571f..daa5364a7f6 100644 --- a/dev/highlevels/model_libraries_and_importers/index.html +++ b/dev/highlevels/model_libraries_and_importers/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Model Libraries and Importers

Models are passed on from generation to generation. Many models are not built from scratch but have a legacy of the known physics, biology, and chemistry embedded into them. Julia's SciML offers a range of pre-built modeling tools, from reusable acausal components to direct imports from common file formats.

ModelingToolkitStandardLibrary.jl: A Standard Library for ModelingToolkit

Given the composable nature of acausal modeling systems, it's helpful to not have to define every component from scratch and instead build off a common base of standard components. ModelingToolkitStandardLibrary.jl is that library. It provides components for standard models to start building everything from circuits and engines to robots.

DiffEqCallbacks.jl: Pre-Made Callbacks for DifferentialEquations.jl

DiffEqCallbacks.jl has many event handling and callback definitions which allow for quickly building up complex differential equation models. It includes:

  • Callbacks for specialized output and saving procedures
  • Callbacks for enforcing domain constraints, positivity, and manifolds
  • Timed callbacks for periodic dosing, presetting of tstops, and more
  • Callbacks for determining and terminating at steady state
  • Callbacks for controlling stepsizes and enforcing CFL conditions
  • Callbacks for quantifying uncertainty with respect to numerical errors

SBMLToolkit.jl: SBML Import

SBMLToolkit.jl is a library for reading SBML files into the standard formats for Catalyst.jl and ModelingToolkit.jl. There are well over one thousand biological models available in the BioModels Repository.

CellMLToolkit.jl: CellML Import

CellMLToolkit.jl is a library for reading CellML files into the standard formats for ModelingToolkit.jl. There are several hundred biological models available in the CellML Model Repository.

ReactionNetworkImporters.jl: BioNetGen Import

ReactionNetworkImporters.jl is a library for reading BioNetGen .net files and various stoichiometry matrix representations into the standard formats for Catalyst.jl and ModelingToolkit.jl.

+
Edit on GitHub

Model Libraries and Importers

Models are passed on from generation to generation. Many models are not built from scratch but have a legacy of the known physics, biology, and chemistry embedded into them. Julia's SciML offers a range of pre-built modeling tools, from reusable acausal components to direct imports from common file formats.

ModelingToolkitStandardLibrary.jl: A Standard Library for ModelingToolkit

Given the composable nature of acausal modeling systems, it's helpful to not have to define every component from scratch and instead build off a common base of standard components. ModelingToolkitStandardLibrary.jl is that library. It provides components for standard models to start building everything from circuits and engines to robots.

DiffEqCallbacks.jl: Pre-Made Callbacks for DifferentialEquations.jl

DiffEqCallbacks.jl has many event handling and callback definitions which allow for quickly building up complex differential equation models. It includes:

  • Callbacks for specialized output and saving procedures
  • Callbacks for enforcing domain constraints, positivity, and manifolds
  • Timed callbacks for periodic dosing, presetting of tstops, and more
  • Callbacks for determining and terminating at steady state
  • Callbacks for controlling stepsizes and enforcing CFL conditions
  • Callbacks for quantifying uncertainty with respect to numerical errors

SBMLToolkit.jl: SBML Import

SBMLToolkit.jl is a library for reading SBML files into the standard formats for Catalyst.jl and ModelingToolkit.jl. There are well over one thousand biological models available in the BioModels Repository.

CellMLToolkit.jl: CellML Import

CellMLToolkit.jl is a library for reading CellML files into the standard formats for ModelingToolkit.jl. There are several hundred biological models available in the CellML Model Repository.

ReactionNetworkImporters.jl: BioNetGen Import

ReactionNetworkImporters.jl is a library for reading BioNetGen .net files and various stoichiometry matrix representations into the standard formats for Catalyst.jl and ModelingToolkit.jl.

diff --git a/dev/highlevels/modeling_languages/index.html b/dev/highlevels/modeling_languages/index.html index 65517408936..6ebeb4c7991 100644 --- a/dev/highlevels/modeling_languages/index.html +++ b/dev/highlevels/modeling_languages/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Modeling Languages

While in theory one can build perfect code for all models from scratch, in practice many scientists and engineers need or want some help! The SciML modeling tools provide a higher level interface over the equation solver, which helps the translation from good models to good simulations in a way that abstracts away the mathematical and computational details without giving up performance.

ModelingToolkit.jl: Acausal Symbolic Modeling

Acausal modeling is an extension of causal modeling that is more composable and allows for more code reuse. Build a model of an electric engine, then build a model of a battery, and now declare connections by stating "the voltage at the engine equals the voltage at the connector of the battery", and generate the composed model. The tool for this is ModelingToolkit.jl. ModelingToolkit.jl is a sophisticated symbolic modeling library which allows for specifying these types of large-scale differential equation models in a simple way, abstracting away the computational details. However, its symbolic analysis allows for generating much more performant code for differential-algebraic equations than most users could ever write by hand, with its structural_simplify automatically correcting the model to improve parallelism, numerical stability, and automatically remove variables which it can show are redundant.

ModelingToolkit.jl is the base of the SciML symbolic modeling ecosystem, defining the AbstractSystem types, such as ODESystem, SDESystem, OptimizationSystem, PDESystem, and more, which are then used by all the other modeling tools. As such, when using other modeling tools like Catalyst.jl, the reference for all the things that can be done with the symbolic representation is simply ModelingToolkit.jl.

Catalyst.jl: Chemical Reaction Networks (CRN), Systems Biology, and Quantitative Systems Pharmacology (QSP) Modeling

Catalyst.jl is a modeling interface for efficient simulation of mass action ODE, chemical Langevin SDE, and stochastic chemical kinetics jump process (i.e. chemical master equation) models for chemical reaction networks and population processes. It uses a highly intuitive chemical reaction syntax interface, which generates all the extra functionality necessary for the fastest use with JumpProcesses.jl, DifferentialEquations.jl, and higher level SciML libraries. Its ReactionSystem type is a programmable extension of the ModelingToolkit AbstractSystem interface, meaning that complex reaction systems are represented symbolically, and then compiled to optimized representations automatically when converting ReactionSystems to concrete ODE/SDE/jump process representations. Catalyst also provides functionality to support chemical reaction network and steady-state analysis.

For an overview of the library, see Modeling Biochemical Systems with Catalyst.jl - Samuel Isaacson

NBodySimulator.jl: A differentiable simulator for N-body problems, including astrophysical and molecular dynamics

NBodySimulator.jl is a differentiable simulator for N-body problems, including astrophysical and molecular dynamics. It uses the DifferentialEquations.jl solvers, allowing for one to choose between a large variety of symplectic integration schemes. It implements many of the thermostats required for doing standard molecular dynamics approximations.

DiffEqFinancial.jl: Financial models for use in the DifferentialEquations ecosystem

The goal of DiffEqFinancial.jl is to be a feature-complete set of solvers for the types of problems found in libraries like QuantLib, such as the Heston process or the Black-Scholes model.

ParameterizedFunctions.jl: Simple Differential Equation Definitions Made Easy

This image that went viral is actually runnable code from ParameterizedFunctions.jl. Define equations and models using a very simple high-level syntax and let the code generation tools build symbolic fast Jacobian, gradient, etc. functions for you.

Third-Party Tools of Note

MomentClosure.jl: Automated Generation of Moment Closure Equations

MomentClosure.jl is a library for generating the moment closure equations for a given chemical master equation or stochastic differential equation. Thus instead of solving a stochastic model thousands of times to find the mean and variance, this library can generate the deterministic equations for how the mean and variance evolve in order to be solved in a single run. MomentClosure.jl uses Catalyst ReactionSystem and ModelingToolkit SDESystem types as the input for its symbolic generation processes.

Agents.jl: Agent-Based Modeling Framework in Julia

If one wants to do agent-based modeling in Julia, Agents.jl is the go-to library. It's fast and flexible, making it a solid foundation for any agent-based model.

Unitful.jl: A Julia package for physical units

Supports not only SI units, but also any other unit system. Unitful.jl has minimal run-time penalty of units. Includes facilities for dimensional analysis, and integrates easily with the usual mathematical operations and collections that are defined in Julia.

ReactionMechanismSimulator.jl: Simulation and Analysis of Large Chemical Reaction Systems

ReactionMechanismSimulator.jl is a tool for simulating and analyzing large chemical reaction mechanisms. It interfaces with the ReactionMechanismGenerator suite for automatically constructing reaction pathways from chemical components to quickly build realistic models of chemical systems.

FiniteStateProjection.jl: Direct Solution of Chemical Master Equations

FiniteStateProjection.jl is a library for finite state projection direct solving of the chemical master equation. It automatically converts the Catalyst ReactionSystem definitions into ModelingToolkit ODESystem representations for the evolution of probability distributions to allow for directly solving the weak form of the stochastic model.

AlgebraicPetri.jl: Applied Category Theory of Modeling

AlgebraicPetri.jl is a library for automating the intuitive generation of dynamical models using a Category theory-based approach.

QuantumOptics.jl: Simulating quantum systems.

QuantumOptics.jl makes it easy to simulate various kinds of quantum systems. It is inspired by the Quantum Optics Toolbox for MATLAB and the Python framework QuTiP.

+
Edit on GitHub

Modeling Languages

While in theory one can build perfect code for all models from scratch, in practice many scientists and engineers need or want some help! The SciML modeling tools provide a higher level interface over the equation solver, which helps the translation from good models to good simulations in a way that abstracts away the mathematical and computational details without giving up performance.

ModelingToolkit.jl: Acausal Symbolic Modeling

Acausal modeling is an extension of causal modeling that is more composable and allows for more code reuse. Build a model of an electric engine, then build a model of a battery, and now declare connections by stating "the voltage at the engine equals the voltage at the connector of the battery", and generate the composed model. The tool for this is ModelingToolkit.jl. ModelingToolkit.jl is a sophisticated symbolic modeling library which allows for specifying these types of large-scale differential equation models in a simple way, abstracting away the computational details. However, its symbolic analysis allows for generating much more performant code for differential-algebraic equations than most users could ever write by hand, with its structural_simplify automatically correcting the model to improve parallelism, numerical stability, and automatically remove variables which it can show are redundant.

ModelingToolkit.jl is the base of the SciML symbolic modeling ecosystem, defining the AbstractSystem types, such as ODESystem, SDESystem, OptimizationSystem, PDESystem, and more, which are then used by all the other modeling tools. As such, when using other modeling tools like Catalyst.jl, the reference for all the things that can be done with the symbolic representation is simply ModelingToolkit.jl.

Catalyst.jl: Chemical Reaction Networks (CRN), Systems Biology, and Quantitative Systems Pharmacology (QSP) Modeling

Catalyst.jl is a modeling interface for efficient simulation of mass action ODE, chemical Langevin SDE, and stochastic chemical kinetics jump process (i.e. chemical master equation) models for chemical reaction networks and population processes. It uses a highly intuitive chemical reaction syntax interface, which generates all the extra functionality necessary for the fastest use with JumpProcesses.jl, DifferentialEquations.jl, and higher level SciML libraries. Its ReactionSystem type is a programmable extension of the ModelingToolkit AbstractSystem interface, meaning that complex reaction systems are represented symbolically, and then compiled to optimized representations automatically when converting ReactionSystems to concrete ODE/SDE/jump process representations. Catalyst also provides functionality to support chemical reaction network and steady-state analysis.

For an overview of the library, see Modeling Biochemical Systems with Catalyst.jl - Samuel Isaacson

NBodySimulator.jl: A differentiable simulator for N-body problems, including astrophysical and molecular dynamics

NBodySimulator.jl is a differentiable simulator for N-body problems, including astrophysical and molecular dynamics. It uses the DifferentialEquations.jl solvers, allowing for one to choose between a large variety of symplectic integration schemes. It implements many of the thermostats required for doing standard molecular dynamics approximations.

DiffEqFinancial.jl: Financial models for use in the DifferentialEquations ecosystem

The goal of DiffEqFinancial.jl is to be a feature-complete set of solvers for the types of problems found in libraries like QuantLib, such as the Heston process or the Black-Scholes model.

ParameterizedFunctions.jl: Simple Differential Equation Definitions Made Easy

This image that went viral is actually runnable code from ParameterizedFunctions.jl. Define equations and models using a very simple high-level syntax and let the code generation tools build symbolic fast Jacobian, gradient, etc. functions for you.

Third-Party Tools of Note

MomentClosure.jl: Automated Generation of Moment Closure Equations

MomentClosure.jl is a library for generating the moment closure equations for a given chemical master equation or stochastic differential equation. Thus instead of solving a stochastic model thousands of times to find the mean and variance, this library can generate the deterministic equations for how the mean and variance evolve in order to be solved in a single run. MomentClosure.jl uses Catalyst ReactionSystem and ModelingToolkit SDESystem types as the input for its symbolic generation processes.

Agents.jl: Agent-Based Modeling Framework in Julia

If one wants to do agent-based modeling in Julia, Agents.jl is the go-to library. It's fast and flexible, making it a solid foundation for any agent-based model.

Unitful.jl: A Julia package for physical units

Supports not only SI units, but also any other unit system. Unitful.jl has minimal run-time penalty of units. Includes facilities for dimensional analysis, and integrates easily with the usual mathematical operations and collections that are defined in Julia.

ReactionMechanismSimulator.jl: Simulation and Analysis of Large Chemical Reaction Systems

ReactionMechanismSimulator.jl is a tool for simulating and analyzing large chemical reaction mechanisms. It interfaces with the ReactionMechanismGenerator suite for automatically constructing reaction pathways from chemical components to quickly build realistic models of chemical systems.

FiniteStateProjection.jl: Direct Solution of Chemical Master Equations

FiniteStateProjection.jl is a library for finite state projection direct solving of the chemical master equation. It automatically converts the Catalyst ReactionSystem definitions into ModelingToolkit ODESystem representations for the evolution of probability distributions to allow for directly solving the weak form of the stochastic model.

AlgebraicPetri.jl: Applied Category Theory of Modeling

AlgebraicPetri.jl is a library for automating the intuitive generation of dynamical models using a Category theory-based approach.

QuantumOptics.jl: Simulating quantum systems.

QuantumOptics.jl makes it easy to simulate various kinds of quantum systems. It is inspired by the Quantum Optics Toolbox for MATLAB and the Python framework QuTiP.

diff --git a/dev/highlevels/numerical_utilities/index.html b/dev/highlevels/numerical_utilities/index.html index 34bdc5dbe9c..562a34dd184 100644 --- a/dev/highlevels/numerical_utilities/index.html +++ b/dev/highlevels/numerical_utilities/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

SciML Numerical Utility Libraries

ExponentialUtilities.jl: Faster Matrix Exponentials

ExponentialUtilities.jl is a library for efficient computation of matrix exponentials. While Julia has a built-in exp(A) method, ExponentialUtilities.jl offers many features around this to improve performance in scientific contexts, including:

  • Faster methods for (non-allocating) matrix exponentials via exponential!
  • Methods for computing matrix exponential that are generic to number types and arrays (i.e. GPUs)
  • Methods for computing Arnoldi iterations on Krylov subspaces
  • Direct computation of exp(t*A)*v, i.e. exponentiation of a matrix times a vector, without computing the matrix exponential
  • Direct computation of ϕ_m(t*A)*v operations, where ϕ_0(z) = exp(z) and ϕ_(k+1)(z) = (ϕ_k(z) - 1) / z

ExponentialUtilities.jl includes complex adaptive time stepping techniques such as KIOPS in order to perform these calculations in a fast and numerically-stable way.

QuasiMonteCarlo.jl: Fast Quasi-Random Number Generation

QuasiMonteCarlo.jl is a library for fast generation of low discrepancy Quasi-Monte Carlo samples, using methods like:

  • GridSample(dx) where the grid is given by lb:dx[i]:ub in the ith direction.
  • UniformSample for uniformly distributed random numbers.
  • SobolSample for the Sobol sequence.
  • LatinHypercubeSample for a Latin Hypercube.
  • LatticeRuleSample for a randomly-shifted rank-1 lattice rule.
  • LowDiscrepancySample(base) where base[i] is the base in the ith direction.
  • GoldenSample for a Golden Ratio sequence.
  • KroneckerSample(alpha, s0) for a Kronecker sequence, where alpha is a length-d vector of irrational numbers (often sqrt(d)) and s0 is a length-d seed vector (often 0).
  • SectionSample(x0, sampler) where sampler is any sampler above and x0 is a vector of either NaN for a free dimension or some scalar for a constrained dimension.

PoissonRandom.jl: Fast Poisson Random Number Generation

PoissonRandom.jl is just fast Poisson random number generation for Poisson processes, like chemical master equations.

PreallocationTools.jl: Write Non-Allocating Code Easier

PreallocationTools.jl is a library of tools for writing non-allocating code that interacts well with advanced features like automatic differentiation and symbolics.

RuntimeGeneratedFunctions.jl: Efficient Staged Programming in Julia

RuntimeGeneratedFunctions.jl allows for staged programming in Julia, compiling functions at runtime with full optimizations. This is used by many libraries such as ModelingToolkit.jl to allow for runtime code generation for improved performance.

EllipsisNotation.jl: Implementation of Ellipsis Array Slicing

EllipsisNotation.jl defines the ellipsis array slicing notation for Julia. It uses .. as a catch-all for “all dimensions”, allowing for indexing like [..,1] to mean [:,:,:,1] on four dimensional arrays, in a way that is generic to the number of dimensions in the underlying array.

Third-Party Libraries to Note

Distributions.jl: Representations of Probability Distributions

Distributions.jl is a library for defining distributions in Julia. It's used all throughout the SciML libraries for specifications of probability distributions.

Note

For full compatibility with automatic differentiation, see DistributionsAD.jl

FFTW.jl: Fastest Fourier Transformation in the West

FFTW.jl is the preferred library for fast Fourier Transformations on the CPU.

SpecialFunctions.jl: Implementations of Mathematical Special Functions

SpecialFunctions.jl is a library of implementations of special functions, like Bessel functions and error functions (erf). This library is compatible with automatic differentiation.

LoopVectorization.jl: Automated Loop Accelerator

LoopVectorization.jl is a library which provides the @turbo and @tturbo macros for accelerating the computation of loops. This can be used to accelerating the model functions sent to the equation solvers, for example, accelerating handwritten PDE discretizations.

Polyester.jl: Cheap Threads

Polyester.jl is a cheaper version of threads for Julia, which use a set pool of threads for lower overhead. Note that Polyester does not compose with the standard Julia composable threading infrastructure, and thus one must take care not to compose two levels of Polyester, as this will oversubscribe the computation and lead to performance degradation. Many SciML solvers have options to use Polyester for threading to achieve the top performance.

Tullio.jl: Fast Tensor Calculations and Einstein Notation

Tullio.jl is a library for fast tensor calculations with Einstein notation. It allows for defining operations which are compatible with automatic differentiation, GPUs, and more.

ParallelStencil.jl: High-Level Code for Parallelized Stencil Computations

ParallelStencil.jl is a library for writing high-level code for parallelized stencil computations. It is compatible with SciML equation solvers and is thus a good way to generate GPU and distributed parallel model code.

DataInterpolations.jl: One-Dimensional Interpolations

DataInterpolations.jl is a library of one-dimensional interpolation schemes which are composable with automatic differentiation and the SciML ecosystem. It includes direct interpolation methods and regression techniques for handling noisy data. Its methods include:

  • ConstantInterpolation(u,t) - A piecewise constant interpolation.

  • LinearInterpolation(u,t) - A linear interpolation.

  • QuadraticInterpolation(u,t) - A quadratic interpolation.

  • LagrangeInterpolation(u,t,n) - A Lagrange interpolation of order n.

  • QuadraticSpline(u,t) - A quadratic spline interpolation.

  • CubicSpline(u,t) - A cubic spline interpolation.

  • BSplineInterpolation(u,t,d,pVec,knotVec) - An interpolation B-spline. This is a B-spline which hits each of the data points. The argument choices are:

    • d - degree of B-spline
    • pVec - Symbol to Parameters Vector, pVec = :Uniform for uniform spaced parameters and pVec = :ArcLen for parameters generated by chord length method.
    • knotVec - Symbol to Knot Vector, knotVec = :Uniform for uniform knot vector, knotVec = :Average for average spaced knot vector.

  • BSplineApprox(u,t,d,h,pVec,knotVec) - A regression B-spline which smooths the fitting curve. The argument choices are the same as the BSplineInterpolation, with the additional parameter h<length(t) which is the number of control points to use, with smaller h indicating more smoothing.

  • Curvefit(u,t,m,p,alg) - An interpolation which is done by fitting a user-given functional form m(t,p) where p is the vector of parameters. The user's input p is an initial value for a least-square fitting, alg is the algorithm choice used to optimize the cost function (sum of squared deviations) via Optim.jl and optimal ps are used in the interpolation.

These interpolations match the SciML interfaces and have direct support for packages like ModelingToolkit.jl.

Julia Utilities

StaticCompiler.jl

StaticCompiler.jl is a package for generating static binaries from Julia code. It only supports a subset of Julia, so not all equation solver algorithms are compatible with StaticCompiler.jl.

PackageCompiler.jl

PackageCompiler.jl is a package for generating shared libraries from Julia code. It builds the entirety of Julia by bundling a system image with the Julia runtime. It thus builds complete binaries that can hold all the functionality of SciML. Furthermore, it can also be used to generate new system images to decrease startup times and remove JIT-compilation from SciML usage.

+
Edit on GitHub

SciML Numerical Utility Libraries

ExponentialUtilities.jl: Faster Matrix Exponentials

ExponentialUtilities.jl is a library for efficient computation of matrix exponentials. While Julia has a built-in exp(A) method, ExponentialUtilities.jl offers many features around this to improve performance in scientific contexts, including:

  • Faster methods for (non-allocating) matrix exponentials via exponential!
  • Methods for computing matrix exponential that are generic to number types and arrays (i.e. GPUs)
  • Methods for computing Arnoldi iterations on Krylov subspaces
  • Direct computation of exp(t*A)*v, i.e. exponentiation of a matrix times a vector, without computing the matrix exponential
  • Direct computation of ϕ_m(t*A)*v operations, where ϕ_0(z) = exp(z) and ϕ_(k+1)(z) = (ϕ_k(z) - 1) / z

ExponentialUtilities.jl includes complex adaptive time stepping techniques such as KIOPS in order to perform these calculations in a fast and numerically-stable way.

QuasiMonteCarlo.jl: Fast Quasi-Random Number Generation

QuasiMonteCarlo.jl is a library for fast generation of low discrepancy Quasi-Monte Carlo samples, using methods like:

  • GridSample(dx) where the grid is given by lb:dx[i]:ub in the ith direction.
  • UniformSample for uniformly distributed random numbers.
  • SobolSample for the Sobol sequence.
  • LatinHypercubeSample for a Latin Hypercube.
  • LatticeRuleSample for a randomly-shifted rank-1 lattice rule.
  • LowDiscrepancySample(base) where base[i] is the base in the ith direction.
  • GoldenSample for a Golden Ratio sequence.
  • KroneckerSample(alpha, s0) for a Kronecker sequence, where alpha is a length-d vector of irrational numbers (often sqrt(d)) and s0 is a length-d seed vector (often 0).
  • SectionSample(x0, sampler) where sampler is any sampler above and x0 is a vector of either NaN for a free dimension or some scalar for a constrained dimension.

PoissonRandom.jl: Fast Poisson Random Number Generation

PoissonRandom.jl is just fast Poisson random number generation for Poisson processes, like chemical master equations.

PreallocationTools.jl: Write Non-Allocating Code Easier

PreallocationTools.jl is a library of tools for writing non-allocating code that interacts well with advanced features like automatic differentiation and symbolics.

RuntimeGeneratedFunctions.jl: Efficient Staged Programming in Julia

RuntimeGeneratedFunctions.jl allows for staged programming in Julia, compiling functions at runtime with full optimizations. This is used by many libraries such as ModelingToolkit.jl to allow for runtime code generation for improved performance.

EllipsisNotation.jl: Implementation of Ellipsis Array Slicing

EllipsisNotation.jl defines the ellipsis array slicing notation for Julia. It uses .. as a catch-all for “all dimensions”, allowing for indexing like [..,1] to mean [:,:,:,1] on four dimensional arrays, in a way that is generic to the number of dimensions in the underlying array.

Third-Party Libraries to Note

Distributions.jl: Representations of Probability Distributions

Distributions.jl is a library for defining distributions in Julia. It's used all throughout the SciML libraries for specifications of probability distributions.

Note

For full compatibility with automatic differentiation, see DistributionsAD.jl

FFTW.jl: Fastest Fourier Transformation in the West

FFTW.jl is the preferred library for fast Fourier Transformations on the CPU.

SpecialFunctions.jl: Implementations of Mathematical Special Functions

SpecialFunctions.jl is a library of implementations of special functions, like Bessel functions and error functions (erf). This library is compatible with automatic differentiation.

LoopVectorization.jl: Automated Loop Accelerator

LoopVectorization.jl is a library which provides the @turbo and @tturbo macros for accelerating the computation of loops. This can be used to accelerating the model functions sent to the equation solvers, for example, accelerating handwritten PDE discretizations.

Polyester.jl: Cheap Threads

Polyester.jl is a cheaper version of threads for Julia, which use a set pool of threads for lower overhead. Note that Polyester does not compose with the standard Julia composable threading infrastructure, and thus one must take care not to compose two levels of Polyester, as this will oversubscribe the computation and lead to performance degradation. Many SciML solvers have options to use Polyester for threading to achieve the top performance.

Tullio.jl: Fast Tensor Calculations and Einstein Notation

Tullio.jl is a library for fast tensor calculations with Einstein notation. It allows for defining operations which are compatible with automatic differentiation, GPUs, and more.

ParallelStencil.jl: High-Level Code for Parallelized Stencil Computations

ParallelStencil.jl is a library for writing high-level code for parallelized stencil computations. It is compatible with SciML equation solvers and is thus a good way to generate GPU and distributed parallel model code.

DataInterpolations.jl: One-Dimensional Interpolations

DataInterpolations.jl is a library of one-dimensional interpolation schemes which are composable with automatic differentiation and the SciML ecosystem. It includes direct interpolation methods and regression techniques for handling noisy data. Its methods include:

  • ConstantInterpolation(u,t) - A piecewise constant interpolation.

  • LinearInterpolation(u,t) - A linear interpolation.

  • QuadraticInterpolation(u,t) - A quadratic interpolation.

  • LagrangeInterpolation(u,t,n) - A Lagrange interpolation of order n.

  • QuadraticSpline(u,t) - A quadratic spline interpolation.

  • CubicSpline(u,t) - A cubic spline interpolation.

  • BSplineInterpolation(u,t,d,pVec,knotVec) - An interpolation B-spline. This is a B-spline which hits each of the data points. The argument choices are:

    • d - degree of B-spline
    • pVec - Symbol to Parameters Vector, pVec = :Uniform for uniform spaced parameters and pVec = :ArcLen for parameters generated by chord length method.
    • knotVec - Symbol to Knot Vector, knotVec = :Uniform for uniform knot vector, knotVec = :Average for average spaced knot vector.

  • BSplineApprox(u,t,d,h,pVec,knotVec) - A regression B-spline which smooths the fitting curve. The argument choices are the same as the BSplineInterpolation, with the additional parameter h<length(t) which is the number of control points to use, with smaller h indicating more smoothing.

  • Curvefit(u,t,m,p,alg) - An interpolation which is done by fitting a user-given functional form m(t,p) where p is the vector of parameters. The user's input p is an initial value for a least-square fitting, alg is the algorithm choice used to optimize the cost function (sum of squared deviations) via Optim.jl and optimal ps are used in the interpolation.

These interpolations match the SciML interfaces and have direct support for packages like ModelingToolkit.jl.

Julia Utilities

StaticCompiler.jl

StaticCompiler.jl is a package for generating static binaries from Julia code. It only supports a subset of Julia, so not all equation solver algorithms are compatible with StaticCompiler.jl.

PackageCompiler.jl

PackageCompiler.jl is a package for generating shared libraries from Julia code. It builds the entirety of Julia by bundling a system image with the Julia runtime. It thus builds complete binaries that can hold all the functionality of SciML. Furthermore, it can also be used to generate new system images to decrease startup times and remove JIT-compilation from SciML usage.

diff --git a/dev/highlevels/parameter_analysis/index.html b/dev/highlevels/parameter_analysis/index.html index f3fb15978a5..18fa805c4f6 100644 --- a/dev/highlevels/parameter_analysis/index.html +++ b/dev/highlevels/parameter_analysis/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Parameter Analysis Utilities

GlobalSensitivity.jl: Global Sensitivity Analysis

Derivatives calculate the local sensitivity of a model, i.e. the change in the simulation's outcome if one were to change the parameter with respect to some chosen part of the parameter space. But how does a simulation's output change “in general” with respect to a given parameter? That is what global sensitivity analysis (GSA) computes, and thus GlobalSensitivity.jl is the way to answer that question. GlobalSensitivity.jl includes a wide array of methods, including:

  • Morris's method
  • Sobol's method
  • Regression methods (PCC, SRC, Pearson)
  • eFAST
  • Delta Moment-Independent method
  • Derivative-based Global Sensitivity Measures (DGSM)
  • EASI
  • Fractional Factorial method
  • Random Balance Design FAST method

StructuralIdentifiability.jl: Identifiability Analysis Made Simple

Performing parameter estimation from a data set means attempting to recover parameters like reaction rates by fitting some model to the data. But how do you know whether you have enough data to even consider getting the “correct” parameters back? StructuralIdentifiability.jl allows for running a structural identifiability analysis on a given model to determine whether it's theoretically possible to recover the correct parameters. It can state whether a given type of output data can be used to globally recover the parameters (i.e. only a unique parameter set for the model produces a given output), whether the parameters are only locally identifiable (i.e. there are finitely many parameter sets which could generate the seen data), or whether it's unidentifiable (there are infinitely many parameters which generate the same output data).

For more information on what StructuralIdentifiability.jl is all about, see the SciMLCon 2022 tutorial video.

MinimallyDisruptiveCurves.jl

MinimallyDisruptiveCurves.jl is a library for finding relationships between parameters of models, finding the curves on which the solution is constant.

Third-Party Libraries to Note

SIAN.jl: Structural Identifiability Analyzer

SIAN.jl is a structural identifiability analysis package which uses an entirely different algorithm from StructuralIdentifiability.jl. For information on the differences between the two approaches, see the Structural Identifiability Tools in Julia tutorial.

DynamicalSystems.jl: A Suite of Dynamical Systems Analysis

DynamicalSystems.jl is an entire ecosystem of dynamical systems analysis methods, for computing measures of chaos (dimension estimation, Lyapunov coefficients), generating delay embeddings, and much more. It uses the SciML tools for its internal equation solving and thus shares much of its API, adding a layer of new tools for extended analyses.

For more information, watch the tutorial Introduction to DynamicalSystems.jl.

BifurcationKit.jl

BifurcationKit.jl is a tool for performing bifurcation analysis. It uses and composes with many SciML equation solvers.

ReachabilityAnalysis.jl

ReachabilityAnalysis.jl is a library for performing reachability analysis of dynamical systems, determining for a given uncertainty interval the full set of possible outcomes from a dynamical system.

ControlSystems.jl

ControlSystems.jl is a library for building and analyzing control systems.

+
Edit on GitHub

Parameter Analysis Utilities

GlobalSensitivity.jl: Global Sensitivity Analysis

Derivatives calculate the local sensitivity of a model, i.e. the change in the simulation's outcome if one were to change the parameter with respect to some chosen part of the parameter space. But how does a simulation's output change “in general” with respect to a given parameter? That is what global sensitivity analysis (GSA) computes, and thus GlobalSensitivity.jl is the way to answer that question. GlobalSensitivity.jl includes a wide array of methods, including:

  • Morris's method
  • Sobol's method
  • Regression methods (PCC, SRC, Pearson)
  • eFAST
  • Delta Moment-Independent method
  • Derivative-based Global Sensitivity Measures (DGSM)
  • EASI
  • Fractional Factorial method
  • Random Balance Design FAST method

StructuralIdentifiability.jl: Identifiability Analysis Made Simple

Performing parameter estimation from a data set means attempting to recover parameters like reaction rates by fitting some model to the data. But how do you know whether you have enough data to even consider getting the “correct” parameters back? StructuralIdentifiability.jl allows for running a structural identifiability analysis on a given model to determine whether it's theoretically possible to recover the correct parameters. It can state whether a given type of output data can be used to globally recover the parameters (i.e. only a unique parameter set for the model produces a given output), whether the parameters are only locally identifiable (i.e. there are finitely many parameter sets which could generate the seen data), or whether it's unidentifiable (there are infinitely many parameters which generate the same output data).

For more information on what StructuralIdentifiability.jl is all about, see the SciMLCon 2022 tutorial video.

MinimallyDisruptiveCurves.jl

MinimallyDisruptiveCurves.jl is a library for finding relationships between parameters of models, finding the curves on which the solution is constant.

Third-Party Libraries to Note

SIAN.jl: Structural Identifiability Analyzer

SIAN.jl is a structural identifiability analysis package which uses an entirely different algorithm from StructuralIdentifiability.jl. For information on the differences between the two approaches, see the Structural Identifiability Tools in Julia tutorial.

DynamicalSystems.jl: A Suite of Dynamical Systems Analysis

DynamicalSystems.jl is an entire ecosystem of dynamical systems analysis methods, for computing measures of chaos (dimension estimation, Lyapunov coefficients), generating delay embeddings, and much more. It uses the SciML tools for its internal equation solving and thus shares much of its API, adding a layer of new tools for extended analyses.

For more information, watch the tutorial Introduction to DynamicalSystems.jl.

BifurcationKit.jl

BifurcationKit.jl is a tool for performing bifurcation analysis. It uses and composes with many SciML equation solvers.

ReachabilityAnalysis.jl

ReachabilityAnalysis.jl is a library for performing reachability analysis of dynamical systems, determining for a given uncertainty interval the full set of possible outcomes from a dynamical system.

ControlSystems.jl

ControlSystems.jl is a library for building and analyzing control systems.

diff --git a/dev/highlevels/partial_differential_equation_solvers/index.html b/dev/highlevels/partial_differential_equation_solvers/index.html index fb95b61cfc8..9fc7b2fe498 100644 --- a/dev/highlevels/partial_differential_equation_solvers/index.html +++ b/dev/highlevels/partial_differential_equation_solvers/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Partial Differential Equations (PDE)

NeuralPDE.jl: Physics-Informed Neural Network (PINN) PDE Solvers

NeuralPDE.jl is a partial differential equation solver library which uses physics-informed neural networks (PINNs) to solve the equations. It uses the ModelingToolkit.jl symbolic PDESystem as its input and can handle a wide variety of equation types, including systems of partial differential equations, partial differential-algebraic equations, and integro-differential equations. Its benefit is its flexibility, and it can be used to easily generate surrogate solutions over entire parameter ranges. However, its downside is solver speed: PINN solvers tend to be a lot slower than other methods for solving PDEs.

MethodOflines.jl: Automated Finite Difference Method (FDM)

MethodOflines.jl is a partial differential equation solver library which automates the discretization of PDEs via the finite difference method. It uses the ModelingToolkit.jl symbolic PDESystem as its input, and generates AbstractSystems and SciMLProblems whose numerical solution gives the solution to the PDE.

FEniCS.jl: Wrappers for the Finite Element Method (FEM)

FEniCS.jl is a wrapper for the popular FEniCS finite element method library.

HighDimPDE.jl: High-dimensional PDE Solvers

HighDimPDE.jl is a partial differential equation solver library which implements algorithms that break down the curse of dimensionality to solve the equations. It implements deep-learning based and Picard-iteration based methods to approximately solve high-dimensional, nonlinear, non-local PDEs in up to 10,000 dimensions. Its cons are accuracy: high-dimensional solvers are stochastic, and might result in wrong solutions if the solver meta-parameters are not appropriate.

NeuralOperators.jl: (Fourier) Neural Operators and DeepONets for PDE Solving

NeuralOperators.jl is a library for operator learning based PDE solvers. This includes techniques like:

  • Fourier Neural Operators (FNO)
  • Deep Operator Networks (DeepONets)
  • Markov Neural Operators (MNO)

Currently, its connection to PDE solving must be specified manually, though an interface for ModelingToolkit PDESystems is in progress.

DiffEqOperators.jl: Operators for Finite Difference Method (FDM) Discretizations

DiffEqOperators.jl is a library for defining finite difference operators to easily perform manual FDM semi-discretizations of partial differential equations. This library is fairly incomplete and most cases should receive better performance using MethodOflines.jl.

Third-Party Libraries to Note

ApproxFun.jl: Automated Spectral Discretizations

ApproxFun.jl is a package for approximating functions in basis sets. One particular use case is with spectral basis sets, such as Chebyshev functions and Fourier decompositions, making it easy to represent spectral and pseudospectral discretizations of partial differential equations as ordinary differential equations for the SciML equation solvers.

Gridap.jl: Julia-Based Tools for Finite Element Discretizations

Gridap.jl is a package for grid-based approximation of partial differential equations, particularly notable for its use of conforming and nonconforming finite element (FEM) discretizations.

Trixi.jl: Adaptive High-Order Numerical Simulations of Hyperbolic Equations

Trixi.jl is a package for numerical simulation of hyperbolic conservation laws, i.e. a large set of hyperbolic partial differential equations, which interfaces and uses the SciML ordinary differential equation solvers.

VoronoiFVM.jl: Tools for the Voronoi Finite Volume Discretizations

VoronoiFVM.jl is a library for generating FVM discretizations of systems of PDEs. It interfaces with many of the SciML equation solver libraries to allow for ease of discretization and flexibility in the solver choice.

+
Edit on GitHub

Partial Differential Equations (PDE)

NeuralPDE.jl: Physics-Informed Neural Network (PINN) PDE Solvers

NeuralPDE.jl is a partial differential equation solver library which uses physics-informed neural networks (PINNs) to solve the equations. It uses the ModelingToolkit.jl symbolic PDESystem as its input and can handle a wide variety of equation types, including systems of partial differential equations, partial differential-algebraic equations, and integro-differential equations. Its benefit is its flexibility, and it can be used to easily generate surrogate solutions over entire parameter ranges. However, its downside is solver speed: PINN solvers tend to be a lot slower than other methods for solving PDEs.

MethodOflines.jl: Automated Finite Difference Method (FDM)

MethodOflines.jl is a partial differential equation solver library which automates the discretization of PDEs via the finite difference method. It uses the ModelingToolkit.jl symbolic PDESystem as its input, and generates AbstractSystems and SciMLProblems whose numerical solution gives the solution to the PDE.

FEniCS.jl: Wrappers for the Finite Element Method (FEM)

FEniCS.jl is a wrapper for the popular FEniCS finite element method library.

HighDimPDE.jl: High-dimensional PDE Solvers

HighDimPDE.jl is a partial differential equation solver library which implements algorithms that break down the curse of dimensionality to solve the equations. It implements deep-learning based and Picard-iteration based methods to approximately solve high-dimensional, nonlinear, non-local PDEs in up to 10,000 dimensions. Its cons are accuracy: high-dimensional solvers are stochastic, and might result in wrong solutions if the solver meta-parameters are not appropriate.

NeuralOperators.jl: (Fourier) Neural Operators and DeepONets for PDE Solving

NeuralOperators.jl is a library for operator learning based PDE solvers. This includes techniques like:

  • Fourier Neural Operators (FNO)
  • Deep Operator Networks (DeepONets)
  • Markov Neural Operators (MNO)

Currently, its connection to PDE solving must be specified manually, though an interface for ModelingToolkit PDESystems is in progress.

DiffEqOperators.jl: Operators for Finite Difference Method (FDM) Discretizations

DiffEqOperators.jl is a library for defining finite difference operators to easily perform manual FDM semi-discretizations of partial differential equations. This library is fairly incomplete and most cases should receive better performance using MethodOflines.jl.

Third-Party Libraries to Note

ApproxFun.jl: Automated Spectral Discretizations

ApproxFun.jl is a package for approximating functions in basis sets. One particular use case is with spectral basis sets, such as Chebyshev functions and Fourier decompositions, making it easy to represent spectral and pseudospectral discretizations of partial differential equations as ordinary differential equations for the SciML equation solvers.

Gridap.jl: Julia-Based Tools for Finite Element Discretizations

Gridap.jl is a package for grid-based approximation of partial differential equations, particularly notable for its use of conforming and nonconforming finite element (FEM) discretizations.

Trixi.jl: Adaptive High-Order Numerical Simulations of Hyperbolic Equations

Trixi.jl is a package for numerical simulation of hyperbolic conservation laws, i.e. a large set of hyperbolic partial differential equations, which interfaces and uses the SciML ordinary differential equation solvers.

VoronoiFVM.jl: Tools for the Voronoi Finite Volume Discretizations

VoronoiFVM.jl is a library for generating FVM discretizations of systems of PDEs. It interfaces with many of the SciML equation solver libraries to allow for ease of discretization and flexibility in the solver choice.

diff --git a/dev/highlevels/plots_visualization/index.html b/dev/highlevels/plots_visualization/index.html index d609fa2cde9..47a48400be7 100644 --- a/dev/highlevels/plots_visualization/index.html +++ b/dev/highlevels/plots_visualization/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

SciML-Supported Plotting and Visualization Libraries

The following libraries are the plotting and visualization libraries which are supported and co-developed by the SciML developers. Other libraries may be used, though these are the libraries used in the tutorials and which have special hooks to ensure ergonomic usage with SciML tooling.

Plots.jl

Plots.jl is the current standard plotting system for the SciML ecosystem. SciML types attempt to include plot recipes for as many types as possible, allowing for automatic visualization with the Plots.jl system. All current tutorials and documentation default to using Plots.jl.

Makie.jl

Makie.jl is a high-performance interactive plotting system for the Julia programming language. It's planned to be the default plotting system used by the SciML organization in the near future.

+
Edit on GitHub

SciML-Supported Plotting and Visualization Libraries

The following libraries are the plotting and visualization libraries which are supported and co-developed by the SciML developers. Other libraries may be used, though these are the libraries used in the tutorials and which have special hooks to ensure ergonomic usage with SciML tooling.

Plots.jl

Plots.jl is the current standard plotting system for the SciML ecosystem. SciML types attempt to include plot recipes for as many types as possible, allowing for automatic visualization with the Plots.jl system. All current tutorials and documentation default to using Plots.jl.

Makie.jl

Makie.jl is a high-performance interactive plotting system for the Julia programming language. It's planned to be the default plotting system used by the SciML organization in the near future.

diff --git a/dev/highlevels/symbolic_learning/index.html b/dev/highlevels/symbolic_learning/index.html index 78a61c45a9d..3423e0d5a3c 100644 --- a/dev/highlevels/symbolic_learning/index.html +++ b/dev/highlevels/symbolic_learning/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Symbolic Learning and Artificial Intelligence

Symbolic learning, the classical artificial intelligence, is a set of methods for learning symbolic equations from data and numerical functions. SciML offers an array of symbolic learning utilities which connect with the other machine learning and equation solver functionalities to make it easy to embed prior knowledge and discover missing physics. For more information, see Universal Differential Equations for Scientific Machine Learning.

DataDrivenDiffEq.jl: Data-Driven Modeling and Automated Discovery of Dynamical Systems

DataDrivenDiffEq.jl is a general interface for data-driven modeling, containing a large array of techniques such as:

  • Koopman operator methods (Dynamic-Mode Decomposition (DMD) and variations)
  • Sparse Identification of Dynamical Systems (SINDy and variations like iSINDy)
  • Sparse regression methods (STSLQ, SR3, etc.)
  • PDEFind
  • Wrappers for SymbolicRegression.jl
  • AI Feynman
  • OccamNet

SymbolicNumericIntegration.jl: Symbolic Integration via Numerical Methods

SymbolicNumericIntegration.jl is a package computing the solution to symbolic integration problem using numerical methods (numerical integration mixed with sparse regression).

Third-Party Libraries to Note

SymbolicRegression.jl

SymbolicRegression.jl is a symbolic regression library which uses genetic algorithms with parallelization to achieve fast and robust symbolic learning.

+
Edit on GitHub

Symbolic Learning and Artificial Intelligence

Symbolic learning, the classical artificial intelligence, is a set of methods for learning symbolic equations from data and numerical functions. SciML offers an array of symbolic learning utilities which connect with the other machine learning and equation solver functionalities to make it easy to embed prior knowledge and discover missing physics. For more information, see Universal Differential Equations for Scientific Machine Learning.

DataDrivenDiffEq.jl: Data-Driven Modeling and Automated Discovery of Dynamical Systems

DataDrivenDiffEq.jl is a general interface for data-driven modeling, containing a large array of techniques such as:

  • Koopman operator methods (Dynamic-Mode Decomposition (DMD) and variations)
  • Sparse Identification of Dynamical Systems (SINDy and variations like iSINDy)
  • Sparse regression methods (STSLQ, SR3, etc.)
  • PDEFind
  • Wrappers for SymbolicRegression.jl
  • AI Feynman
  • OccamNet

SymbolicNumericIntegration.jl: Symbolic Integration via Numerical Methods

SymbolicNumericIntegration.jl is a package computing the solution to symbolic integration problem using numerical methods (numerical integration mixed with sparse regression).

Third-Party Libraries to Note

SymbolicRegression.jl

SymbolicRegression.jl is a symbolic regression library which uses genetic algorithms with parallelization to achieve fast and robust symbolic learning.

diff --git a/dev/highlevels/symbolic_tools/index.html b/dev/highlevels/symbolic_tools/index.html index a5ecb7ab522..67ed1fef6b5 100644 --- a/dev/highlevels/symbolic_tools/index.html +++ b/dev/highlevels/symbolic_tools/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Symbolic Model Tooling and JuliaSymbolics

JuliaSymbolics is a sister organization of SciML. It spawned out of the symbolic modeling tools being developed within SciML (ModelingToolkit.jl) to become its own organization dedicated to building a fully-featured Julia-based Computer Algebra System (CAS). As such, the two organizations are closely aligned in terms of its developer community, and many of the SciML libraries use Symbolics.jl extensively.

ModelOrderReduction.jl: Automated Model Reduction for Fast Approximations of Solutions

ModelOrderReduction.jl is a package for automating the reduction of models. These methods function a submodel with a projection, where solving the smaller model provides approximation information about the full model. MOR.jl uses ModelingToolkit.jl as a system description and automatically transforms equations to the subform, defining the observables to automatically lazily reconstruct the full model on-demand in a fast and stable form.

Symbolics.jl: The Computer Algebra System (CAS) of the Julia Programming Language

Symbolics.jl is the CAS of the Julia programming language. If something needs to be done symbolically, most likely Symbolics.jl is the answer.

MetaTheory.jl: E-Graphs to Automate Symbolic Transformations

Metatheory.jl is a library for defining e-graph rewriters for use on the common symbolic interface. This can be used to do all sorts of analysis and code transformations, such as improving code performance, numerical stability, and more. See Automated Code Optimization with E-Graphs for more details.

SymbolicUtils.jl: Define Your Own Computer Algebra System

SymbolicUtils.jl is the underlying utility library and rule-based rewriting language on which Symbolics.jl is developed. Symbolics.jl is standardized type and rule definitions built using SymbolicUtils.jl. However, if non-standard types are required, such as symbolic computing over Fock algebras, then SymbolicUtils.jl is the library from which the new symbolic types can be implemented.

+
Edit on GitHub

Symbolic Model Tooling and JuliaSymbolics

JuliaSymbolics is a sister organization of SciML. It spawned out of the symbolic modeling tools being developed within SciML (ModelingToolkit.jl) to become its own organization dedicated to building a fully-featured Julia-based Computer Algebra System (CAS). As such, the two organizations are closely aligned in terms of its developer community, and many of the SciML libraries use Symbolics.jl extensively.

ModelOrderReduction.jl: Automated Model Reduction for Fast Approximations of Solutions

ModelOrderReduction.jl is a package for automating the reduction of models. These methods function a submodel with a projection, where solving the smaller model provides approximation information about the full model. MOR.jl uses ModelingToolkit.jl as a system description and automatically transforms equations to the subform, defining the observables to automatically lazily reconstruct the full model on-demand in a fast and stable form.

Symbolics.jl: The Computer Algebra System (CAS) of the Julia Programming Language

Symbolics.jl is the CAS of the Julia programming language. If something needs to be done symbolically, most likely Symbolics.jl is the answer.

MetaTheory.jl: E-Graphs to Automate Symbolic Transformations

Metatheory.jl is a library for defining e-graph rewriters for use on the common symbolic interface. This can be used to do all sorts of analysis and code transformations, such as improving code performance, numerical stability, and more. See Automated Code Optimization with E-Graphs for more details.

SymbolicUtils.jl: Define Your Own Computer Algebra System

SymbolicUtils.jl is the underlying utility library and rule-based rewriting language on which Symbolics.jl is developed. Symbolics.jl is standardized type and rule definitions built using SymbolicUtils.jl. However, if non-standard types are required, such as symbolic computing over Fock algebras, then SymbolicUtils.jl is the library from which the new symbolic types can be implemented.

diff --git a/dev/highlevels/uncertainty_quantification/index.html b/dev/highlevels/uncertainty_quantification/index.html index 80dc9c4562b..ed2ec8c3cb5 100644 --- a/dev/highlevels/uncertainty_quantification/index.html +++ b/dev/highlevels/uncertainty_quantification/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Uncertainty Quantification

There's always uncertainty in our models. Whether it's in the form of the model's equations or in the model's parameters, the uncertainty in our simulation's output often needs to be quantified. The following tools automate this process.

For Measurements.jl vs MonteCarloMeasurements.jl vs Intervals.jl, and the relation to other methods, see the Uncertainty Programming chapter of the SciML Book.

PolyChaos.jl: Intrusive Polynomial Chaos Expansions Made Unintrusive

PolyChaos.jl is a library for calculating intrusive polynomial chaos expansions (PCE) on arbitrary Julia functions. This allows for inputting representations of probability distributions into functions to compute the output distribution in an expansion representation. While normally this would require deriving the PCE-expanded equations by hand, PolyChaos.jl does this at the compiler level using Julia's multiple dispatch, giving a high-performance implementation to a normally complex and tedious mathematical transformation.

SciMLExpectations.jl: Fast Calculations of Expectations of Equation Solutions

SciMLExpectations.jl is a library for accelerating the calculation of expectations of equation solutions with respect to input probability distributions, allowing for applications like robust optimization with respect to uncertainty. It uses Koopman operator techniques to calculate these expectations without requiring the propagation of uncertainties through a solver, effectively performing the adjoint of uncertainty quantification and being much more efficient in the process.

Third-Party Libraries to Note

Measurements.jl: Automated Linear Error Propagation

Measurements.jl is a library for automating linear error propagation. Uncertain numbers are defined as x = 3.8 ± 0.4 and are pushed through calculations using a normal distribution approximation in order to compute an approximate uncertain output. Measurements.jl uses a dictionary-based approach to keep track of correlations to improve the accuracy over naive implementations, though note that linear error propagation theory still has some major issues handling some types of equations, as described in detail in the MonteCarloMeasurements.jl documentation.

MonteCarloMeasurements.jl: Automated Monte Carlo Error Propagation

MonteCarloMeasurements.jl is a library for automating the uncertainty quantification of equation solution using Monte Carlo methods. It defines number types which sample from an input distribution to receive a representative set of parameters that propagate through the solver to calculate a representative set of possible solutions. Note that Monte Carlo techniques can be expensive but are exact, in the sense that as the number of sample points increases to infinity it will compute a correct approximation of the output uncertainty.

ProbNumDiffEq.jl: Probabilistic Numerics Based Differential Equation Solvers

ProbNumDiffEq.jl is a set of probabilistic numerical ODE solvers which compute the solution of a differential equation along with a posterior distribution to estimate its numerical approximation error. Thus these specialized integrators compute an uncertainty output similar to the ProbInts technique of DiffEqUncertainty, but use specialized integration techniques in order to do it much faster for specific kinds of equations.

TaylorIntegration.jl: Taylor Series Integration for Rigorous Numerical Bounds

TaylorIntegration.jl is a library for Taylor series integrators, which has special functionality for computing the interval bound of possible solutions with respect to numerical approximation error.

IntervalArithmetic.jl: Rigorous Numerical Intervals

IntervalArithmetic.jl is a library for performing interval arithmetic calculations on arbitrary Julia code. Interval arithmetic computes rigorous computations with respect to finite-precision floating-point arithmetic, i.e. its intervals are guaranteed to include the true solution. However, interval arithmetic intervals can grow at exponential rates in many problems, thus being unsuitable for analyses in many equation solver contexts.

+
Edit on GitHub

Uncertainty Quantification

There's always uncertainty in our models. Whether it's in the form of the model's equations or in the model's parameters, the uncertainty in our simulation's output often needs to be quantified. The following tools automate this process.

For Measurements.jl vs MonteCarloMeasurements.jl vs Intervals.jl, and the relation to other methods, see the Uncertainty Programming chapter of the SciML Book.

PolyChaos.jl: Intrusive Polynomial Chaos Expansions Made Unintrusive

PolyChaos.jl is a library for calculating intrusive polynomial chaos expansions (PCE) on arbitrary Julia functions. This allows for inputting representations of probability distributions into functions to compute the output distribution in an expansion representation. While normally this would require deriving the PCE-expanded equations by hand, PolyChaos.jl does this at the compiler level using Julia's multiple dispatch, giving a high-performance implementation to a normally complex and tedious mathematical transformation.

SciMLExpectations.jl: Fast Calculations of Expectations of Equation Solutions

SciMLExpectations.jl is a library for accelerating the calculation of expectations of equation solutions with respect to input probability distributions, allowing for applications like robust optimization with respect to uncertainty. It uses Koopman operator techniques to calculate these expectations without requiring the propagation of uncertainties through a solver, effectively performing the adjoint of uncertainty quantification and being much more efficient in the process.

Third-Party Libraries to Note

Measurements.jl: Automated Linear Error Propagation

Measurements.jl is a library for automating linear error propagation. Uncertain numbers are defined as x = 3.8 ± 0.4 and are pushed through calculations using a normal distribution approximation in order to compute an approximate uncertain output. Measurements.jl uses a dictionary-based approach to keep track of correlations to improve the accuracy over naive implementations, though note that linear error propagation theory still has some major issues handling some types of equations, as described in detail in the MonteCarloMeasurements.jl documentation.

MonteCarloMeasurements.jl: Automated Monte Carlo Error Propagation

MonteCarloMeasurements.jl is a library for automating the uncertainty quantification of equation solution using Monte Carlo methods. It defines number types which sample from an input distribution to receive a representative set of parameters that propagate through the solver to calculate a representative set of possible solutions. Note that Monte Carlo techniques can be expensive but are exact, in the sense that as the number of sample points increases to infinity it will compute a correct approximation of the output uncertainty.

ProbNumDiffEq.jl: Probabilistic Numerics Based Differential Equation Solvers

ProbNumDiffEq.jl is a set of probabilistic numerical ODE solvers which compute the solution of a differential equation along with a posterior distribution to estimate its numerical approximation error. Thus these specialized integrators compute an uncertainty output similar to the ProbInts technique of DiffEqUncertainty, but use specialized integration techniques in order to do it much faster for specific kinds of equations.

TaylorIntegration.jl: Taylor Series Integration for Rigorous Numerical Bounds

TaylorIntegration.jl is a library for Taylor series integrators, which has special functionality for computing the interval bound of possible solutions with respect to numerical approximation error.

IntervalArithmetic.jl: Rigorous Numerical Intervals

IntervalArithmetic.jl is a library for performing interval arithmetic calculations on arbitrary Julia code. Interval arithmetic computes rigorous computations with respect to finite-precision floating-point arithmetic, i.e. its intervals are guaranteed to include the true solution. However, interval arithmetic intervals can grow at exponential rates in many problems, thus being unsuitable for analyses in many equation solver contexts.

diff --git a/dev/index.html b/dev/index.html index 3906fd947f5..2f633d29c8d 100644 --- a/dev/index.html +++ b/dev/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

SciML: Differentiable Modeling and Simulation Combined with Machine Learning

The SciML organization is a collection of tools for solving equations and modeling systems developed in the Julia programming language with bindings to other languages such as R and Python. The organization provides well-maintained tools which compose together as a coherent ecosystem. It has a coherent development principle, unified APIs over large collections of equation solvers, pervasive differentiability and sensitivity analysis, and features many of the highest performance and parallel implementations one can find.

Scientific Machine Learning (SciML) = Scientific Computing + Machine Learning

Where to Start?

And for diving into the details, use the bar on the top to navigate to the submodule of interest!

+
Edit on GitHub

SciML: Differentiable Modeling and Simulation Combined with Machine Learning

The SciML organization is a collection of tools for solving equations and modeling systems developed in the Julia programming language with bindings to other languages such as R and Python. The organization provides well-maintained tools which compose together as a coherent ecosystem. It has a coherent development principle, unified APIs over large collections of equation solvers, pervasive differentiability and sensitivity analysis, and features many of the highest performance and parallel implementations one can find.

Scientific Machine Learning (SciML) = Scientific Computing + Machine Learning

Where to Start?

And for diving into the details, use the bar on the top to navigate to the submodule of interest!

diff --git a/dev/overview/index.html b/dev/overview/index.html index 54d63f77570..92bd29e87b1 100644 --- a/dev/overview/index.html +++ b/dev/overview/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -
Edit on GitHub

Detailed Overview of the SciML Software Ecosystem

SciML: Combining High-Performance Scientific Computing and Machine Learning

SciML is not standard machine learning, SciML is the combination of scientific computing techniques with machine learning. Thus the SciML organization is not an organization for machine learning libraries (see FluxML for machine learning in Julia), rather SciML is an organization dedicated to the development of scientific computing tools which work seamlessly in conjunction with next-generation machine learning workflows. This includes:

  • High-performance and accurate tools for standard scientific computing modeling and simulation
  • Compatibility with differentiable programming and automatic differentiation
  • Tools for building complex multiscale models
  • Methods for handling inverse problems, model calibration, controls, and Bayesian analysis
  • Symbolic modeling tools for generating efficient code for numerical equation solvers
  • Methods for automatic discovery of (bio)physical equations

and much more. For an overview of the broad goals of the SciML organization, watch:

Overview of Computational Science in Julia with SciML

Below is a simplification of the user-facing packages for use in scientific computing and SciML workflows.

Workflow ElementSciML-Supported Julia packages
Plotting and VisualizationPlots*, Makie*
Sparse matrixSparseArrays*
Interpolation/approximationDataInterpolations*, ApproxFun*
Linear system / least squaresLinearSolve
Nonlinear system / rootfindingNonlinearSolve
Polynomial rootsPolynomials*
IntegrationIntegrals
Nonlinear OptimizationOptimization
Other Optimization (linear, quadratic, convex, etc.)JuMP*
Initial-value problemDifferentialEquations
Boundary-value problemDifferentialEquations
Continuous-Time Markov Chains (Poisson Jumps), Jump DiffusionsJumpProcesses
Finite differencesFiniteDifferences*, FiniteDiff*
Automatic DifferentiationForwardDiff*, Enzyme*, DiffEqSensitivity
Bayesian ModelingTuring*
Deep LearningFlux*
Acausal Modeling / DAEsModelingToolkit
Chemical Reaction NetworksCatalyst
Symbolic ComputingSymbolics
Fast Fourier TransformFFTW*
Partial Differential Equation DiscretizationsAssociated Julia packages
–-–-
Finite DifferencesMethodOfLines
Discontinuous GalerkinTrixi*
Finite ElementGridap*
Physics-Informed Neural NetworksNeuralPDE
Neural OperatorsNeuralOperators
High Dimensional Deep LearningHighDimPDE

* Denotes a non-SciML package that is heavily tested against as part of SciML workflows and has frequent collaboration with the SciML developers.

SciML Mind Map

Domains of SciML

The SciML common interface covers the following domains:

  • Linear systems (LinearProblem)

    • Direct methods for dense and sparse
    • Iterative solvers with preconditioning

  • Nonlinear Systems (NonlinearProblem)

    • Systems of nonlinear equations
    • Scalar bracketing systems

  • Integrals (quadrature) (IntegralProblem)

  • Differential Equations

    • Discrete equations (function maps, discrete stochastic (Gillespie/Markov) simulations) (DiscreteProblem and JumpProblem)
    • Ordinary differential equations (ODEs) (ODEProblem)
    • Split and Partitioned ODEs (Symplectic integrators, IMEX Methods) (SplitODEProblem)
    • Stochastic ordinary differential equations (SODEs or SDEs) (SDEProblem)
    • Stochastic differential-algebraic equations (SDAEs) (SDEProblem with mass matrices)
    • Random differential equations (RODEs or RDEs) (RODEProblem)
    • Differential algebraic equations (DAEs) (DAEProblem and ODEProblem with mass matrices)
    • Delay differential equations (DDEs) (DDEProblem)
    • Neutral, retarded, and algebraic delay differential equations (NDDEs, RDDEs, and DDAEs)
    • Stochastic delay differential equations (SDDEs) (SDDEProblem)
    • Experimental support for stochastic neutral, retarded, and algebraic delay differential equations (SNDDEs, SRDDEs, and SDDAEs)
    • Mixed discrete and continuous equations (Hybrid Equations, Jump Diffusions) (DEProblems with callbacks and JumpProblem)

  • Optimization (OptimizationProblem)

    • Nonlinear (constrained) optimization

  • (Stochastic/Delay/Differential-Algebraic) Partial Differential Equations (PDESystem)

    • Finite difference and finite volume methods
    • Interfaces to finite element methods
    • Physics-Informed Neural Networks (PINNs)
    • Integro-Differential Equations
    • Fractional Differential Equations

  • Specialized Forms

    • Partial Integro-Differential Equations (PIPDEProblem)

  • Data-driven modeling

    • Discrete-time data-driven dynamical systems (DiscreteDataDrivenProblem)
    • Continuous-time data-driven dynamical systems (ContinuousDataDrivenProblem)
    • Symbolic regression (DirectDataDrivenProblem)

  • Uncertainty quantification and expected values (ExpectationProblem)

The SciML common interface also includes ModelingToolkit.jl for defining such systems symbolically, allowing for optimizations like automated generation of parallel code, symbolic simplification, and generation of sparsity patterns.

Inverse Problems, Parameter Estimation, and Structural Identification

Parameter estimation and inverse problems are solved directly on their constituent problem types using tools like SciMLSensitivity.jl. Thus for example, there is no ODEInverseProblem, and instead ODEProblem is used to find the parameters p that solve the inverse problem. Check out the SciMLSensitivity documentation for a discussion on connections to automatic differentiation, optimization, and adjoints.

Common Interface High-Level Overview

The SciML interface is common as the usage of arguments is standardized across all of the problem domains. Underlying high-level ideas include:

  • All domains use the same interface of defining a AbstractSciMLProblem which is then solved via solve(prob,alg;kwargs), where alg is a AbstractSciMLAlgorithm. The keyword argument namings are standardized across the organization.
  • AbstractSciMLProblems are generally defined by a AbstractSciMLFunction which can define extra details about a model function, such as its analytical Jacobian, its sparsity patterns and so on.
  • There is an organization-wide method for defining linear and nonlinear solvers used within other solvers, giving maximum control of performance to the user.
  • Types used within the packages are defined by the input types. For example, packages attempt to internally use the type of the initial condition as the type for the state within differential equation solvers.
  • solve calls should be thread-safe and parallel-safe.
  • init(prob,alg;kwargs) returns an iterator which allows for directly iterating over the solution process
  • High performance is key. Any performance that is not at the top level is considered a bug and should be reported as such.
  • All functions have an in-place and out-of-place form, where the in-place form is made to utilize mutation for high performance on large-scale problems and the out-of-place form is for compatibility with tooling like static arrays and some reverse-mode automatic differentiation systems.

Flowchart Example for PDE-Constrained Optimal Control

The following example showcases how the pieces of the common interface connect to solve a problem that mixes inference, symbolics, and numerics.

External Binding Libraries

  • diffeqr

    • Solving differential equations in R using DifferentialEquations.jl with ModelingToolkit for JIT compilation and GPU-acceleration

  • diffeqpy

    • Solving differential equations in Python using DifferentialEquations.jl

Note About Third-Party Libraries

The SciML documentation references and recommends many third-party libraries for improving ones modeling, simulation, and analysis workflow in Julia. Take these as a positive affirmation of the quality of these libraries, as these libraries are commonly tested by SciML developers who are in contact with the development teams of these groups. It also documents the libraries which are commonly chosen by SciML as dependencies. Do not take omissions as negative affirmations against a given library, i.e. a library left off of the list by SciML is not a negative endorsement. Rather, it means that compatibility with SciML is untested, SciML developers may have a personal preference for another choice, or SciML developers may be simply unaware of the library's existence. If one would like to add a third-party library to the SciML documentation, open a pull request with the requested text.

Note that the libraries in this documentation are only those that are meant to be used in the SciML extended universe of modeling, simulation, and analysis and thus there are many high-quality libraries in other domains (machine learning, data science, etc.) which are purposefully not included. For an overview of the Julia package ecosystem, see the JuliaHub Search Engine.

+
Edit on GitHub

Detailed Overview of the SciML Software Ecosystem

SciML: Combining High-Performance Scientific Computing and Machine Learning

SciML is not standard machine learning, SciML is the combination of scientific computing techniques with machine learning. Thus the SciML organization is not an organization for machine learning libraries (see FluxML for machine learning in Julia), rather SciML is an organization dedicated to the development of scientific computing tools which work seamlessly in conjunction with next-generation machine learning workflows. This includes:

  • High-performance and accurate tools for standard scientific computing modeling and simulation
  • Compatibility with differentiable programming and automatic differentiation
  • Tools for building complex multiscale models
  • Methods for handling inverse problems, model calibration, controls, and Bayesian analysis
  • Symbolic modeling tools for generating efficient code for numerical equation solvers
  • Methods for automatic discovery of (bio)physical equations

and much more. For an overview of the broad goals of the SciML organization, watch:

Overview of Computational Science in Julia with SciML

Below is a simplification of the user-facing packages for use in scientific computing and SciML workflows.

Workflow ElementSciML-Supported Julia packages
Plotting and VisualizationPlots*, Makie*
Sparse matrixSparseArrays*
Interpolation/approximationDataInterpolations*, ApproxFun*
Linear system / least squaresLinearSolve
Nonlinear system / rootfindingNonlinearSolve
Polynomial rootsPolynomials*
IntegrationIntegrals
Nonlinear OptimizationOptimization
Other Optimization (linear, quadratic, convex, etc.)JuMP*
Initial-value problemDifferentialEquations
Boundary-value problemDifferentialEquations
Continuous-Time Markov Chains (Poisson Jumps), Jump DiffusionsJumpProcesses
Finite differencesFiniteDifferences*, FiniteDiff*
Automatic DifferentiationForwardDiff*, Enzyme*, DiffEqSensitivity
Bayesian ModelingTuring*
Deep LearningFlux*
Acausal Modeling / DAEsModelingToolkit
Chemical Reaction NetworksCatalyst
Symbolic ComputingSymbolics
Fast Fourier TransformFFTW*
Partial Differential Equation DiscretizationsAssociated Julia packages
–-–-
Finite DifferencesMethodOfLines
Discontinuous GalerkinTrixi*
Finite ElementGridap*
Physics-Informed Neural NetworksNeuralPDE
Neural OperatorsNeuralOperators
High Dimensional Deep LearningHighDimPDE

* Denotes a non-SciML package that is heavily tested against as part of SciML workflows and has frequent collaboration with the SciML developers.

SciML Mind Map

Domains of SciML

The SciML common interface covers the following domains:

  • Linear systems (LinearProblem)

    • Direct methods for dense and sparse
    • Iterative solvers with preconditioning

  • Nonlinear Systems (NonlinearProblem)

    • Systems of nonlinear equations
    • Scalar bracketing systems

  • Integrals (quadrature) (IntegralProblem)

  • Differential Equations

    • Discrete equations (function maps, discrete stochastic (Gillespie/Markov) simulations) (DiscreteProblem and JumpProblem)
    • Ordinary differential equations (ODEs) (ODEProblem)
    • Split and Partitioned ODEs (Symplectic integrators, IMEX Methods) (SplitODEProblem)
    • Stochastic ordinary differential equations (SODEs or SDEs) (SDEProblem)
    • Stochastic differential-algebraic equations (SDAEs) (SDEProblem with mass matrices)
    • Random differential equations (RODEs or RDEs) (RODEProblem)
    • Differential algebraic equations (DAEs) (DAEProblem and ODEProblem with mass matrices)
    • Delay differential equations (DDEs) (DDEProblem)
    • Neutral, retarded, and algebraic delay differential equations (NDDEs, RDDEs, and DDAEs)
    • Stochastic delay differential equations (SDDEs) (SDDEProblem)
    • Experimental support for stochastic neutral, retarded, and algebraic delay differential equations (SNDDEs, SRDDEs, and SDDAEs)
    • Mixed discrete and continuous equations (Hybrid Equations, Jump Diffusions) (DEProblems with callbacks and JumpProblem)

  • Optimization (OptimizationProblem)

    • Nonlinear (constrained) optimization

  • (Stochastic/Delay/Differential-Algebraic) Partial Differential Equations (PDESystem)

    • Finite difference and finite volume methods
    • Interfaces to finite element methods
    • Physics-Informed Neural Networks (PINNs)
    • Integro-Differential Equations
    • Fractional Differential Equations

  • Specialized Forms

    • Partial Integro-Differential Equations (PIPDEProblem)

  • Data-driven modeling

    • Discrete-time data-driven dynamical systems (DiscreteDataDrivenProblem)
    • Continuous-time data-driven dynamical systems (ContinuousDataDrivenProblem)
    • Symbolic regression (DirectDataDrivenProblem)

  • Uncertainty quantification and expected values (ExpectationProblem)

The SciML common interface also includes ModelingToolkit.jl for defining such systems symbolically, allowing for optimizations like automated generation of parallel code, symbolic simplification, and generation of sparsity patterns.

Inverse Problems, Parameter Estimation, and Structural Identification

Parameter estimation and inverse problems are solved directly on their constituent problem types using tools like SciMLSensitivity.jl. Thus for example, there is no ODEInverseProblem, and instead ODEProblem is used to find the parameters p that solve the inverse problem. Check out the SciMLSensitivity documentation for a discussion on connections to automatic differentiation, optimization, and adjoints.

Common Interface High-Level Overview

The SciML interface is common as the usage of arguments is standardized across all of the problem domains. Underlying high-level ideas include:

  • All domains use the same interface of defining a AbstractSciMLProblem which is then solved via solve(prob,alg;kwargs), where alg is a AbstractSciMLAlgorithm. The keyword argument namings are standardized across the organization.
  • AbstractSciMLProblems are generally defined by a AbstractSciMLFunction which can define extra details about a model function, such as its analytical Jacobian, its sparsity patterns and so on.
  • There is an organization-wide method for defining linear and nonlinear solvers used within other solvers, giving maximum control of performance to the user.
  • Types used within the packages are defined by the input types. For example, packages attempt to internally use the type of the initial condition as the type for the state within differential equation solvers.
  • solve calls should be thread-safe and parallel-safe.
  • init(prob,alg;kwargs) returns an iterator which allows for directly iterating over the solution process
  • High performance is key. Any performance that is not at the top level is considered a bug and should be reported as such.
  • All functions have an in-place and out-of-place form, where the in-place form is made to utilize mutation for high performance on large-scale problems and the out-of-place form is for compatibility with tooling like static arrays and some reverse-mode automatic differentiation systems.

Flowchart Example for PDE-Constrained Optimal Control

The following example showcases how the pieces of the common interface connect to solve a problem that mixes inference, symbolics, and numerics.

External Binding Libraries

  • diffeqr

    • Solving differential equations in R using DifferentialEquations.jl with ModelingToolkit for JIT compilation and GPU-acceleration

  • diffeqpy

    • Solving differential equations in Python using DifferentialEquations.jl

Note About Third-Party Libraries

The SciML documentation references and recommends many third-party libraries for improving ones modeling, simulation, and analysis workflow in Julia. Take these as a positive affirmation of the quality of these libraries, as these libraries are commonly tested by SciML developers who are in contact with the development teams of these groups. It also documents the libraries which are commonly chosen by SciML as dependencies. Do not take omissions as negative affirmations against a given library, i.e. a library left off of the list by SciML is not a negative endorsement. Rather, it means that compatibility with SciML is untested, SciML developers may have a personal preference for another choice, or SciML developers may be simply unaware of the library's existence. If one would like to add a third-party library to the SciML documentation, open a pull request with the requested text.

Note that the libraries in this documentation are only those that are meant to be used in the SciML extended universe of modeling, simulation, and analysis and thus there are many high-quality libraries in other domains (machine learning, data science, etc.) which are purposefully not included. For an overview of the Julia package ecosystem, see the JuliaHub Search Engine.

diff --git a/dev/search/index.html b/dev/search/index.html index bd2b7f8cc0d..29e819d76ef 100644 --- a/dev/search/index.html +++ b/dev/search/index.html @@ -3,4 +3,4 @@ function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash}); -

Loading search...

    +

    Loading search...

      diff --git a/dev/showcase/bayesian_neural_ode/index.html b/dev/showcase/bayesian_neural_ode/index.html index bd72f239301..f54806ffef2 100644 --- a/dev/showcase/bayesian_neural_ode/index.html +++ b/dev/showcase/bayesian_neural_ode/index.html @@ -24,24 +24,24 @@ prob_neuralode = NeuralODE(dudt2, tspan, Tsit5(), saveat = tsteps) rng = Random.default_rng() p = Float64.(prob_neuralode.p)
      252-element Vector{Float64}:
-  0.0451086163520813
-  0.20711150765419006
- -0.14898575842380524
- -0.03306718170642853
- -0.2572575509548187
-  0.21571092307567596
- -0.18696726858615875
-  0.08312129229307175
-  0.09962267428636551
- -0.08519560843706131
+  0.16585227847099304
+ -0.23816539347171783
+ -0.01296249870210886
+  0.08566386997699738
+  0.22036786377429962
+  0.31189286708831787
+  0.05173519626259804
+  0.19525820016860962
+ -0.33867040276527405
+ -0.22815316915512085
   ⋮
-  0.2837769687175751
-  0.26344507932662964
- -0.03320996090769768
-  0.192137211561203
- -0.29417839646339417
- -0.018976453691720963
- -0.09159481525421143
+  0.14065802097320557
+  0.18694020807743073
+  0.04320073127746582
+ -0.33843815326690674
+ -0.23239505290985107
+ -0.21973831951618195
+ -0.17280490696430206
   0.0
   0.0

      Note that the f64 is required to put the Flux neural network into Float64 precision.

      Step 3: Define the loss function for the Neural ODE.

      function predict_neuralode(p)
     Array(prob_neuralode(u0, p))
@@ -61,492 +61,510 @@
 h = Hamiltonian(metric, l, dldθ)
      Hamiltonian(metric=DiagEuclideanMetric([1.0, 1.0, 1.0, 1.0, 1.0, 1 ...]), kinetic=AdvancedHMC.GaussianKinetic())

      We use the NUTS sampler with an acceptance ratio of δ= 0.45 in this example. In addition, we use Nesterov Dual Averaging for the Step Size adaptation.

      We sample using 500 warmup samples and 500 posterior samples.

      integrator = Leapfrog(find_good_stepsize(h, p))
 kernel = HMCKernel(Trajectory{MultinomialTS}(integrator, GeneralisedNoUTurn()))
 adaptor = StanHMCAdaptor(MassMatrixAdaptor(metric), StepSizeAdaptor(0.45, integrator))
-samples, stats = sample(h, kernel, p, 500, adaptor, 500; progress = true)
      ([[0.18676403312204592, 0.21609711063995982, -0.17012745028275753, -0.11812898002147008, -0.17847739028933224, 0.20111061806528585, -0.19180452290908215, 0.15276355622857585, 0.11045224233922014, -0.023317181515159643  …  0.12985469095505286, 0.2877110061184443, 0.2156011615910796, -0.028010335113935247, 0.1914616916245018, -0.32394848898082396, 0.043864600756598204, -0.11260415863071241, -0.042398406597111395, 0.033928502668466566], [0.18676403312204592, 0.21609711063995982, -0.17012745028275753, -0.11812898002147008, -0.17847739028933224, 0.20111061806528585, -0.19180452290908215, 0.15276355622857585, 0.11045224233922014, -0.023317181515159643  …  0.12985469095505286, 0.2877110061184443, 0.2156011615910796, -0.028010335113935247, 0.1914616916245018, -0.32394848898082396, 0.043864600756598204, -0.11260415863071241, -0.042398406597111395, 0.033928502668466566], [0.18676403312204592, 0.21609711063995982, -0.17012745028275753, -0.11812898002147008, -0.17847739028933224, 0.20111061806528585, -0.19180452290908215, 0.15276355622857585, 0.11045224233922014, -0.023317181515159643  …  0.12985469095505286, 0.2877110061184443, 0.2156011615910796, -0.028010335113935247, 0.1914616916245018, -0.32394848898082396, 0.043864600756598204, -0.11260415863071241, -0.042398406597111395, 0.033928502668466566], [0.33587589753340585, -0.08420241695878447, -0.08732451646376457, -0.1610889836667806, -0.44424030420730315, 0.18009134491115808, -0.083421727333038, 0.29729087840191204, 0.1871020548113957, 0.05977897480449604  …  0.3771275659357269, 0.1606167602670201, 0.30322135795690147, -0.07745102494978894, 0.3236683975878397, -0.15681169605268408, 0.220973487506181, -0.11009046908924983, -0.3713827853789473, -0.2374559287709279], [0.6768508357989308, -0.49172361051657243, 1.1504362684744074, -0.5888038795299849, -0.7304923003012764, 1.3694797692552148, 1.2674204107633154, 0.48850721014836995, -1.5189358924321021, 0.8828656468254842  …  1.2050597805390906, -0.8603663749126834, 0.7555047258473578, -0.5436937826588962, -0.014516809927783947, -0.30932280037309406, -0.1810922665254449, -0.28258472471955554, -0.199299962458853, -0.5361848657088452], [-0.29393097899463877, -0.7919186772984815, 0.19642854521352354, 0.5142904480454779, -0.22822780009119334, 0.4887994264962238, -0.07776055084685096, 0.5565203794401352, -0.007241606168178108, -0.017549831377202722  …  -0.44271137914135517, -0.9951435616015811, -0.6962748784814936, -0.5297910293600305, 0.06946320592062166, -1.2481727085573737, 0.015519284936937699, -0.7094129665202394, -0.884984811019001, -0.22024295555006457], [-0.29393097899463877, -0.7919186772984815, 0.19642854521352354, 0.5142904480454779, -0.22822780009119334, 0.4887994264962238, -0.07776055084685096, 0.5565203794401352, -0.007241606168178108, -0.017549831377202722  …  -0.44271137914135517, -0.9951435616015811, -0.6962748784814936, -0.5297910293600305, 0.06946320592062166, -1.2481727085573737, 0.015519284936937699, -0.7094129665202394, -0.884984811019001, -0.22024295555006457], [1.3037189833853589, -1.9371444076308602, 0.6346087419829276, 0.19137543214965014, -0.7510243559644917, 1.0778166489549816, -0.8365595316524854, 0.26447040954493745, 0.05913296883031817, -0.5884021275579585  …  -0.3667133668801474, -0.20368867199859486, -0.6802209278350754, 0.19642264657990546, 0.14710735640656378, -1.5157644823062284, 0.3730596337249744, -0.5158809399578445, -0.7191309414768713, 0.5457014279615509], [1.3037189833853589, -1.9371444076308602, 0.6346087419829276, 0.19137543214965014, -0.7510243559644917, 1.0778166489549816, -0.8365595316524854, 0.26447040954493745, 0.05913296883031817, -0.5884021275579585  …  -0.3667133668801474, -0.20368867199859486, -0.6802209278350754, 0.19642264657990546, 0.14710735640656378, -1.5157644823062284, 0.3730596337249744, -0.5158809399578445, -0.7191309414768713, 0.5457014279615509], [-0.6562981503719209, 0.4424586509083523, 0.2967963188797707, -0.9910260508697947, 0.5633238569084831, -0.7820301773491801, 1.637651117267107, 0.1370543173851005, -1.357354591627086, -0.7525833714483455  …  -0.42339663468578154, -0.8891398197723833, -0.11634739098966952, -0.8787947790544147, -0.7804182562113252, 0.015188766060720976, -1.0068251913597623, -0.609654224913516, 0.23330473268917457, 0.5480139407041471]  …  [-0.7845343258310572, 0.5385417539622476, -0.08977010422765301, 0.6525573737044335, -0.06348788913330124, 0.25422068826238137, 1.0602970966375398, -0.5067285042740767, 0.22180621286862554, 0.16171794119010338  …  -0.25871330323174546, -0.4491844561593326, 0.7720525277670123, 1.0979626747020872, -0.25431177301362784, -0.04278813572375713, 0.1552076829800462, -0.7146428788112393, -0.6703558578440056, 0.08663383761381498], [-0.6864711339738019, 0.7170445950472322, -0.12876142660422668, 0.8166971133195504, -0.09254288608544023, 0.28723822640735247, 1.0524854613198076, -0.7517652248976678, 0.12056677187275153, 0.16223600221725554  …  -0.1698356629785493, -0.5448206546728898, 0.937147046361127, 1.0483130473259545, -0.4268292093092123, -0.20940891206603543, 0.16655604420654582, -0.7600151626550482, -0.49178642914889104, 0.08948148980459952], [-0.29614960941052365, 0.7275861418534532, -0.12024086159684397, 0.90895156524207, -0.8283219648448144, 0.36293582379461353, 0.9388003775040511, -0.424383209349355, 0.20061540911480621, -0.4146837061657735  …  -0.4173614042664097, -0.3129564648982009, 0.3039336787823661, 0.8566002385093907, -0.2873030470861492, -0.723144158597639, 0.6718732680093789, -0.15401628101209536, -0.7531726819352929, 0.03425500641220298], [-0.22668789609393242, 0.4057705689222362, 0.5023439210891694, 1.5203238242393906, -0.006401735051681616, 0.0849134803481488, 0.7158143787053971, -0.5262928123750186, 0.20037814149108474, -0.183723346856272  …  0.4954396501889638, 0.1598189041250047, 0.2361826113093644, 1.4143944895424758, 0.07549920238256284, -0.8192053461709132, 0.9012743987645029, 0.1883169007775938, -0.3493953878598314, -0.2632427100309172], [-1.5220430528039706, 0.20788301108701993, -0.22756670890912148, -0.23469606072198063, 1.0346512788897204, 0.6259741161153541, 0.4804709697434912, -0.6407718344779109, 0.5386689612282283, 0.38399820575705146  …  -0.7067252732311327, 0.6623904171216489, 0.032315010975192654, -0.06092582813756671, -0.8717666877765747, 0.3823558714965239, 0.5330702197981853, 0.3911535807897424, -0.5729668469234246, 1.0715982292760375], [-1.7386799072477563, 0.4284590108281192, -0.886772857124223, 0.9023094260594566, 0.7261121726804206, 0.0982779411180534, 0.6928746749842987, 0.6479410673452699, 0.9004531772288454, -0.7462736913852449  …  -0.8759805358721823, -0.33688477125526045, -0.6549510249400744, -0.11194369975696364, -0.6684555522993304, 0.18798097608134626, 0.9114920801592646, 0.8216929481729349, -0.30878299876055704, 0.7917571314990903], [1.0350384108636388, -1.138089665359205, 0.3130511749604718, -1.3300638363399435, 0.034319386791141986, 0.09615310235611607, 0.08254715165994456, -0.7578913707587728, -1.0984072912872729, -0.07819677015877612  …  1.018449418812572, 0.39127122478496623, -0.5501817248424135, 1.7225663437757515, 0.3817045479386266, 0.20906802754529008, -1.3553444287784733, -1.6668043854606494, -0.3905284035981563, -1.2878970138643178], [0.9582429570664223, -1.050049587247462, 0.30400649705198446, -1.2777556912670185, 0.19735679741457798, 0.15093792518152666, 0.004008015148230223, -0.6656027435883253, -1.0120523490697075, -0.04000869287618457  …  1.0539964725286863, 0.4091358028596301, -0.48974113804595754, 1.7151018812436276, 0.2982188157694456, 0.20559831735877773, -1.4223575582442027, -1.6721890089926117, -0.42037989462695574, -1.2900341550497088], [1.2059179242227356, -0.021292780302137687, 0.7635048460457541, -1.9215200286735115, 0.43907316783597355, 0.3482521839766149, -0.27447636075199233, -0.7854375985924724, -0.8361852887368687, 0.27429731755362907  …  0.5672938726639952, 0.2782350845641874, -0.5783335213493608, 1.6498728037007213, -0.38495105110763933, -0.21286509453945182, -1.6184916718257645, -1.45024539811073, -0.710671239239038, -1.3625144450278985], [1.254454107886771, -0.37668536131835145, 0.6390329441936057, -1.292427944778886, 0.8893550066365192, 0.3509255865647791, -0.5460885653493375, -0.6608520160532483, -0.7759760804917021, 0.19603100129060944  …  0.3344905378496288, 0.7075094097165124, -0.2640079648329097, 2.1361806538502415, -0.332412132057542, 0.5481892743319243, -1.7283251755566948, -1.6092052054000594, -0.6140651915195048, -1.1331808611830212]], NamedTuple[(n_steps = 7, is_accept = true, acceptance_rate = 1.0, log_density = -265.800608937807, hamiltonian_energy = 584.7119476217438, hamiltonian_energy_error = -53.32899128239535, max_hamiltonian_energy_error = -53.61562242017419, tree_depth = 3, numerical_error = false, step_size = 0.025, nom_step_size = 0.025, is_adapt = true), (n_steps = 1, is_accept = true, acceptance_rate = 0.0, log_density = -265.800608937807, hamiltonian_energy = 414.9242117196844, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 382947.88947529614, tree_depth = 0, numerical_error = true, step_size = 0.6795704571147614, nom_step_size = 0.6795704571147614, is_adapt = true), (n_steps = 3, is_accept = true, acceptance_rate = 0.0, log_density = -265.800608937807, hamiltonian_energy = 400.12569134445897, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 74602.46515354213, tree_depth = 1, numerical_error = true, step_size = 0.3164493440113639, nom_step_size = 0.3164493440113639, is_adapt = true), (n_steps = 13, is_accept = true, acceptance_rate = 0.1538461538544849, log_density = -154.09941960426906, hamiltonian_energy = 400.05087969030245, hamiltonian_energy_error = -5.461821850738772, max_hamiltonian_energy_error = 5650.369268790535, tree_depth = 3, numerical_error = true, step_size = 0.09837809577101383, nom_step_size = 0.09837809577101383, is_adapt = true), (n_steps = 63, is_accept = true, acceptance_rate = 0.2966011158703339, log_density = -142.19865120995328, hamiltonian_energy = 299.6453226034396, hamiltonian_energy_error = -3.19817942068795, max_hamiltonian_energy_error = 319.4294315130142, tree_depth = 5, numerical_error = false, step_size = 0.03946077228465222, nom_step_size = 0.03946077228465222, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.9386536650085927, log_density = -120.96113078782729, hamiltonian_energy = 252.84348597655702, hamiltonian_energy_error = 0.13180082850959707, max_hamiltonian_energy_error = -0.6843833990407688, tree_depth = 7, numerical_error = false, step_size = 0.023049757356196264, nom_step_size = 0.023049757356196264, is_adapt = true), (n_steps = 2, is_accept = true, acceptance_rate = 2.7028545467171583e-22, log_density = -120.96113078782729, hamiltonian_energy = 254.13681216773938, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 12490.838607809683, tree_depth = 1, numerical_error = true, step_size = 0.09649882466157923, nom_step_size = 0.09649882466157923, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.9528889163061325, log_density = -127.07252147397072, hamiltonian_energy = 233.30139829319688, hamiltonian_energy_error = -1.941140381151797, max_hamiltonian_energy_error = -2.0621787981961006, tree_depth = 7, numerical_error = false, step_size = 0.023407336682579737, nom_step_size = 0.023407336682579737, is_adapt = true), (n_steps = 2, is_accept = true, acceptance_rate = 0.0, log_density = -127.07252147397072, hamiltonian_energy = 256.21599914396216, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 3430.529293196024, tree_depth = 1, numerical_error = true, step_size = 0.11112028319523479, nom_step_size = 0.11112028319523479, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.3945413705975654, log_density = -144.4715222232628, hamiltonian_energy = 256.1603913094665, hamiltonian_energy_error = 1.4380852394406816, max_hamiltonian_energy_error = 317.5037366930614, tree_depth = 7, numerical_error = false, step_size = 0.026726067362175678, nom_step_size = 0.026726067362175678, is_adapt = true)  …  (n_steps = 127, is_accept = true, acceptance_rate = 0.7314974266976035, log_density = -119.36995613550589, hamiltonian_energy = 229.94517874203493, hamiltonian_energy_error = 0.7176032785845621, max_hamiltonian_energy_error = 1.3205969926871148, tree_depth = 7, numerical_error = false, step_size = 0.031083403169598903, nom_step_size = 0.031083403169598903, is_adapt = true), (n_steps = 51, is_accept = true, acceptance_rate = 0.09760842176148626, log_density = -119.98753974574807, hamiltonian_energy = 244.06120192155925, hamiltonian_energy_error = -0.49693991744931054, max_hamiltonian_energy_error = 9298.453187226896, tree_depth = 5, numerical_error = true, step_size = 0.0643349711004939, nom_step_size = 0.0643349711004939, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.5871293510158464, log_density = -134.2799894538999, hamiltonian_energy = 243.38806448947736, hamiltonian_energy_error = -0.22792867593540223, max_hamiltonian_energy_error = 13.947107814592584, tree_depth = 7, numerical_error = false, step_size = 0.027132590273561224, nom_step_size = 0.027132590273561224, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.4458499680509089, log_density = -131.3788924171561, hamiltonian_energy = 251.620507142184, hamiltonian_energy_error = -0.3662399488741812, max_hamiltonian_energy_error = 100.2433198176733, tree_depth = 7, numerical_error = false, step_size = 0.038906199498723475, nom_step_size = 0.038906199498723475, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.3750754529073392, log_density = -137.08893288515333, hamiltonian_energy = 261.56738699052704, hamiltonian_energy_error = 0.4254458070412852, max_hamiltonian_energy_error = 74.59896153165624, tree_depth = 7, numerical_error = false, step_size = 0.039221185248918294, nom_step_size = 0.039221185248918294, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.3635025195206693, log_density = -123.910320430363, hamiltonian_energy = 258.7550874860741, hamiltonian_energy_error = 0.29451048914199873, max_hamiltonian_energy_error = 117.69356899322025, tree_depth = 7, numerical_error = false, step_size = 0.03326274403547096, nom_step_size = 0.03326274403547096, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.940450242340551, log_density = -138.36801550743343, hamiltonian_energy = 267.5850426471035, hamiltonian_energy_error = -0.2963013745563785, max_hamiltonian_energy_error = 0.4870623317717673, tree_depth = 7, numerical_error = false, step_size = 0.0274907550637594, nom_step_size = 0.0274907550637594, is_adapt = true), (n_steps = 46, is_accept = true, acceptance_rate = 0.030233451531046084, log_density = -140.8372522400232, hamiltonian_energy = 272.937354992254, hamiltonian_energy_error = 0.062356858182226915, max_hamiltonian_energy_error = 2696.7948624377887, tree_depth = 5, numerical_error = true, step_size = 0.09125458656878631, nom_step_size = 0.09125458656878631, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.36825050495440953, log_density = -155.18892243622426, hamiltonian_energy = 270.3578664083662, hamiltonian_energy_error = -0.22155100185261745, max_hamiltonian_energy_error = 18.64294511729844, tree_depth = 7, numerical_error = false, step_size = 0.03386569232854646, nom_step_size = 0.03386569232854646, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.8785317734366309, log_density = -145.10526779894874, hamiltonian_energy = 273.6560075221083, hamiltonian_energy_error = -0.22747208648894457, max_hamiltonian_energy_error = 0.7468416209230213, tree_depth = 7, numerical_error = false, step_size = 0.028405271830852622, nom_step_size = 0.028405271830852622, is_adapt = true)])

      Step 5: Plot diagnostics

      Now let's make sure the fit is good. This can be done by looking at the chain mixing plot and the autocorrelation plot. First, let's create the chain mixing plot using the plot recipes from ????

      samples = hcat(samples...)
+samples, stats = sample(h, kernel, p, 500, adaptor, 500; progress = true)
      ([[0.15908119871237822, -0.21538027044255442, -0.0004366441120075837, 0.04587494925162808, 0.21602284163727042, 0.34508078590634794, -0.003284138352754054, 0.22203263597582276, -0.315425379820629, -0.2429354852260034  …  -0.0461481207877002, 0.17611005229456217, 0.1808996298521113, 0.017458959431862348, -0.3047186475909503, -0.2583434457883221, -0.30565997147243973, -0.22471312336492916, -0.03863256138705982, 0.02168724672716667], [0.15908119871237822, -0.21538027044255442, -0.0004366441120075837, 0.04587494925162808, 0.21602284163727042, 0.34508078590634794, -0.003284138352754054, 0.22203263597582276, -0.315425379820629, -0.2429354852260034  …  -0.0461481207877002, 0.17611005229456217, 0.1808996298521113, 0.017458959431862348, -0.3047186475909503, -0.2583434457883221, -0.30565997147243973, -0.22471312336492916, -0.03863256138705982, 0.02168724672716667], [0.15908119871237822, -0.21538027044255442, -0.0004366441120075837, 0.04587494925162808, 0.21602284163727042, 0.34508078590634794, -0.003284138352754054, 0.22203263597582276, -0.315425379820629, -0.2429354852260034  …  -0.0461481207877002, 0.17611005229456217, 0.1808996298521113, 0.017458959431862348, -0.3047186475909503, -0.2583434457883221, -0.30565997147243973, -0.22471312336492916, -0.03863256138705982, 0.02168724672716667], [0.19921279645144777, -0.1670461299199364, -0.015590572122768166, -0.04527198330413584, 0.20680755613899868, 0.41052749779719067, -0.33444379934827095, 0.3346326939011659, -0.27636015969992866, -0.2549972771338459  …  0.06785539087490572, 0.08336222863854925, 0.0855962465842955, 0.035193497710074316, -0.4475254935642552, -0.3035581665353245, -0.30234977348328085, -0.12820112643849801, -0.24599035354742216, -0.16629389214470358], [-0.11860003739281756, 0.33793434935292394, -0.1946866640684005, -0.6067623359928771, 0.9241475232713863, -0.21245381239264644, -0.5489861743506737, 0.45017086400311634, -0.2650668973216496, -1.15820839709366  …  -0.0873146905515509, 0.2839468249596357, 0.39423133976644775, 0.1337426080683739, -0.767591211130176, -0.12963565949934525, -0.5508420484176926, 0.7541731216901948, -1.2362597873506556, -0.4943051500087496], [-0.11860003739281756, 0.33793434935292394, -0.1946866640684005, -0.6067623359928771, 0.9241475232713863, -0.21245381239264644, -0.5489861743506737, 0.45017086400311634, -0.2650668973216496, -1.15820839709366  …  -0.0873146905515509, 0.2839468249596357, 0.39423133976644775, 0.1337426080683739, -0.767591211130176, -0.12963565949934525, -0.5508420484176926, 0.7541731216901948, -1.2362597873506556, -0.4943051500087496], [-0.14561038201162255, 0.33790794710980493, -0.2923783397492392, -0.5655393730307768, 0.9472639022915389, -0.2928266492382462, -0.5075423117638398, 0.44847360122071356, -0.3066486937873758, -1.239681494897553  …  -0.09130560236138686, 0.3132354811179049, 0.347574039163111, 0.21692872962685578, -0.8481307205520096, -0.027953932637337392, -0.6114180951045894, 0.680781791721626, -1.2782688766476829, -0.4264728816000611], [0.12091395854743421, 0.49489994410717286, 0.4334865031570059, 1.2424439605953712, -0.7315061472420058, -0.1987161091300785, 0.2362597390018883, -0.21322547833977365, 0.33159409079883617, 1.3374788688080748  …  0.44090869932119103, -0.10668320413994381, -0.129663773961017, 0.16210036280934972, 0.4094913533419201, 0.5605737806911416, 0.9834177180943646, -0.6126846194717928, 0.6798398102659883, 0.3225497515883276], [0.08446042544034962, 0.4789407476233035, 0.35560906288335503, 1.362031970290423, -0.7473878667265889, -0.323675184624083, 0.27441040761955837, -0.24025034720301275, 0.27377522410248484, 1.4001442721706752  …  0.441506725629084, -0.10718296043236869, -0.06504431167815321, 0.27057289468322354, 0.342860593946726, 0.49317465469319405, 1.0267764000278221, -0.7438528814085722, 0.6213569160460908, 0.25235007203455134], [0.6353178299785874, -0.6022432959623563, -0.8848056629105944, -1.4438086332633606, -0.22831466660668745, -0.28478727479073646, 0.34359704568073535, -0.2020793071563279, 0.48960395288053316, -0.2026966500191596  …  -0.7740510937532274, -1.3821537777488728, -0.07375470644316626, -0.5288890345365687, -0.31628787990941737, -0.49891663896704147, 0.1889572487158116, 0.6791825584013498, -1.3383783395411524, 0.6011630612116853]  …  [0.1723420009021007, -1.2056456494000953, -0.5850382399732998, -0.08247327383504152, -0.5166631160921399, -0.15805212905495428, -0.7803546311627647, -1.006581106623415, -0.5224813525824629, 0.8109432766278183  …  0.9957501724782515, -0.39365066080201117, -2.4341721889460137, 1.3285806146371204, 1.2269672450396791, 1.4515959122230164, 0.3548785694849195, 0.04734322897435771, -0.22953442991381598, -0.47082474673114966], [0.4201487210781185, -1.098508882199228, -0.505318968786294, -0.5646576063203708, -0.7260392340830485, 0.4396431762690715, -0.541585838525141, -0.947618149001512, -0.7285216979316578, 1.180890459598155  …  0.7502594335152439, -0.5316138343388987, -1.9260483693175685, 0.860111606119412, 0.9722558462201711, 0.809529821669976, 0.8274267436661599, 0.2957621994151195, -0.1275550592512557, -0.09040815668144091], [0.029742759024123835, 0.5771418612367976, -0.3371422972895501, -0.2922228634395665, 0.9422534178995946, -0.16046613428765522, -0.3625092344221658, -0.3500409703655089, -0.3772324575720081, -1.1854421858850561  …  0.2695870919264112, -0.21655178075077264, -1.0855956344261324, 0.49455195480529923, 0.19287083317712647, -0.716860528684752, -0.9478616913848767, 0.868005930243717, -0.3196947419525855, 1.1889006104564062], [0.029742759024123835, 0.5771418612367976, -0.3371422972895501, -0.2922228634395665, 0.9422534178995946, -0.16046613428765522, -0.3625092344221658, -0.3500409703655089, -0.3772324575720081, -1.1854421858850561  …  0.2695870919264112, -0.21655178075077264, -1.0855956344261324, 0.49455195480529923, 0.19287083317712647, -0.716860528684752, -0.9478616913848767, 0.868005930243717, -0.3196947419525855, 1.1889006104564062], [0.0006244566503138191, 0.3732048229182208, -1.097707680587978, -0.06066810702499206, 1.1264560173823355, -0.7921278418253955, -0.784621235367813, -0.2457749014240743, 0.40108156908481374, -0.9861931293915824  …  -0.31060653592370613, -0.27858300278592624, -1.3094104084828029, 0.5843248418347881, 0.09910989880871383, -0.5240762206122644, -0.766732804671995, 1.2133977845825812, -0.8078570457687485, 0.8120894864372535], [0.4160866335401733, 0.8658880165451155, -0.09457111393574326, -0.8313162695564281, 0.5892438187266243, 0.9082695591495631, -0.48169999504543837, -1.9002482337385183, -0.20203372514662918, 0.41643492506644286  …  -0.5664180529797629, -0.44111732935796333, -0.5533300039502442, 0.2619899795344053, 0.44940394441467396, 0.5586692620809669, -0.396849334791409, -0.9839555607107752, -0.06270694604853819, -0.18482513450452753], [0.7989262184170991, -0.2490466355913471, -0.39878265566503085, -0.4094593136538975, -0.38708805257148393, 0.7320719816953895, -0.3526880223590782, -0.7797657509357403, 0.9869625948957326, -0.4867844685689372  …  0.20552086535880015, -1.1630174476276258, -0.34140706451050234, -0.02765213584516364, 0.44467773211509726, 0.36269689062562, 0.14267880620452233, -0.6044241046141733, -0.732242913093104, -0.676620684979649], [-0.22577084090930638, -1.3188289387915277, -0.6209296760583372, -0.6547463870874968, 0.647915791508397, 0.5071705166232929, -0.4131341927411677, 0.21661735325726272, 0.053052553659201654, -0.18637123814567927  …  0.7427740523160904, -0.34315342990394254, -0.774689707986285, 0.03257985291976137, 0.4011545189044017, 0.25223087641286085, 0.08201393012366301, -0.6422685991673048, -1.5242424215796824, 0.008592271345421412], [-0.1037829739270197, -1.2094107441971922, -0.7578810230246329, -0.5740230955391404, 0.47243180997362283, 0.5612249533984395, -0.39504098184531616, 0.23201245850059912, 0.17635525975996227, -0.19418811302994093  …  0.9957615145541325, -0.2592230499818961, -0.7931450208970142, 0.01266074030584258, 0.3333924549909077, 0.19185468288301444, -0.14707663588136113, -0.6443399641996634, -1.459488778407006, 0.05822581143638483], [-0.09362785331074644, 0.7033361676804596, 0.21036198293709849, 1.2917189316544782, -0.6126418927794969, -0.4487949364615542, 0.06340112885153963, 0.031950681445666054, 0.02624804867219669, 0.2957237756489801  …  0.2995293848060039, -0.8548833484644914, -0.21367577001187976, -0.5310016900721, -0.40110597831236255, 0.6442353872267603, 0.5282114334961626, 1.2104266969558153, 0.3822417380878732, 0.04332793162272654]], NamedTuple[(n_steps = 57, is_accept = true, acceptance_rate = 0.9649122807017544, log_density = -247.7999788842341, hamiltonian_energy = 523.104627650538, hamiltonian_energy_error = -69.61780376453487, max_hamiltonian_energy_error = 3276.310227099483, tree_depth = 5, numerical_error = true, step_size = 0.025, nom_step_size = 0.025, is_adapt = true), (n_steps = 1, is_accept = true, acceptance_rate = 0.0, log_density = -247.7999788842341, hamiltonian_energy = 365.63875654682727, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 4174.50623768164, tree_depth = 0, numerical_error = true, step_size = 0.6375706309820263, nom_step_size = 0.6375706309820263, is_adapt = true), (n_steps = 3, is_accept = true, acceptance_rate = 1.8670318440261677e-242, log_density = -247.7999788842341, hamiltonian_energy = 371.29767440496397, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 9101.446194808344, tree_depth = 1, numerical_error = true, step_size = 0.29133116109109264, nom_step_size = 0.29133116109109264, is_adapt = true), (n_steps = 11, is_accept = true, acceptance_rate = 0.27272920320841837, log_density = -168.2379970913328, hamiltonian_energy = 359.2741783208446, hamiltonian_energy_error = -16.197932331307697, max_hamiltonian_energy_error = 1836.5817229355266, tree_depth = 3, numerical_error = true, step_size = 0.08959684936478572, nom_step_size = 0.08959684936478572, is_adapt = true), (n_steps = 22, is_accept = true, acceptance_rate = 0.8872180245454132, log_density = -89.02298347508406, hamiltonian_energy = 282.07296272306377, hamiltonian_energy_error = -4.5061364096918055, max_hamiltonian_energy_error = 1388.2989224564121, tree_depth = 4, numerical_error = true, step_size = 0.05013514707057901, nom_step_size = 0.05013514707057901, is_adapt = true), (n_steps = 2, is_accept = true, acceptance_rate = 0.021816762227286263, log_density = -89.02298347508406, hamiltonian_energy = 197.51136645303177, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 2407.5113879784576, tree_depth = 1, numerical_error = true, step_size = 0.17214963279921688, nom_step_size = 0.17214963279921688, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.06467990880994419, log_density = -89.37929393774105, hamiltonian_energy = 208.61528533734344, hamiltonian_energy_error = 0.1459342723211421, max_hamiltonian_energy_error = 943.4729640096461, tree_depth = 6, numerical_error = false, step_size = 0.04593649589247621, nom_step_size = 0.04593649589247621, is_adapt = true), (n_steps = 255, is_accept = true, acceptance_rate = 0.9862541378573565, log_density = -73.59915950983259, hamiltonian_energy = 215.50511644428434, hamiltonian_energy_error = -0.18210532506623167, max_hamiltonian_energy_error = -0.38129165066220594, tree_depth = 8, numerical_error = false, step_size = 0.013460945176478495, nom_step_size = 0.013460945176478495, is_adapt = true), (n_steps = 27, is_accept = true, acceptance_rate = 0.059120534922688706, log_density = -72.62729541615757, hamiltonian_energy = 201.01844791203592, hamiltonian_energy_error = -0.1125664941918103, max_hamiltonian_energy_error = 1058.7328318767472, tree_depth = 4, numerical_error = true, step_size = 0.07058884630878967, nom_step_size = 0.07058884630878967, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.8367213672819365, log_density = -112.31876634191029, hamiltonian_energy = 185.27925161257025, hamiltonian_energy_error = 0.29647691371181395, max_hamiltonian_energy_error = 0.5866415138774244, tree_depth = 7, numerical_error = false, step_size = 0.020417664874126107, nom_step_size = 0.020417664874126107, is_adapt = true)  …  (n_steps = 127, is_accept = true, acceptance_rate = 0.5127213322430865, log_density = -141.5454139082045, hamiltonian_energy = 271.6984955129464, hamiltonian_energy_error = -0.7377916322265605, max_hamiltonian_energy_error = 47.17944857795527, tree_depth = 7, numerical_error = false, step_size = 0.03783589306360171, nom_step_size = 0.03783589306360171, is_adapt = true), (n_steps = 63, is_accept = true, acceptance_rate = 0.2832111502744439, log_density = -139.5061149388818, hamiltonian_energy = 290.9021644908987, hamiltonian_energy_error = 0.015823048500919867, max_hamiltonian_energy_error = 1438.3479531674407, tree_depth = 5, numerical_error = true, step_size = 0.045228205940248484, nom_step_size = 0.045228205940248484, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.9157784113209231, log_density = -121.97668831587238, hamiltonian_energy = 247.37790505101862, hamiltonian_energy_error = -0.04391636132754684, max_hamiltonian_energy_error = 1.9217264455908776, tree_depth = 7, numerical_error = false, step_size = 0.030429051647544333, nom_step_size = 0.030429051647544333, is_adapt = true), (n_steps = 12, is_accept = true, acceptance_rate = 0.03672361348561503, log_density = -121.97668831587238, hamiltonian_energy = 223.30145404860914, hamiltonian_energy_error = 0.0, max_hamiltonian_energy_error = 1379.7680202866086, tree_depth = 3, numerical_error = true, step_size = 0.09849754375580728, nom_step_size = 0.09849754375580728, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.3262502469185735, log_density = -141.09440729985428, hamiltonian_energy = 249.36347594114142, hamiltonian_energy_error = 0.22865327137412805, max_hamiltonian_energy_error = 42.57178743747335, tree_depth = 7, numerical_error = false, step_size = 0.03616499338345883, nom_step_size = 0.03616499338345883, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.7407712937037125, log_density = -139.82783992105607, hamiltonian_energy = 266.5582992152863, hamiltonian_energy_error = 0.30753071007222843, max_hamiltonian_energy_error = 0.7925039451118892, tree_depth = 7, numerical_error = false, step_size = 0.027290753951218185, nom_step_size = 0.027290753951218185, is_adapt = true), (n_steps = 63, is_accept = true, acceptance_rate = 0.2950223465686377, log_density = -138.7486422692576, hamiltonian_energy = 267.14664573343373, hamiltonian_energy_error = -1.4455910476726217, max_hamiltonian_energy_error = 278.477950651717, tree_depth = 6, numerical_error = false, step_size = 0.05642603059128759, nom_step_size = 0.05642603059128759, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.47615016035497987, log_density = -114.75397216136341, hamiltonian_energy = 240.0397633360368, hamiltonian_energy_error = -3.414368861027981, max_hamiltonian_energy_error = 168.57481010074468, tree_depth = 7, numerical_error = false, step_size = 0.039523883520132766, nom_step_size = 0.039523883520132766, is_adapt = true), (n_steps = 127, is_accept = true, acceptance_rate = 0.17341355337165268, log_density = -111.14114585197784, hamiltonian_energy = 262.1428737168977, hamiltonian_energy_error = -0.039570466024997586, max_hamiltonian_energy_error = 238.65582212474533, tree_depth = 7, numerical_error = false, step_size = 0.042879368438435726, nom_step_size = 0.042879368438435726, is_adapt = true), (n_steps = 255, is_accept = true, acceptance_rate = 0.9309603280609502, log_density = -113.91400359407555, hamiltonian_energy = 234.80202108696687, hamiltonian_energy_error = -0.035621480456427435, max_hamiltonian_energy_error = 0.8780711519849831, tree_depth = 8, numerical_error = false, step_size = 0.022653706400803726, nom_step_size = 0.022653706400803726, is_adapt = true)])

      Step 5: Plot diagnostics

      Now let's make sure the fit is good. This can be done by looking at the chain mixing plot and the autocorrelation plot. First, let's create the chain mixing plot using the plot recipes from ????

      samples = hcat(samples...)
 samples_reduced = samples[1:5, :]
 samples_reshape = reshape(samples_reduced, (500, 5, 1))
 Chain_Spiral = Chains(samples_reshape)
 plot(Chain_Spiral)
      - + - + - + - + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + +

      Now we check the autocorrelation plot:

      autocorplot(Chain_Spiral)
      - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + +

      As another diagnostic, let's check the result on retrodicted data. To do this, we generate solutions of the Neural ODE on samples of the neural network parameters, and check the results of the predictions against the data. Let's start by looking at the time series:

      pl = scatter(tsteps, ode_data[1, :], color = :red, label = "Data: Var1", xlabel = "t",
              title = "Spiral Neural ODE")
@@ -565,734 +583,734 @@
       ylims = (-2.5, 3.5))
      - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

      That showed the time series form. We can similarly do a phase-space plot:

      pl = scatter(ode_data[1, :], ode_data[2, :], color = :red, label = "Data", xlabel = "Var1",
              ylabel = "Var2", title = "Spiral Neural ODE")
 for k in 1:300
@@ -1303,390 +1321,390 @@
       label = "Best fit prediction", ylims = (-2.5, 3))
      - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/blackhole/index.html b/dev/showcase/blackhole/index.html index 11750049cd0..e16e3c9e20c 100644 --- a/dev/showcase/blackhole/index.html +++ b/dev/showcase/blackhole/index.html @@ -371,304 +371,304 @@ label="waveform data", xlabel="Time", ylabel="Waveform") - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

      Looks great!

      Automating the Discovery of Relativistic Equations from Newtonian Physics

      Now let's learn the relativistic corrections directly from the data. To define the UDE, we will define a Lux neural network and pass it into our Newtonian Physics + Nerual Network ODE definition from above:

      NN = Lux.Chain((x) -> cos.(x),
     Lux.Dense(1, 32, cos),
     Lux.Dense(32, 32, cos),
@@ -751,109 +751,109 @@
 plot!(plt, Newt_orbit[1,:], Newt_orbit[2,:], linewidth = 2, label = "Newtonian")
      - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 
      plt = plot(tsteps,true_waveform, linewidth = 2, label = "truth", xlabel="Time", ylabel="Waveform")
 plot!(plt,tsteps,pred_waveform, linestyle = :dash, linewidth = 2, label = "prediction")
 plot!(plt,tsteps,Newt_waveform, linewidth = 2, label = "Newtonian")
      - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

      Now we'll do the same, but extrapolating the model out in time.

      factor=5
 
 extended_tspan = (tspan[1], factor*tspan[2])
@@ -873,53 +873,53 @@
 plot!(plt, Newt_orbit[1,:], Newt_orbit[2,:], linewidth = 2, label = "Newtonian")
      - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 
      true_waveform = compute_waveform(dt_data, reference_solution, mass_ratio, model_params)[1]
 pred_waveform = compute_waveform(dt_data, optimized_solution, mass_ratio, model_params)[1]
 Newt_waveform = compute_waveform(dt_data, Newtonian_solution, mass_ratio, model_params)[1]
@@ -928,49 +928,49 @@
 plot!(plt,extended_tsteps,Newt_waveform, linewidth = 2, label = "Newtonian")
      - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dev/showcase/brusselator/index.html b/dev/showcase/brusselator/index.html index 7ddf6d6b528..e5090385fdf 100644 --- a/dev/showcase/brusselator/index.html +++ b/dev/showcase/brusselator/index.html @@ -382,4 +382,4 @@ ydomain:([0.0, 11.5], 0.0:0.03125:1.0, 0.0:0.03125:1.0) u: Dict{Symbolics.Num, Array{Float64, 3}} with 2 entries: u(x, y, t) => [0.0 0.115882 … 0.115882 0.0; 0.0 0.115882 … 0.115882 0.0; … ; … - v(x, y, t) => [0.0 0.0 … 0.0 0.0; 0.142219 0.142219 … 0.142219 0.142219; … ; …

      And now we're zooming! For more information on these performance improvements, check out the deeper dive in the DifferentialEquations.jl tutorials.

      If you're interested in figuring out what's the fastest current solver for this kind of PDE, check out the Brusselator benchmark in SciMLBenchmarks.jl

      + v(x, y, t) => [0.0 0.0 … 0.0 0.0; 0.142219 0.142219 … 0.142219 0.142219; … ; …

      And now we're zooming! For more information on these performance improvements, check out the deeper dive in the DifferentialEquations.jl tutorials.

      If you're interested in figuring out what's the fastest current solver for this kind of PDE, check out the Brusselator benchmark in SciMLBenchmarks.jl

      diff --git a/dev/showcase/gpu_spde/index.html b/dev/showcase/gpu_spde/index.html index 168226eaaf4..557bd86888a 100644 --- a/dev/showcase/gpu_spde/index.html +++ b/dev/showcase/gpu_spde/index.html @@ -213,56 +213,56 @@ plot(p1, p2, p3, layout = grid(3, 1)) - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - +

      and see the pretty gradients. Using this 2nd order ROCK method we solve this equation in about 2 seconds. That's okay.

      Some Optimizations

      There are some optimizations that can still be done. When we do A*B as matrix multiplication, we create another temporary matrix. These allocations can bog down the system. Instead, we can pre-allocate the outputs and use the inplace functions mul! to make better use of memory. The easiest way to store these cache arrays are constant globals, but you can use closures (anonymous functions which capture data, i.e. (x)->f(x,y)) or call-overloaded types to do it without globals. The globals way (the easy way) is simply:

      const MyA = zeros(N, N)
 const AMx = zeros(N, N)
@@ -1187,58 +1187,58 @@
 plot(p1, p2, p3, layout = grid(3, 1))
      - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - +

      Making Use of GPU Parallelism

      That was all using the CPU. How do we turn on GPU parallelism with DifferentialEquations.jl? Well, you don't. DifferentialEquations.jl "doesn't have GPU bits". So, wait... can we not do GPU parallelism? No, this is the glory of type-genericness, especially in broadcasted operations. To make things use the GPU, we simply use a CuArray from CUDA.jl. If instead of zeros(N,M) we used CuArray(zeros(N,M)), then the array lives on the GPU. CuArray naturally overrides broadcast such that dotted operations are performed on the GPU. DifferentialEquations.jl uses broadcast internally, and thus just by putting the array as a CuArray, the array-type will take over how all internal updates are performed and turn this algorithm into a fully GPU-parallelized algorithm that doesn't require copying to the CPU. Wasn't that simple?

      From that you can probably also see how to multithread everything, or how to set everything up with distributed parallelism. You can make the ODE solvers do whatever you want by defining an array type where the broadcast does whatever special behavior you want.

      So to recap, the entire difference from above is changing to:

      using CUDA
 const gMx = CuArray(Float32.(Mx))
@@ -1873,7 +1873,7 @@ 
      g (generic function with 1 method)

      Now we just define and solve the system of SDEs:

      prob = SDEProblem(f, g, u0, (0.0, 100.0))
 @time sol = solve(prob, SRIW1());
      retcode: Success
 Interpolation: 1st order linear
-t: 81555-element Vector{Float64}:
+t: 81839-element Vector{Float64}:
    0.0
    9.999999999999999e-5
    0.0002125
@@ -1885,36 +1885,36 @@ 
      using Plots;
+ [0.104060550857274 0.1653883261596072 … 0.5333137233968575 0.2853989799777908; 0.1616107663999656 0.43677022309105284 … 1.1844735401069573 0.5338190726863127; … ; 0.19792023228400876 0.3963751846829697 … 1.3443480335511593 0.49047353742314903; 0.1011323205819408 0.20291043559171287 … 0.5288988240849009 0.3165265982824968;;; 1.4457923830393076 1.2747578918338351 … 0.971375314318768 1.3365679766046608; 1.3437497376091263 1.4598657481493316 … 1.2325292884746961 1.299723197103482; … ; 1.3223175879623568 1.4495403379191718 … 1.122735992897897 1.1320171304938695; 1.4513207133175507 1.2343774676060586 … 1.2194075250497198 1.3616578624527444;;; 0.5671882965386423 0.6081830929559953 … 0.740947959603401 0.5761857277658581; 0.5913691760093642 0.7400887788882192 … 0.9685114928039689 0.7880377791287201; … ; 0.5907290800228191 0.7825240434920354 … 1.0172469242698596 0.8053842975972743; 0.5802038080689469 0.5552445929752783 … 0.6507129658448888 0.85761506622199]
+ [0.10056968288329887 0.17265202969590712 … 0.5397001084177289 0.28378485580962126; 0.16941556412308065 0.41527001583763895 … 1.1661114515981965 0.5412287231680786; … ; 0.19785384068801412 0.3971410284929768 … 1.2953112009576786 0.516379863853168; 0.10092990650932808 0.20214298098470243 … 0.5520248532494524 0.30697195612006284;;; 1.4456238530012353 1.2749351525737933 … 0.9712914286739704 1.3361839385845504; 1.3444394223858696 1.4595119178349385 … 1.2345957749684833 1.3032646694149115; … ; 1.3229120570916972 1.4490149457089092 … 1.1245085134644281 1.1319928672433646; 1.451627851065372 1.2344073199919041 … 1.2185947185931538 1.3617257824323996;;; 0.5670334599954301 0.6080242713782382 … 0.7415759163040838 0.5770497305072141; 0.5909521109164466 0.7401802265468154 … 0.9712318481954749 0.7874333055879799; … ; 0.5913521272069686 0.781737772505142 … 1.016550697684067 0.8061278564496909; 0.5801642108985516 0.5548413721719198 … 0.6535243868785641 0.8567262875666191]
+ [0.09796681575329125 0.17747655591618952 … 0.5447320861806395 0.2813946862533547; 0.17531820764344647 0.40007086209066434 … 1.1517739728306564 0.5462023489551027; … ; 0.19801589535608669 0.3973770466043276 … 1.2555755009080531 0.5362919136132934; 0.10099533983877251 0.2019781278933161 … 0.5716750972601247 0.30036995243787257;;; 1.445242128073513 1.275133045695468 … 0.9716407577168679 1.3367406252508987; 1.34487465459477 1.4592425757385106 … 1.2314612265352904 1.3015197893055268; … ; 1.3237278159466537 1.4498341406431146 … 1.1248450699384358 1.1294556860694824; 1.451717128841021 1.23472538948263 … 1.218575692242538 1.3602238095096921;;; 0.5674418012656576 0.6079820380834047 … 0.7409462523176191 0.5771038243840418; 0.5909628982928595 0.741442406648767 … 0.9748331387196537 0.7858381012386995; … ; 0.5914361101063688 0.7827944991244208 … 1.0126116420050135 0.8055154880594825; 0.5802865010881412 0.5545435895463084 … 0.6544394804160284 0.8568071773197784]
+ [0.09623891779399069 0.18089522139777542 … 0.5497775804297189 0.27964814517634784; 0.1791967243592887 0.3877629131256667 … 1.1357354495701077 0.552054112126012; … ; 0.19807393949301216 0.3967059936164295 … 1.223561806505113 0.5521513511401392; 0.1007622923574284 0.20145892018024197 … 0.582737602846329 0.2965877481842178;;; 1.4451827707137606 1.2748803339937038 … 0.9691987045774247 1.336738905899438; 1.3452768944913247 1.4595318691781827 … 1.2370297564360972 1.298103134719289; … ; 1.3244778525053633 1.4502442439727845 … 1.126224440816476 1.129325917437125; 1.451784459669119 1.2352783327325254 … 1.2201234062162438 1.3609214539152574;;; 0.5674612675455455 0.6071946563440839 … 0.7392255972855891 0.5766453930369189; 0.5905038599087117 0.7410263899541102 … 0.9715188980093631 0.7854326451218816; … ; 0.5920504640435316 0.7835450299962149 … 1.0132229589344621 0.8061888835200816; 0.5801200497087311 0.5542773814182085 … 0.6560159641105644 0.8555018664452224]
+ [0.09497546061878598 0.1829376056177569 … 0.5537446628594158 0.27938807165152446; 0.18206693899228957 0.378259973837804 … 1.1265686929938623 0.552641640138367; … ; 0.19802601973433492 0.3968303265219384 … 1.206769710239703 0.5632956442599034; 0.10091199070288558 0.20166325295110987 … 0.587241178157111 0.2941591927964666;;; 1.4452294234425913 1.2738439233865726 … 0.9690584791078634 1.336898497159824; 1.3446870726759803 1.4594014264005055 … 1.2390297639576897 1.2986181738101457; … ; 1.324471137210116 1.4506243446229017 … 1.1303226537525028 1.1304324175696263; 1.451583772003599 1.235004762897029 … 1.2198814816961134 1.3604503694615844;;; 0.5677426271247816 0.6078713115175683 … 0.7387308243474803 0.5790877709937182; 0.5907460596969141 0.7397268314210875 … 0.9691750724606445 0.7847327789288486; … ; 0.5909010702670413 0.7829379520968149 … 1.0106830713401724 0.8071301906155031; 0.5800122150796866 0.554274695766254 … 0.6582149743439429 0.8535556762201951]
+ [0.09412419302715229 0.18388488733709338 … 0.556269577272296 0.2786646342500474; 0.18266179958731274 0.37128003200808846 … 1.1185371492211698 0.5530965288141965; … ; 0.19842600955311862 0.3954290637239681 … 1.190124991616021 0.5705062282333637; 0.10129924804151788 0.201008622421785 … 0.5905146101390465 0.2937151218317173;;; 1.4453690651499478 1.273332488454687 … 0.9696971616667928 1.3379754372816866; 1.345296719305943 1.460482209869548 … 1.2378286486170265 1.3015541479610768; … ; 1.3244851271620033 1.451017564105682 … 1.1267889366274162 1.129542600028797; 1.4517327897644672 1.234566931153011 … 1.2195520902227273 1.3596748827725278;;; 0.5680046193478472 0.6077232092165167 … 0.7395017578412952 0.5806533071707254; 0.5902126265595362 0.7403949534507812 … 0.9631601019588111 0.7865869789734286; … ; 0.5909842991453463 0.7819326094438062 … 1.0142704556212705 0.8082389898096203; 0.5798166016669347 0.5535557499591934 … 0.6589662870067231 0.8520550284286473]
+ [0.09327715527566885 0.1837520833157658 … 0.5547538314218026 0.2784756768137166; 0.18273524499743704 0.36730292345817445 … 1.1106389006317166 0.5550337127071412; … ; 0.1990583879614706 0.3956515871178991 … 1.1775472866364791 0.573762972958295; 0.10121448674021521 0.20054466778381835 … 0.5895078868503143 0.29318546494428943;;; 1.445141727023285 1.2730769261612394 … 0.9673798489785529 1.3373863479133983; 1.3448456614482478 1.4597648841944755 … 1.2333773952925295 1.300690506684802; … ; 1.324603723069899 1.4495259683062545 … 1.128277920850338 1.1315017800491995; 1.451426502017765 1.2348715333750937 … 1.2201302035657233 1.3582287969897149;;; 0.5680517338624106 0.6072077528751204 … 0.7434381664141598 0.5807969522078669; 0.5903688931194528 0.7411635159023564 … 0.9620511526274366 0.7855194103016877; … ; 0.5905243616149379 0.7821469420269525 … 1.0110836831624619 0.8069936099168757; 0.5793963982251842 0.5531637271635883 … 0.6597882338659553 0.8521463960698725]
+ [0.09287816144061661 0.18280273015201975 … 0.5532454417846732 0.27831600049910377; 0.18257646391711482 0.3629339578781889 … 1.1015241226387116 0.5543210662742505; … ; 0.19852074554388763 0.3943176880676157 … 1.1686124804727418 0.5771072242397944; 0.10093151793706776 0.19972643950445584 … 0.5886628985523648 0.29389037763546694;;; 1.4451896143129657 1.2737452364950101 … 0.967477813009569 1.3373018921600086; 1.3452332966836933 1.4605871527288148 … 1.238032963883403 1.2997372705371248; … ; 1.3251844684299787 1.4485057407557138 … 1.1289514659205644 1.1311725747645553; 1.4521470707336102 1.2351027882768437 … 1.2187352201793291 1.35841268382234;;; 0.5683039777292934 0.6067233449584638 … 0.7396769556840253 0.5802082150741914; 0.589694830223565 0.741720920262472 … 0.969354664383331 0.7860846976884792; … ; 0.5911039364182317 0.7815734819017596 … 1.0117843891324805 0.8055608039955612; 0.5793065059097425 0.5532367446875438 … 0.6578942772299073 0.8512079113679398]
+ [0.09295889512216786 0.18322102505618745 … 0.5518402167504026 0.27726148073884743; 0.18243150166413225 0.36211891621031944 … 1.0988945252850282 0.5535677683521893; … ; 0.19715813845509672 0.39406145752233773 … 1.1641011807278796 0.5779195450324202; 0.10101134587499007 0.19941773652677997 … 0.5894438577527904 0.29405156493022216;;; 1.4454807696163767 1.2740263104193381 … 0.9676590089970752 1.3373588513848322; 1.3447585207430728 1.4628349679807675 … 1.233125657416319 1.2988171815959233; … ; 1.3261933018633436 1.4486750832417727 … 1.1222110208877918 1.1325111906042904; 1.4523924824479926 1.2353404645055188 … 1.2172895936386336 1.3596180263616449;;; 0.568583380599479 0.607145125370601 … 0.7378661908330169 0.5802495562031937; 0.5898980942726849 0.7440481295816956 … 0.9624766264132941 0.7868672814961741; … ; 0.5906986949714643 0.7821765505314462 … 1.0138327068033026 0.803384235277239; 0.5791546945288137 0.5532452423268204 … 0.6569439479322771 0.8500709783570762]
      using Plots;
 gr();
 
 # Use `Array` to transform the result back into a CPU-based `Array` for plotting
@@ -1924,491 +1924,494 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 
 
-  
+  
     
   
 
-
+
 
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 
 
-  
+  
     
   
 
-
+
 
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 
 
-  
+  
     
   
 
-
+
 
 
-
+
 
-
      We can see the cool effect that diffusion dampens the noise in [A] but is unable to dampen the noise in [B] which results in a very noisy [C]. The stiff SPDE takes much longer to solve even using high order plus adaptivity because stochastic problems are just that much more difficult (current research topic is to make new algorithms for this!). It gets GPU'd just by using CuArray like before. But there we go: solving systems of stochastic PDEs using high order adaptive algorithms with within-method GPU parallelism. That's gotta be a first? The cool thing is that nobody ever had to implement the GPU-parallelism either, it just exists by virtue of the Julia type system.
      (Note: We can also use one of the SROCK methods for better performance here, but they will require a choice of dt. This is left to the reader to try.)
      Note
      This can take a while to solve! An explicit Runge-Kutta algorithm isn't necessarily great here, though to use a stiff solver on a problem of this size requires once again smartly choosing sparse linear solvers. The high order adaptive method is pretty much necessary though, since something like Euler-Maruyama is simply not stable enough to solve this at a reasonable dt. Also, the current algorithms are not so great at handling this problem. Good thing there's a publication coming along with some new stuff...
      
+
      We can see the cool effect that diffusion dampens the noise in [A] but is unable to dampen the noise in [B] which results in a very noisy [C]. The stiff SPDE takes much longer to solve even using high order plus adaptivity because stochastic problems are just that much more difficult (current research topic is to make new algorithms for this!). It gets GPU'd just by using CuArray like before. But there we go: solving systems of stochastic PDEs using high order adaptive algorithms with within-method GPU parallelism. That's gotta be a first? The cool thing is that nobody ever had to implement the GPU-parallelism either, it just exists by virtue of the Julia type system.
      (Note: We can also use one of the SROCK methods for better performance here, but they will require a choice of dt. This is left to the reader to try.)
      Note
      This can take a while to solve! An explicit Runge-Kutta algorithm isn't necessarily great here, though to use a stiff solver on a problem of this size requires once again smartly choosing sparse linear solvers. The high order adaptive method is pretty much necessary though, since something like Euler-Maruyama is simply not stable enough to solve this at a reasonable dt. Also, the current algorithms are not so great at handling this problem. Good thing there's a publication coming along with some new stuff...
      
diff --git a/dev/showcase/massively_parallel_gpu/index.html b/dev/showcase/massively_parallel_gpu/index.html
index ea4f4900b7c..bd557ff81a3 100644
--- a/dev/showcase/massively_parallel_gpu/index.html
+++ b/dev/showcase/massively_parallel_gpu/index.html
@@ -27,4 +27,4 @@
 sol = solve(monteprob, Tsit5(), EnsembleThreads(), trajectories = 10_000, saveat = 1.0f0)
      EnsembleSolution Solution of length 10000 with uType:
 SciMLBase.ODESolution{Float32, 2, Vector{StaticArraysCore.SVector{3, Float32}}, Nothing, Nothing, Vector{Float32}, Vector{Vector{StaticArraysCore.SVector{3, Float32}}}, SciMLBase.ODEProblem{StaticArraysCore.SVector{3, Float32}, Tuple{Float32, Float32}, false, StaticArraysCore.SVector{3, Float32}, SciMLBase.ODEFunction{false, SciMLBase.AutoSpecialize, typeof(Main.lorenz), LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, OrdinaryDiffEq.Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.InterpolationData{SciMLBase.ODEFunction{false, SciMLBase.AutoSpecialize, typeof(Main.lorenz), LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Vector{StaticArraysCore.SVector{3, Float32}}, Vector{Float32}, Vector{Vector{StaticArraysCore.SVector{3, Float32}}}, OrdinaryDiffEq.Tsit5ConstantCache}, DiffEqBase.Stats, Nothing}

      Taking the Ensemble to the GPU

      Now uhh, we just change EnsembleThreads() to EnsembleGPUArray()

      sol = solve(monteprob, Tsit5(), EnsembleGPUArray(CUDA.CUDABackend()), trajectories = 10_000, saveat = 1.0f0)
      EnsembleSolution Solution of length 10000 with uType:
 SciMLBase.ODESolution{Float32, 2, uType, Nothing, Nothing, Vector{Float32}, rateType, SciMLBase.ODEProblem{StaticArraysCore.SVector{3, Float32}, Tuple{Float32, Float32}, false, StaticArraysCore.SVector{3, Float32}, SciMLBase.ODEFunction{false, SciMLBase.AutoSpecialize, typeof(Main.lorenz), LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, OrdinaryDiffEq.Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, IType, DiffEqBase.Stats, Nothing} where {uType, rateType, IType}

      Or for a more efficient version, EnsembleGPUKernel(). But that requires special solvers, so we also change to GPUTsit5().

      sol = solve(monteprob, GPUTsit5(), EnsembleGPUKernel(CUDA.CUDABackend()), trajectories = 10_000)
      EnsembleSolution Solution of length 10000 with uType:
-SciMLBase.ODESolution{Float32, 2, uType, Nothing, Nothing, tType, Nothing, P, A, IType, Nothing, Nothing} where {uType, tType, P, A, IType}

      Okay, so that was anticlimactic, but that's the point: if it were harder than that, it wouldn't be automatic! Now go check out DiffEqGPU.jl's documentation for more details, that's the end of our show.

      +SciMLBase.ODESolution{Float32, 2, uType, Nothing, Nothing, tType, Nothing, P, A, IType, Nothing, Nothing} where {uType, tType, P, A, IType}

      Okay, so that was anticlimactic, but that's the point: if it were harder than that, it wouldn't be automatic! Now go check out DiffEqGPU.jl's documentation for more details, that's the end of our show.

      diff --git a/dev/showcase/missing_physics/index.html b/dev/showcase/missing_physics/index.html index b02976e4f85..a5cebebb434 100644 --- a/dev/showcase/missing_physics/index.html +++ b/dev/showcase/missing_physics/index.html @@ -43,95 +43,95 @@ scatter!(t, transpose(Xₙ), color = :red, label = ["Noisy Data" nothing]) - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

      Definition of the Universal Differential Equation

      Now let's define our UDE. We will use Lux.jl to define the neural network as follows:

      rbf(x) = exp.(-(x .^ 2))
 
 # Multilayer FeedForward
@@ -285,39 +285,39 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
      Next, we compare the original data to the output of the UDE predictor. Note that we can even create more samples from the underlying model by simply adjusting the time steps!
      ## Analysis of the trained network
 # Plot the data and the approximation
 ts = first(solution.t):(mean(diff(solution.t)) / 2):last(solution.t)
@@ -328,95 +328,95 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
      Let's see how well the unknown term has been approximated:
      # Ideal unknown interactions of the predictor
 Ȳ = [-p_[2] * (X̂[1, :] .* X̂[2, :])'; p_[3] * (X̂[1, :] .* X̂[2, :])']
 # Neural network guess
@@ -427,55 +427,55 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
      And have a nice look at all the information:
      # Plot the error
 pl_reconstruction_error = plot(ts, norm.(eachcol(Ȳ - Ŷ)), yaxis = :log, xlabel = "t",
                                ylabel = "L2-Error", label = nothing, color = :red)
@@ -484,161 +484,161 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 
      That looks pretty good. And if we are happy with deep learning, we can leave it at that: we have trained a neural network to capture our missing dynamics.
      But...
      Can we also make it print out the LaTeX for what the missing equations were? Find out more after the break!
      Symbolic regression via sparse regression (SINDy based)
      This part of the showcase is still a work in progress... shame on us. But be back in a jiffy and we'll have it done.
      Okay, that was a quick break, and that's good because this next part is pretty cool. Let's use DataDrivenDiffEq.jl to transform our trained neural network from machine learning mumbo jumbo into predictions of missing mechanistic equations. To do this, we first generate a symbolic basis that represents the space of mechanistic functions we believe this neural network should map to. Let's choose a bunch of polynomial functions:
      @variables u[1:2]
 b = polynomial_basis(u, 4)
@@ -748,55 +748,55 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
      We are still a bit off, so we fine tune the parameters by simply minimizing the residuals between the UDE predictor and our recovered parametrized equations:
      function parameter_loss(p)
     Y = reduce(hcat, map(Base.Fix2(nn_eqs, p), eachcol(X̂)))
     sum(abs2, Ŷ .- Y)
@@ -815,109 +815,109 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
      true_prob = ODEProblem(lotka!, u0, t_long, p_)
 true_solution_long = solve(true_prob, Tsit5(), saveat = estimate_long.t)
 plot!(true_solution_long)
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
      Post Processing and Plots
      c1 = 3 # RGBA(174/255,192/255,201/255,1) # Maroon
 c2 = :orange # RGBA(132/255,159/255,173/255,1) # Red
 c3 = :blue # RGBA(255/255,90/255,0,1) # Orange
@@ -954,228 +954,228 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 Timeseries of UODE Error
 
-
-
-
-
-
-
+
+
+
+
+
+
 x(t)
 
-
-
+
+
 y(t)
 
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 Neural Network Fit of U2(t)
 
-
-
-
-
-
-
+
+
+
+
+
+
 Neural Network
 
-
-
+
+
 True Missing Term
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 Extrapolated Fit From Short Training Data
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 x data
 
-
-
+
+
 y data
 
-
-
+
+
 True x(t)
 
-
-
+
+
 True y(t)
 
-
-
+
+
 Estimated x(t)
 
-
-
+
+
 Estimated y(t)
 
-
+
 Training 
 
-
+
 Data
 
 
-
+
diff --git a/dev/showcase/ode_types/index.html b/dev/showcase/ode_types/index.html
index 991b884c7fd..ba81777210b 100644
--- a/dev/showcase/ode_types/index.html
+++ b/dev/showcase/ode_types/index.html
@@ -160,53 +160,53 @@
 plot(sol, lw = 3)
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
      Measurements.jl: Numbers with Linear Uncertainty Propagation
      The result of a measurement should be given as a number with an attached uncertainty, besides the physical unit, and all operations performed involving the result of the measurement should propagate the uncertainty, taking care of correlation between quantities.
      There is a Julia package for dealing with numbers with uncertainties: Measurements.jl. Thanks to Julia's features, DifferentialEquations.jl easily works together with Measurements.jl out-of-the-box.
      Let's try to automate uncertainty propagation through number types on some classical physics examples!
      Warning about Measurement type
      Before going on with the tutorial, we must point up a subtlety of Measurements.jl that you should be aware of:
      using Measurements
 5.23 ± 0.14 === 5.23 ± 0.14
      false
      (5.23 ± 0.14) - (5.23 ± 0.14)
      \[0.0 \pm 0.2\]
      (5.23 ± 0.14) / (5.23 ± 0.14)
      \[1.0 \pm 0.038\]
      The two numbers above, even though have the same nominal value and the same uncertainties, are actually two different measurements that only by chance share the same figures and their difference and their ratio have a non-zero uncertainty.  It is common in physics to get very similar, or even equal, results for a repeated measurement, but the two measurements are not the same thing.
      Instead, if you have one measurement and want to perform some operations involving it, you have to assign it to a variable:
      x = 5.23 ± 0.14
 x === x
      true
      x - x
      \[0.0 \pm 0.0\]
      x / x
      \[1.0 \pm 0.0\]
      With that in mind, let's start using Measurements.jl for realsies.
      Automated UQ on an ODE: Radioactive Decay of Carbon-14
      The rate of decay of carbon-14 is governed by a first order linear ordinary differential equation:
      \[\frac{\mathrm{d}u(t)}{\mathrm{d}t} = -\frac{u(t)}{\tau}\]
      where $\tau$ is the mean lifetime of carbon-14, which is related to the half-life $t_{1/2} = (5730 \pm 40)$ years by the relation $\tau = t_{1/2}/\ln(2)$. Writing this in DifferentialEquations.jl syntax, this looks like:
      # Half-life and mean lifetime of radiocarbon, in years
@@ -274,139 +274,139 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
      The two curves are perfectly superimposed, indicating that the numerical solution matches the analytic one.  We can check that also the uncertainties are correctly propagated in the numerical solution:
      println("Quantity of carbon-14 after ", sol.t[11], " years:")
 println("Numerical: ", sol[11])
 println("Analytic:  ", u[11])
      Quantity of carbon-14 after 5207.541628507064 years:
@@ -479,713 +479,713 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
      Bingo. Also in this case there is a perfect superimposition between the two curves, including their uncertainties.
      We can also have a look at the difference between the two solutions:
      plot(sol.t, getindex.(sol.u, 2) .- u, label = "")
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 
      Tiny difference on the order of the chosen 1e-6 tolerance.
      Simple pendulum: Arbitrary amplitude
      Now that we know how to solve differential equations involving numbers with uncertainties, we can solve the simple pendulum problem without any approximation. This time, the differential equation to solve is the following:
      \[\ddot{\theta} + \frac{g}{L} \sin(\theta) = 0\]
      That would be done via:
      g = 9.79 ± 0.02; # Gravitational constants
 L = 1.00 ± 0.01; # Length of the pendulum
@@ -1209,375 +1209,375 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
      Warning about Linear Uncertainty Propagation
      Measurements.jl uses linear uncertainty propagation, which has an error associated with it. MonteCarloMeasurements.jl has a page which showcases where this method can lead to incorrect uncertainty measurements. Thus for more nonlinear use cases, it's suggested that one uses one of the more powerful UQ methods, such as:
      Basically, types can make the algorithm you want to run exceedingly simple to do, but make sure it's the correct algorithm!
      
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
      Warning about Linear Uncertainty Propagation
      Measurements.jl uses linear uncertainty propagation, which has an error associated with it. MonteCarloMeasurements.jl has a page which showcases where this method can lead to incorrect uncertainty measurements. Thus for more nonlinear use cases, it's suggested that one uses one of the more powerful UQ methods, such as:
      Basically, types can make the algorithm you want to run exceedingly simple to do, but make sure it's the correct algorithm!
      
diff --git a/dev/showcase/optimization_under_uncertainty/index.html b/dev/showcase/optimization_under_uncertainty/index.html
index a0a427ac777..eb00031c5f5 100644
--- a/dev/showcase/optimization_under_uncertainty/index.html
+++ b/dev/showcase/optimization_under_uncertainty/index.html
@@ -25,47 +25,47 @@
 plot(sol, vars = (1, 3), label = nothing, xlabel = "x", ylabel = "y")
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 
      For this particular problem, we wish to measure the impact distance from a point $y=25$ on a wall at $x=25$. So, we introduce an additional callback that terminates the simulation on wall impact.
      stop_condition(u, t, integrator) = u[1] - 25.0
 stop_cb = ContinuousCallback(stop_condition, terminate!)
@@ -77,49 +77,49 @@
 plot(sol, vars = (1, 3), label = nothing, xlabel = "x", ylabel = "y")
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 
      To help visualize this problem, we plot as follows, where the star indicates a desired impact location
      rectangle(xc, yc, w, h) = Shape(xc .+ [-w, w, w, -w] ./ 2.0, yc .+ [-h, -h, h, h] ./ 2.0)
 
@@ -133,52 +133,52 @@
 end
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 
      Considering Uncertainty
      We now wish to introduce uncertainty in p[2], the coefficient of restitution. This is defined via a continuous univariate distribution from Distributions.jl. We can then run a Monte Carlo simulation of 100 trajectories via the EnsembleProblem interface.
      using Distributions
 
@@ -201,154 +201,154 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
-
      Here, we plot the first 350 Monte Carlo simulations along with the trajectory corresponding to the mean of the distribution (dashed line).
      We now wish to compute the expected squared impact distance from the star. This is called an “observation” of our system or an “observable” of interest.
      We define this observable as
      obs(sol, p) = abs2(sol[3, end] - 25)
      obs (generic function with 1 method)
      With the observable defined, we can compute the expected squared miss distance from our Monte Carlo simulation results as
      mean_ensemble = mean([obs(sol, p) for sol in ensemblesol])
      35.04924690760609
      Alternatively, we can use the Koopman() algorithm in SciMLExpectations.jl to compute this expectation much more efficiently as
      using SciMLExpectations
+
      Here, we plot the first 350 Monte Carlo simulations along with the trajectory corresponding to the mean of the distribution (dashed line).
      We now wish to compute the expected squared impact distance from the star. This is called an “observation” of our system or an “observable” of interest.
      We define this observable as
      obs(sol, p) = abs2(sol[3, end] - 25)
      obs (generic function with 1 method)
      With the observable defined, we can compute the expected squared miss distance from our Monte Carlo simulation results as
      mean_ensemble = mean([obs(sol, p) for sol in ensemblesol])
      38.801806864147714
      Alternatively, we can use the Koopman() algorithm in SciMLExpectations.jl to compute this expectation much more efficiently as
      using SciMLExpectations
 gd = GenericDistribution(cor_dist)
 h(x, u, p) = u, [p[1]; x[1]]
 sm = SystemMap(prob, Tsit5(), callback = cbs)
@@ -391,152 +391,152 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 
      Looks pretty good! But, how long did it take? Let's benchmark.
      @time solve(opt_prob, optimizer)
      u: 3-element Vector{Float64}:
  -1.3718308930076236e-8
@@ -554,52 +554,52 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 
      We now wish to minimize the same loss function as before, but introduce an inequality constraint such that the solution must have less than a 1% chance of colliding with the wall at $x=20$. This class of probabilistic constraints is called a chance constraint.
      To do this, we first introduce a new callback and solve the system using the previous optimal solution
      constraint_condition(u, t, integrator) = u[1] - constraint[1]
 function constraint_affect!(integrator)
@@ -627,553 +627,553 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
 
      That doesn't look good!
      We now need a second observable for the system. To compute a probability of impact, we use an indicator function for if a trajectory impacts the wall. In other words, this functions returns 1 if the trajectory hits the wall and 0 otherwise.
      function constraint_obs(sol, p)
     sol((constraint[1] - sol[1, 1]) / sol[2, 1])[3] <= constraint[2] ? one(sol[1, end]) :
@@ -1220,552 +1220,552 @@ 
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
-
+
diff --git a/dev/showcase/pinngpu/index.html b/dev/showcase/pinngpu/index.html
index dd37d67a019..bbef1da870b 100644
--- a/dev/showcase/pinngpu/index.html
+++ b/dev/showcase/pinngpu/index.html
@@ -51,18 +51,18 @@
               Dense(inner, inner, Lux.σ),
               Dense(inner, 1))
 ps = Lux.setup(Random.default_rng(), chain)[1]
-ps = ps |> ComponentArray
      ComponentVector{Float32}(layer_1 = (weight = Float32[0.3636208 -0.39383617 -0.16773413; 0.09720423 -0.4044283 0.014366775; … ; 0.01286949 -0.039151803 -0.102104; 0.23374496 -0.06824405 0.45146722], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_2 = (weight = Float32[-0.3095615 0.18415466 … -0.2950359 -0.048817124; 0.30899292 -0.11487545 … 0.12579848 -0.05060046; … ; -0.011713745 -0.25375083 … 0.3068704 0.036027182; 0.13253382 0.022425996 … -0.25340655 -0.32224253], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_3 = (weight = Float32[0.28352156 -0.04314348 … -0.2725906 -0.028029272; 0.21204546 -0.23346637 … -0.18303514 -0.17179897; … ; -0.062185943 0.1330197 … -0.09081532 0.049860243; -0.2159885 -0.24800158 … -0.037676103 -0.32765475], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_4 = (weight = Float32[-0.17987819 0.10206317 … 0.34234574 -0.1548034; -0.33264536 0.27802503 … 0.25237492 -0.1796244; … ; -0.22964069 -0.20550883 … -0.32485783 -0.026621599; 0.26975062 0.11063468 … -0.2770577 -0.11274284], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_5 = (weight = Float32[-0.22758488 0.2093848 … 0.4643144 0.3476517], bias = Float32[0.0;;]))
      Step 4: Place it on the GPU.
      Just plop it on that sucker. We must ensure that our initial parameters for the neural network are on the GPU. If that is done, then the internal computations will all take place on the GPU. This is done by using the gpu function on the initial parameters, like:
      ps = ps |> gpu .|> Float64
      ComponentVector{Float64}(layer_1 = (weight = [0.363620787858963 -0.3938361704349518 -0.16773413121700287; 0.09720423072576523 -0.40442830324172974 0.014366774819791317; … ; 0.012869490310549736 -0.03915180265903473 -0.10210400074720383; 0.23374496400356293 -0.06824404746294022 0.45146721601486206], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_2 = (weight = [-0.30956149101257324 0.1841546595096588 … -0.2950358986854553 -0.04881712421774864; 0.3089929223060608 -0.11487545073032379 … 0.12579847872257233 -0.05060046166181564; … ; -0.011713745072484016 -0.25375083088874817 … 0.30687040090560913 0.03602718189358711; 0.13253381848335266 0.022425996139645576 … -0.2534065544605255 -0.32224252820014954], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_3 = (weight = [0.2835215628147125 -0.04314348101615906 … -0.27259060740470886 -0.028029272332787514; 0.21204546093940735 -0.23346637189388275 … -0.18303513526916504 -0.171798974275589; … ; -0.06218594312667847 0.13301970064640045 … -0.09081532061100006 0.04986024275422096; -0.21598850190639496 -0.2480015754699707 … -0.03767610341310501 -0.32765474915504456], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_4 = (weight = [-0.17987819015979767 0.10206317156553268 … 0.3423457443714142 -0.15480339527130127; -0.33264535665512085 0.2780250310897827 … 0.25237491726875305 -0.17962439358234406; … ; -0.22964069247245789 -0.20550882816314697 … -0.32485783100128174 -0.02662159875035286; 0.2697506248950958 0.11063467711210251 … -0.2770577073097229 -0.1127428412437439], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_5 = (weight = [-0.22758488357067108 0.20938479900360107 … 0.46431440114974976 0.3476516902446747], bias = [0.0;;]))
      Step 5: Discretize the PDE via a PINN Training Strategy
      strategy = GridTraining(0.05)
+ps = ps |> ComponentArray
      ComponentVector{Float32}(layer_1 = (weight = Float32[-0.23987272 -0.16540182 -0.094613604; 0.023442311 -0.44147596 0.16966002; … ; -0.04755554 -0.26356155 0.32525864; 0.04140118 -0.38139132 -0.11030157], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_2 = (weight = Float32[-0.2858626 0.08378083 … 0.15418388 -0.015769977; 0.03877902 -0.29162017 … 0.2772236 0.12298792; … ; 0.33623242 -0.08231667 … -0.105413914 0.2065517; -0.06672987 0.12262448 … 0.27467957 -0.27302846], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_3 = (weight = Float32[-0.18684463 -0.07954761 … 0.19849238 0.076197855; 0.10584495 -0.22406884 … -0.26656708 -0.082720906; … ; -0.053375963 0.1563273 … 0.05149269 -0.31467757; 0.17352991 -0.32022938 … 0.08902629 0.18233183], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_4 = (weight = Float32[0.2726307 0.1625527 … 0.28894398 0.30599102; -0.27867422 0.28868562 … 0.15480897 0.22057188; … ; 0.15197401 0.34366935 … -0.058764257 0.2095675; 0.16755295 -0.058639627 … -0.13693371 0.24647513], bias = Float32[0.0; 0.0; … ; 0.0; 0.0;;]), layer_5 = (weight = Float32[-0.018786665 -0.43074977 … -0.33528447 0.047822163], bias = Float32[0.0;;]))
      Step 4: Place it on the GPU.
      Just plop it on that sucker. We must ensure that our initial parameters for the neural network are on the GPU. If that is done, then the internal computations will all take place on the GPU. This is done by using the gpu function on the initial parameters, like:
      ps = ps |> gpu .|> Float64
      ComponentVector{Float64}(layer_1 = (weight = [-0.23987272381782532 -0.16540181636810303 -0.09461360424757004; 0.023442311212420464 -0.4414759576320648 0.16966001689434052; … ; -0.04755553975701332 -0.26356154680252075 0.32525864243507385; 0.04140118137001991 -0.381391316652298 -0.11030156910419464], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_2 = (weight = [-0.28586259484291077 0.08378083258867264 … 0.15418387949466705 -0.015769977122545242; 0.03877902030944824 -0.2916201651096344 … 0.2772235870361328 0.12298791855573654; … ; 0.33623242378234863 -0.08231666684150696 … -0.10541391372680664 0.2065517008304596; -0.06672987341880798 0.1226244792342186 … 0.274679571390152 -0.2730284631252289], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_3 = (weight = [-0.1868446320295334 -0.07954760640859604 … 0.1984923779964447 0.07619785517454147; 0.10584495216608047 -0.22406883537769318 … -0.26656708121299744 -0.08272090554237366; … ; -0.0533759631216526 0.15632730722427368 … 0.05149269104003906 -0.31467756628990173; 0.17352990806102753 -0.3202293813228607 … 0.08902628719806671 0.18233183026313782], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_4 = (weight = [0.2726306915283203 0.16255269944667816 … 0.28894397616386414 0.3059910237789154; -0.2786742150783539 0.2886856198310852 … 0.15480896830558777 0.2205718755722046; … ; 0.15197400748729706 0.3436693549156189 … -0.0587642565369606 0.20956750214099884; 0.16755294799804688 -0.05863962695002556 … -0.13693371415138245 0.24647513031959534], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_5 = (weight = [-0.018786665052175522 -0.430749773979187 … -0.3352844715118408 0.04782216250896454], bias = [0.0;;]))
      Step 5: Discretize the PDE via a PINN Training Strategy
      strategy = GridTraining(0.05)
 discretization = PhysicsInformedNN(chain,
                                    strategy,
                                    init_params = ps)
 prob = discretize(pde_system, discretization)
      OptimizationProblem. In-place: true
-u0: ComponentVector{Float64}(layer_1 = (weight = [0.363620787858963 -0.3938361704349518 -0.16773413121700287; 0.09720423072576523 -0.40442830324172974 0.014366774819791317; … ; 0.012869490310549736 -0.03915180265903473 -0.10210400074720383; 0.23374496400356293 -0.06824404746294022 0.45146721601486206], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_2 = (weight = [-0.30956149101257324 0.1841546595096588 … -0.2950358986854553 -0.04881712421774864; 0.3089929223060608 -0.11487545073032379 … 0.12579847872257233 -0.05060046166181564; … ; -0.011713745072484016 -0.25375083088874817 … 0.30687040090560913 0.03602718189358711; 0.13253381848335266 0.022425996139645576 … -0.2534065544605255 -0.32224252820014954], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_3 = (weight = [0.2835215628147125 -0.04314348101615906 … -0.27259060740470886 -0.028029272332787514; 0.21204546093940735 -0.23346637189388275 … -0.18303513526916504 -0.171798974275589; … ; -0.06218594312667847 0.13301970064640045 … -0.09081532061100006 0.04986024275422096; -0.21598850190639496 -0.2480015754699707 … -0.03767610341310501 -0.32765474915504456], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_4 = (weight = [-0.17987819015979767 0.10206317156553268 … 0.3423457443714142 -0.15480339527130127; -0.33264535665512085 0.2780250310897827 … 0.25237491726875305 -0.17962439358234406; … ; -0.22964069247245789 -0.20550882816314697 … -0.32485783100128174 -0.02662159875035286; 0.2697506248950958 0.11063467711210251 … -0.2770577073097229 -0.1127428412437439], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_5 = (weight = [-0.22758488357067108 0.20938479900360107 … 0.46431440114974976 0.3476516902446747], bias = [0.0;;]))
      Step 6: Solve the Optimization Problem
      callback = function (p, l)
+u0: ComponentVector{Float64}(layer_1 = (weight = [-0.23987272381782532 -0.16540181636810303 -0.09461360424757004; 0.023442311212420464 -0.4414759576320648 0.16966001689434052; … ; -0.04755553975701332 -0.26356154680252075 0.32525864243507385; 0.04140118137001991 -0.381391316652298 -0.11030156910419464], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_2 = (weight = [-0.28586259484291077 0.08378083258867264 … 0.15418387949466705 -0.015769977122545242; 0.03877902030944824 -0.2916201651096344 … 0.2772235870361328 0.12298791855573654; … ; 0.33623242378234863 -0.08231666684150696 … -0.10541391372680664 0.2065517008304596; -0.06672987341880798 0.1226244792342186 … 0.274679571390152 -0.2730284631252289], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_3 = (weight = [-0.1868446320295334 -0.07954760640859604 … 0.1984923779964447 0.07619785517454147; 0.10584495216608047 -0.22406883537769318 … -0.26656708121299744 -0.08272090554237366; … ; -0.0533759631216526 0.15632730722427368 … 0.05149269104003906 -0.31467756628990173; 0.17352990806102753 -0.3202293813228607 … 0.08902628719806671 0.18233183026313782], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_4 = (weight = [0.2726306915283203 0.16255269944667816 … 0.28894397616386414 0.3059910237789154; -0.2786742150783539 0.2886856198310852 … 0.15480896830558777 0.2205718755722046; … ; 0.15197400748729706 0.3436693549156189 … -0.0587642565369606 0.20956750214099884; 0.16755294799804688 -0.05863962695002556 … -0.13693371415138245 0.24647513031959534], bias = [0.0; 0.0; … ; 0.0; 0.0;;]), layer_5 = (weight = [-0.018786665052175522 -0.430749773979187 … -0.3352844715118408 0.04782216250896454], bias = [0.0;;]))
      Step 6: Solve the Optimization Problem
      callback = function (p, l)
     println("Current loss is: $l")
     return false
 end
 
-res = Optimization.solve(prob, Adam(0.01); callback = callback, maxiters = 2500);
      u: ComponentVector{Float64}(layer_1 = (weight = [3.1851164068545774 0.28452701554026766 0.38249138298472146; -2.33177587201002 -0.947875086649432 -0.9794376571704106; … ; 1.8445853044472706 0.8727996510993542 0.9955576504008592; 3.1435412603167805 0.18696768036750094 0.4318024922818821], bias = [0.2785076581970372; 2.8846332783923105; … ; -0.9662729467711761; 0.5330462847890354;;]), layer_2 = (weight = [-0.6947993004855113 0.6591124822101054 … -0.27477478583957093 -0.2147079758432564; -0.018005979702136837 -0.5267861494383114 … 0.07444936340701067 -0.1807802019122224; … ; 1.408312706701449 0.9253224278016914 … -0.11067492215618509 1.250404124076771; 0.8785609292569058 -0.7239461655519493 … 0.18160640828220373 0.3708734997318388], bias = [-0.10919666162389878; -0.2481610079946904; … ; 0.25762627110341313; -0.2204389511316291;;]), layer_3 = (weight = [1.4906796040305386 0.4373982089490712 … -0.9126009500024581 0.3437544328384351; 1.1705258403658945 0.2184186360813869 … -1.7468547885432817 0.039350834272007355; … ; 1.2865298603405824 0.8578803762599789 … -1.4972062027062931 0.36263347078410985; 1.0465125918256124 0.4278062340569529 … -1.1959913207729769 0.07143239487100664], bias = [0.0625279945624916; -0.2105362834699611; … ; 0.088618249944539; 0.06354873118800852;;]), layer_4 = (weight = [0.7168724675941679 1.3243174982343764 … 1.5736129741077214 1.0145124106090195; -1.256325932901688 -1.1460343065463985 … -0.6886671951630514 -1.16970445308525; … ; -0.8628940027227808 -1.5741715976067723 … -1.826347286547977 -1.0600535710599828; -0.6180326006397131 -1.2092454017758947 … -1.3777201406330273 -1.1959936002668272], bias = [-0.8258272791342622; 0.8852376404822981; … ; 0.994654979621812; 0.6976595460801794;;]), layer_5 = (weight = [-2.165529102602926 6.076999720882896 … 4.6203871453940115 5.919044638420083], bias = [1.4654647270648016;;]))
      We then use the remake function to rebuild the PDE problem to start a new optimization at the optimized parameters, and continue with a lower learning rate:
      prob = remake(prob, u0 = res.u)
-res = Optimization.solve(prob, Adam(0.001); callback = callback, maxiters = 2500);
      u: ComponentVector{Float64}(layer_1 = (weight = [3.237504121770521 0.26097628701686154 0.3977387824007755; -1.7636891133027153 -0.5014220434569558 -0.5062690835844877; … ; 2.207083935534914 0.8920604291979048 1.0812974356149287; 3.179415548627458 0.09241124817811297 0.5092726336791679], bias = [0.26825313915397614; 3.8980720371198068; … ; -0.9402847773402389; 0.511389034726828;;]), layer_2 = (weight = [-0.7148634983551144 0.4700790508952156 … -0.29665997658257687 -0.2345654189974819; 0.03292773307036516 -0.561333734480355 … 0.11857521747027353 -0.13048278224778329; … ; 1.3445346316189692 1.0853623130390269 … -0.17719100107302588 1.1868556333709859; 0.8410077143944308 -0.6951511374697342 … 0.14143695426124647 0.33973979827342515], bias = [-0.12729532269737512; -0.19820629806051937; … ; 0.19419035300745818; -0.23568173441018414;;]), layer_3 = (weight = [1.5342022652651146 0.4956966475007095 … -0.987296200849101 0.32486466310604684; 1.277244533881837 0.3033903368110561 … -1.6675587325419294 0.15576635312514084; … ; 1.3599605945442708 0.9371617496866123 … -1.8190736633601887 0.29894082679235506; 1.0049217551239376 0.44257348692718956 … -1.4227592021336912 -0.01976490200429816], bias = [0.06595070104476229; -0.11209578906711677; … ; 0.0240297344828615; -0.031736854705696385;;]), layer_4 = (weight = [0.9565361067990705 1.5525604130845527 … 1.7430080707323354 1.212822281597378; -1.3060675128298052 -1.445376659331882 … -0.8248004270610534 -1.3227984934100168; … ; -1.1892417504446011 -1.577092518251138 … -1.5832656818853759 -0.8365647428690726; -0.785181237194682 -1.5827819492582023 … -1.5585119049059737 -1.3868250368124218], bias = [-0.7387712519121331; 0.8836095708306476; … ; 0.8802678459686318; 0.6867212825456077;;]), layer_5 = (weight = [-0.8569546238322109 7.680365604464502 … 4.479794944850389 7.274888738187249], bias = [2.7293365025154643;;]))
      Step 7: Inspect the PINN's Solution
      Finally, we inspect the solution:
      phi = discretization.phi
+res = Optimization.solve(prob, Adam(0.01); callback = callback, maxiters = 2500);
      u: ComponentVector{Float64}(layer_1 = (weight = [-0.8013846349882331 -0.42254003369329884 -0.3992617445519118; -0.22843803867443155 -1.112465542939997 0.631803054527572; … ; 1.0900279419560062 -1.0112499360246103 1.3864575971837405; 0.8262592135775593 -0.7816551159850413 -0.9388727472439767], bias = [1.6922704806773972; 1.7407534801436484; … ; -0.4652786049007732; 1.9791453708518176;;]), layer_2 = (weight = [-0.8858826367551943 -1.5429378142227004 … 0.5008000206438025 -1.5372638850579465; 0.3824943234460711 0.348134927156088 … 0.17348458662613953 -0.27532024418476914; … ; -0.8652121072028977 1.5071472561604367 … -1.790149128590008 -2.314389572958846; -3.75268695738018 -1.9602555035449198 … 0.9745242356239071 -2.012433180547121], bias = [-0.5531912915566448; -0.012492845163063673; … ; 0.02777540706304104; 0.1691826514242651;;]), layer_3 = (weight = [1.1378587320981821 0.34660394706187686 … 0.9894889180518479 -3.196657571878104; 0.39982501479802224 -0.8294942171375344 … -4.52895839337288 -1.0660443942667106; … ; -1.1721957962872074 -0.8952756006572734 … -2.448049382344908 0.2267017993597782; -0.31419885371568274 -1.2237761101891134 … -2.964758161356437 0.02669815783283098], bias = [-0.08656593481644438; -0.1357002884002937; … ; -0.5027589400934099; -0.5876797187775582;;]), layer_4 = (weight = [3.3890626869564597 0.6111048312424561 … 1.307732003872544 0.6102472388197515; 1.494869319733996 0.26019346718406483 … 0.6101597513082045 0.15988425679881033; … ; 2.2100407341169133 0.5241005564416775 … 0.48791027869933457 0.1821198000507933; -1.2138618962846544 -0.06267783544198241 … -0.7873581245300129 0.2153165876096704], bias = [-0.13375418994264793; -0.38952257205341173; … ; -0.2507255662283907; 0.9771733159246795;;]), layer_5 = (weight = [-5.9972667619436475 -7.469636862549854 … -7.082233009978441 6.122351829487544], bias = [-1.4713603950612173;;]))
      We then use the remake function to rebuild the PDE problem to start a new optimization at the optimized parameters, and continue with a lower learning rate:
      prob = remake(prob, u0 = res.u)
+res = Optimization.solve(prob, Adam(0.001); callback = callback, maxiters = 2500);
      u: ComponentVector{Float64}(layer_1 = (weight = [-0.8321561717054271 -0.4071851436326447 -0.3897239899942088; -0.19393906945804024 -1.0674787520202054 0.6340429763606334; … ; 1.0143546952257827 -0.9467262114806599 1.348588370473631; 0.7227743184600564 -0.7896932546487082 -0.9086446679372349], bias = [1.7090761483355272; 1.775085913105235; … ; -0.47682272905106937; 1.9378602535958755;;]), layer_2 = (weight = [-0.8545007412246189 -1.5963227112441232 … 0.47018523077283153 -1.53272140069455; 0.3892645017636183 0.34099882387060537 … 0.1776566892658072 -0.27484340367863186; … ; -0.8131449423356182 1.5321451717153562 … -1.7637026093512296 -2.254511985426994; -3.8089804282515036 -2.0016294723580628 … 0.9377661400239549 -2.081416060624415], bias = [-0.5460561286570272; -0.01653399109980723; … ; 0.019560664520413197; 0.19345371132585396;;]), layer_3 = (weight = [1.1500503953692476 0.354473093398341 … 0.9412377873674208 -3.3750738839353835; 0.4099587659939632 -0.8345434360329778 … -4.439855117925793 -1.7717242117517538; … ; -1.0646027677973022 -0.8878877529639954 … -2.5597867464584643 -0.14278643772367364; -0.15991074834302557 -1.189693856855823 … -3.007669864814197 -0.908797845547977], bias = [-0.07631034630205287; -0.10643234799905549; … ; -0.5019776053132439; -0.5777477369422738;;]), layer_4 = (weight = [3.5512490548842433 0.6365236032811511 … 1.2936541222729483 0.6107418128189199; 1.5315210872202294 0.2570197866110038 … 0.60305347785658 0.16735445377395514; … ; 2.289094738981743 0.5603276513991455 … 0.46368838155865794 0.2098572321211862; -1.1800146710317474 -0.07144926066889122 … -0.8352611091708928 0.19583252567629408], bias = [-0.11461060902694283; -0.3644304557095701; … ; -0.2935099847531113; 1.0469628358315877;;]), layer_5 = (weight = [-6.265553381317045 -7.767562644254733 … -7.284594385929302 6.2502037160828285], bias = [-1.5234004643948582;;]))
      Step 7: Inspect the PINN's Solution
      Finally, we inspect the solution:
      phi = discretization.phi
 ts, xs, ys = [infimum(d.domain):0.1:supremum(d.domain) for d in domains]
 u_real = [analytic_sol_func(t, x, y) for t in ts for x in xs for y in ys]
 u_predict = [first(Array(phi(gpu([t, x, y]), res.u))) for t in ts for x in xs for y in ys]
@@ -87,4 +87,4 @@
     gif(anim, "3pde.gif", fps = 10)
 end
 
-plot_(res)
      3pde
      
+plot_(res)
      3pde
      
diff --git a/dev/showcase/showcase/index.html b/dev/showcase/showcase/index.html
index 9f695b01a2e..bb5b7c99885 100644
--- a/dev/showcase/showcase/index.html
+++ b/dev/showcase/showcase/index.html
@@ -3,4 +3,4 @@
   function gtag(){dataLayer.push(arguments);}
   gtag('js', new Date());
   gtag('config', 'UA-90474609-3', {'page_path': location.pathname + location.search + location.hash});
-
      Edit on GitHub
      The SciML Showcase
      The SciML Showcase is a display of some cool things that can be done by connecting SciML software.
      Note
      The SciML Showcase is not meant to be training/tutorials, but inspirational demonstrations! If you're looking for simple examples to get started with, check out the getting started section.
      Want to see some cool things that you can do with SciML? Check out the following:
      
+
      Edit on GitHub
      The SciML Showcase
      The SciML Showcase is a display of some cool things that can be done by connecting SciML software.
      Note
      The SciML Showcase is not meant to be training/tutorials, but inspirational demonstrations! If you're looking for simple examples to get started with, check out the getting started section.
      Want to see some cool things that you can do with SciML? Check out the following:
      
diff --git a/dev/showcase/symbolic_analysis/index.html b/dev/showcase/symbolic_analysis/index.html
index 7f2cd38756a..72bf4cae108 100644
--- a/dev/showcase/symbolic_analysis/index.html
+++ b/dev/showcase/symbolic_analysis/index.html
@@ -30,57 +30,57 @@
 plot(sol, vars = states(traced_sys))
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
      Explanation
      Attempting to Solve the Equation
      In this tutorial, we will look at the pendulum system:
      \[\begin{aligned}
     x^\prime &= v_x\\
     v_x^\prime &= Tx\\
@@ -135,57 +135,57 @@ 
      Explanat
 plot(sol, vars = states(traced_sys))
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
      Note that plotting using states(traced_sys) is done so that any variables which are symbolically eliminated, or any variable reordering done for enhanced parallelism/performance, still show up in the resulting plot and the plot is shown in the same order as the original numerical code.
      Note that we can even go a bit further. If we use the ODAEProblem constructor, we can remove the algebraic equations from the states of the system and fully transform the index-3 DAE into an index-0 ODE, which can be solved via an explicit Runge-Kutta method:
      traced_sys = modelingtoolkitize(pendulum_prob)
 pendulum_sys = structural_simplify(dae_index_lowering(traced_sys))
 prob = ODAEProblem(pendulum_sys, Pair[], tspan)
@@ -193,57 +193,57 @@ 
      Explanat
 plot(sol, vars = states(traced_sys))
      
 
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
+
 
-  
+  
     
   
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
 
      And there you go: this has transformed the model from being too hard to solve with implicit DAE solvers, to something that is easily solved with explicit Runge-Kutta methods for non-stiff equations.
      Parameter Identifiability in ODE Models
      Ordinary differential equations are commonly used for modeling real-world processes. The problem of parameter identifiability is one of the key design challenges for mathematical models. A parameter is said to be identifiable if one can recover its value from experimental data. Structural identifiability is a theoretical property of a model that answers this question. In this tutorial, we will show how to use StructuralIdentifiability.jl with ModelingToolkit.jl to assess identifiability of parameters in ODE models. The theory behind StructuralIdentifiability.jl is presented in paper [4].
      We will start by illustrating local identifiability in which a parameter is known up to finitely many values, and then proceed to determining global identifiability, that is, which parameters can be identified uniquely.
      To install StructuralIdentifiability.jl, simply run
      using Pkg
 Pkg.add("StructuralIdentifiability")
      The package has a standalone data structure for ordinary differential equations, but is also compatible with ODESystem type from ModelingToolkit.jl.
      Local Identifiability
      Input System
      We will consider the following model:
      \[\begin{cases}
 \frac{d\,x_4}{d\,t} = - \frac{k_5 x_4}{k_6 + x_4},\\
@@ -341,4 +341,4 @@ 
      Explanat
                                    funcs_to_check = to_check, p = 0.9)
 # Dict{Num, Symbol} with 2 entries:
 #   b => :globally
-#   c => :globally
      Both parameters b, c are globally identifiable with probability 0.9 in this case.
      
+#   c => :globally
      Both parameters b, c are globally identifiable with probability 0.9 in this case.