This repository contains scripts for benchmarking dynamical systems on networks. Usage Instructions are found below. Details on our methodology are reported in our [paper]((https://arxiv.org/abs/2012.12696) on NetworkDynamics.jl. Below you see plots of competing runtimes simulating a Kuramoto network of 10, 100 and 1000 oscillators.
In the concrete implementations we tried to strike a balance between usability (writing code should feel "simple") and performance. We compare two implementations in Python (SciPy and JiTCODE, the later JIT-compiles a C function for the ODE), two in Julia (NetworkDynamics.jl and incidence matrix multiplication with SparseArrays.jl) and one in Fortran (similar approach as for SparseArrays.jl).
The JITCODE (and SciPy) version is derived from this publication and does not make use of the symmetry of the sine function 𝑠𝑖𝑛(𝑦−𝑥)=−𝑠𝑖𝑛(𝑥−𝑦), which means that the right hand side function evaluates roughly twice as many floating point operations.
All programs employ Dormand-Prince's 5th order Runge-Kutta method. For Julia we use DifferentialEquations.jl to solve the ODE.
When startup and JIT-compile time is taking into account we see a different picture. The following plot shows the time until the first trajectory is solved.
Fortran isn't shown in this plot since that program doesn't have a JIT compile stage. In fact the whole Fortan benchmark (300 integrations) finishes roughly in the time it takes to startup a Julia session and import all required libraries (Julia 1.5.1). If you know how to improve the benchmarks or want to add your favourite network dynamics solution in yet another language, we are happy about contributions. Just open a pull request.
- sqlite
- make
- python3
- Julia 1.5
We suggest a virtual env for python
python3 -m venv .env
activate and install the requirements
source .env/bin/activate
pip install -r src/python/requirements.pip
The julia env should be activated automatically by the scripts.
If you would like to run the fortran benchmarks, you must have a Fortran 2003 compiler. In the example we use GFortran. If you do not wish to run the fortran benchmark, bypass this section.
The fortran program uses FPM, the fortran package manager. Install from https://github.com/fortran-lang/fpm
Using this tool, you can build the fortran program with fpm
cd src/fortran
fpm build --flag -ldl --flag -lpthread
now, set KURABENCH_FEXE to the newly built binary
cd ../../
export
KURABENCH_FEXE=`pwd`/src/fortran/build/<compiler>_<uuid>/app/FKuraBenchmark
where compiler and uuid are the details of your compilation process resulting from the fpm build command.
In these benchmarks, we use an SQLite database to store the setup of each system (node parameters, edge parameters, system parameters, connectivity) and also the experiment log and trajectory outputs.
You may place your database at a location of your choosing. Each program
uses the environmental variable KURABENCH_DB to identify the location of
the database. In this case, we will place the database in the data
directory.
export KURABENCH_DB=`pwd`/data/kurabench.db
make database
Be sure that your python virtual env is activated
make systems
The "ground-truth" trajectories are generated using the (implicit) radau solver at low tolerance settings. Julia is used to run this generation step. Some timing information will be printed to stdout during the program execution. Compute the trajectories with:
make reference
Now, generate any benchmarks of interest by running
make <X>
where <X> is one of
networkdynamics diffeq jitcode scipy fortran
The resulting experiment summary is stored in the database in the
experiments table. The trajectory of each state is stored in the
states table, indexed by the experiment id found in the experiments
table.
The experiments table includes the timing and err_v_ref, the average
deviation (across all states) of the experiment trajectory and the
reference (radau-computed) trajectory for that system and initial
condition.