AutoBDMC is a tool based on bidirectional Monte Carlo to accurately evaluate and bound log likelihoods of synthetic data via annealed importance sampling in an automated fashion by integrating it with probabilisitic programming languages, such as Stan and WebPPL.
This repository contains an implementation of this tool in Stan and WebPPL, as well as experiments that validate the use of this tool and demonstrate scientific findings produced by it.
Run: make install
This installs WebPPL. Stan need not be installed as it compiles on the go.
Each experiment is represented by a sub-sub-folder in the "experiments" folder. To run an experiment from scratch, run
bash experiments/<EXP_NAME>/<SUBEXP_NAME>/scripts/main.sh [--run] [--parallel] [--clean]--run
To actually run the experiment.
--parallel
An experiment consists of a set of independent jobs, which are run serially by default. If you have access to a CPU cluster with GNU parallel installed, you can use this flag to parallelize the jobs.
--clean
The script generates data files, exact-sample files, job lists, results and plots. Use this flag to delete all generated files and clean the directory, in order to re-run the experiment. Caution: all unsaved work will be lost.
Example: bash experiments/fixed_hp/linear_regression/scripts/main.sh --run --parallel
After the script finishes execution, generated plots will be available in experiments/EXP_NAME/SUBEXP_NAME/plots.