How did I get these results?
A framework for managing hyper-parameter settings, and running experiments. Lightly based on litetracer from Borealis.
This repository is for giving you the tools you need to make experiments reproducible. This repository is mostly built around machine learning and reinforcement learning projects, but there is no reason it is restricted to these types of projects. I've developed this around my own tastes and needs, but should be generally usable for any style of experiment which needs to do massively parallel parameter sweeps of a set of functions.
The best way to see how to use this repository is in the examples folder. parallel.jl shows all the parallel and experiment utilities in action, while running an experiment in experiment.jl which shows off the data management, and arg parsing.
To run the config example:
julia --project examples/parallel_config.jl examples/test_config.toml 5 test_config --numworkers 6Temporarily, I've updated a local version of the SLURM cluster managers as the ClusterManagers.jl package updates to 1.x.
The goal is to make a convenient framework for data analysis for experiments in ML/RL. The goal is to create something like mlflow for Julia. It might be worthwhile to actually take advantage of the mlflow framework, or even a similar framework for the future.
- Define arg_iterators in terms of config files
- Define compatibility w/ arg settings and arg parse setting.
- Allow for TOML/JSON specification -> Implemented through Config.jl
- Data searching (Mostly done. Could probably use more testing...)
- Plotting utilities (Up Next!)
- Frontend for visualizing data.
- Testing with slurm backend. (Make sure your experiments run BEFORE a job is scheduled.)
- Move away from JLD2 for settings files. CSV? TOML?
- More examples. (maybeeee.....)
- Docs
HELP WANTED! Contact me via issues panel, or setup a PR.
- JLD2 has a bug where sometimes the io resource becomes busy. I'm unsure when this happens, or how to reproduce but there is some discussion happening on the Julia Discourse: https://discourse.julialang.org/t/error-ebusy-on-saving-jld2-file-using-julia-1-3-0-rc2/29228/10. If you run into this error close your terminal session and start a new one, this fixes it for me.