Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
ex4sperans authored Apr 9, 2020
1 parent 12f0c0c commit d27461b
Showing 1 changed file with 15 additions and 14 deletions.
29 changes: 15 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

Main issues **mag** (at least partially) solves:

* Removes the need of meditations on what is a proper name for the experiment. Say you are a machine learning researcher/engineer and you want to train a convolutional neural network with a particular set of parameters, say, 50 convolutional layers, dropout 0.5 and relu activations. You might want to create a separate directory for this experiment to store some checkpoints and summaries there. If you do not expect to have a lot of different models you can simply go off with something like "convnet50layers" or "convnet50relu". But if the number of experiments grows, you clearly need some more reliable and automated solution. **mag** offers such a solution, so any experiment you run will have a name derived from the parameters of your model. For the mentioned model it would be "50|relu|0.5".
* Assists reproducibility. Ever experienced situation when results you got a month ago with an "old" model are no longer reproducible? Even if you are using git, you probably have used some command line arguments that are now lost somewhere in the bash history... **mag** stores all command line parameters in a file and duplicates stdout an another file. Additionaly, it saves exact git commit hash so you can easily checkout to it later and run the same code with the same parameters.
* Removes the need of meditations on what is a proper name for the experiment. Say you are a machine learning researcher/engineer and you want to train a convolutional neural network with a particular set of parameters, say, 50 convolutional layers, dropout 0.5 and relu activations. You might want to create a separate directory for this experiment to store some checkpoints and summaries there. If you do not expect to have a lot of different models you can simply go off with something like "convnet50layers" or "convnet50relu". But if the number of experiments grows, you clearly need some more reliable and automated solution. **mag** offers such a solution, so any experiment you run will have a name derived from the parameters of your model. For the mentioned model it would be "50-relu-0.5".
* Assists reproducibility. Ever experienced situation when results you got a month ago with an "old" model are no longer reproducible? Even if you are using git, you probably have used some command line arguments that are now lost somewhere in the bash history... **mag** stores all command line parameters in a file and duplicates the stdout to an another file. Additionaly, it saves exact git commit hash so you can easily checkout to it later and run the same code with the same parameters.
* Restoring a model is now really painless! Since **mag** saves all the parameters you used to run the experiment, all you need to restore a model is to provide a path to a saved experiment.

Let's consider a toy example and train an SVM on Iris dataset.
Expand Down Expand Up @@ -66,20 +66,20 @@ From here you can reach the model identifier:

```
>>> experiment.identifier
5|1.0|0.01
5-1.0-0.01
```

Or the experiment directory:

```
>>> experiment.experiment_dir
./experiments/5|1.0|0.01
./experiments/5-1.0-0.01
```

Lets examine what this directory contains by now.

```
tree experiments/5\|1.0\|0.01/
tree experiments/5-1.0-0.01/
├── command
├── config.json
Expand All @@ -106,7 +106,7 @@ with experiment:
).mean()
```

Note that we can reach parameters using dot notation rather than \["keyword"\] notation, which looks much nicer.
Note that we can reach parameters using dot notation rather than `["keyword"]` notation, which looks much nicer.

We can print accuracy and this will be stored in a log file:

Expand All @@ -130,7 +130,7 @@ This creates `results.json` file in experiment directory with the following cont

Later we can use such files from different experiments to be able to compare them.

Finally, lets save the model using pickle.
Finally, lets save the model using **pickle** module.

```python
with open(os.path.join(experiment.experiment_dir, "model.pkl"), "wb") as f:
Expand All @@ -150,7 +150,8 @@ See how directory structure has changed:
If we want to restore the experiment we can easily do:

```python
with Experiment(resume_from="experiments/5|1.0|0.01") as experiment:
with Experiment(resume_from="experiments/5-1.0-0.01") as experiment:
config = experiment.config # the same config we created on the training phase
...
```

Expand All @@ -174,12 +175,12 @@ python -m mag.summarize experiments --metrics=accuracy
Results for experiments:
accuracy
5|0.001|10.0 0.793333
5|0.001|0.1 0.933333
5|1.0|0.01 0.933333
5|10.0|0.1 0.980000
5|10.0|0.01 0.966667
5|10.0|1.0 0.960000
5-0.001-10.0 0.793333
5-0.001-0.1 0.933333
5-1.0-0.01 0.933333
5-10.0-0.1 0.980000
5-10.0-0.01 0.966667
5-10.0-1.0 0.960000
```

Various other options are available and will be documented soon.
Expand Down

0 comments on commit d27461b

Please sign in to comment.