update/README

.gitignore

@@ -8,6 +8,7 @@ lib/
 pyvenv.cfg
 lop.egg-info
 __pycache__/
+share/
 *.pyc
 data/
 .DS_Store

README.md

@@ -1,26 +1,73 @@
 # Loss of Plasticity in Deep Continual Learning
 
-This repository contains the implementation of three continual supervised learning problems.
-In our forthcoming paper _Loss Plasticity in Deep Continual Learning_,
-we show the loss of plasticity in deep learning in these problems.
+## Contents
+
+- [Overview](#overview)
+- [Repository Contents](#repo-contents)
+- [System Requirements](#system-requirements)
+- [Installation Guide](#installation-guide)
+- [License](./LICENSE)
+- [Citation](./citation.bib)
+
+## Overview
+
+Artificial neural networks, deep-learning methods, and the backpropagation algorithm form the foundation of modern machine learning and artificial intelligence.
+These methods are almost always used in two phases, one in which the network's weights are updated, and one in which the weights are held constant while the network is used or evaluated.
+This contrasts with natural learning and many applications, which require \textit{continual} learning.
+Do deep learning methods work in continual learning settings?
+Here we show that they do not---that standard deep-learning methods gradually lose plasticity in continual learning settings until they learn no better than a shallow network.
+We show such loss of plasticity using the classic ImageNet dataset and reinforcement learning problems across a wide range of variations in the network and the learning algorithm. 
+Plasticity is maintained indefinitely only by algorithms that continually inject diversity into the network, such as our \emph{continual backpropagation} algorithm, a variation of backpropagation in which a small fraction of less-used units are continually and randomly reinitialized.
+Our results suggest that methods based on gradient descent are not enough---that sustained deep learning requires a random, non-gradient component to maintain variability and plasticity.
+
 
 A talk about this work can be found [here](https://www.youtube.com/watch?v=p_zknyfV9fY), 
 and the [paper](https://arxiv.org/abs/2306.13812) is available on arxiv.
+This repository contains the code to reproduce the experiments present in the paper.
+
+## Repository Contents
+- [lop/algos](./lop/algos): All the algorithms used in the paper, including our new continual backpropagation algorithm.
+- [lop/nets](./lop/nets): The network architectures used in the paper.
+- [lop/imagenet](./lop/imagenet): Demonstration and mitigation of loss of plasticity in a task-incremental problem using ImageNet.
+- [lop/incremental_cifar](./lop/incremental_cifar): Demonstration and mitigation of loss of plasticity in a class-incremental problem.
+- [lop/slowly_changing_regression](./lop/slowly_changing_regression): A small problem for quick demonstration of loss of plasticity.
+- [lop/rl](./lop/rl): Loss of plasticity in standard reinforcement learning problems using the PPO algorithm[1].
+
+The README files in each subdirectory contains further information on the contents of the subdirectory.
+
+## System Requirements
+
+This package only requires a standard computed with sufficient RAM (8GB+) to reproduce the experimental results.
+However, a GPU can significantly speed up experiments with larger networks such as the residual networks in [lop/incremental_cifar](./lop/incremental_cifar).
+Internet connection is required to download many of the datasets and packages.
+
 
-# Installation
+The package has been tested on Ubuntu 20.04 and python3.8. We expect this package to work on all machines that support all the packages listed in [`requirements.txt`](requirements.txt)
 
+
+## Installation Guide
+
+Create a virtual environment
 ```sh
 mkdir ~/envs
 virtualenv --no-download --python=/usr/bin/python3.8 ~/envs/lop
 source ~/envs/lop/bin/activate
 pip3 install --no-index --upgrade pip
+```
+
+Download the repository and install the requirements
+```sh
 git clone https://github.com/shibhansh/loss-of-plasticity.git
 cd loss-of-plasticity
 pip3 install -r requirements.txt
 pip3 install -e .
 ```
 
-Add these lines in your ~/.zshrc or ~/.bashrc
+Add this lines in your `~/.zshrc` or `~/.bashrc`
 ```sh
 source ~/envs/lop/bin/activate
-```
+```
+
+Installation on a normal laptop with good internet connection should only take a few minutes
+
+[1] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

citations.bib

@@ -0,0 +1,6 @@
+@article{dohare2023loss,
+      title={Loss of Plasticity in Deep Continual Learning},
+      author={Dohare, Shibhansh and Hernandez-Garcia, J Fernando and Rahman, Parash and Sutton, Richard S and  Mahmood, A Rupam},
+      journal={\textit{arXiv preprint arXiv:2306.13812}},
+      year={2023},
+}

lop/imagenet/README.md

@@ -1,21 +1,22 @@
 # Loss of Plasticity in Continual ImageNet
-This repository contains the implementation of the continual binary ImageNet classification problem.
+This directory contains the implementation of the continual binary ImageNet classification problem.
 
 The first step to replicate the results is to download the data. The data can be downloaded [here](https://drive.google.com/file/d/1i0ok3LT5_mYmFWaN7wlkpHsitUngGJ8z/view?usp=sharing).
 Create a directory named `data` and extract the downloaded file in `data`
 ```sh
+cd lop/imagenet/
 mkdir data
 ```
 
 The next step is to test a deep convolution network that uses backprop to learn.
-The network is specified in `../net/conv_net.py`
+The network is specified in [`../nets/conv_net.py`](../nets/conv_net.py)
 This command produces 30 temporary cfg files in `temp_cfg`.
 
 ```sh
 python3.8 multi_param_expr.py -c cfg/bp.json 
 ```
 
-Each of the new temporary cfg files can then be used to do one run of backprop.
+Each of the new temporary cfg files can then be used to do one run of backprop. Each run takes about 12 hours on an A-100 GPU.
 ```sh
 python3.8 expr.py -c temp_cfg/0.json 
 ```

lop/incremental_cifar/README.md

@@ -3,16 +3,6 @@
 This repository contains the implementation of the incremental CIFAR-100 problem, which should
 allow for the reproduction of the results in the paper.
 
-### Set Up
-The first step to replicate the results is to set the correct python path and activate the 
-virtual environment; this is assuming you already followed the steps in the installation.
-You can run the two commands in your terminal below by replacing `PATH_TO_DIR` with the 
-correct path to where the project `loss-of-plasticity` is located.
-
-```sh
-source PATH_TO_DIR/loss-of-plasticity/bin/activate
-export PYTHONPATH=${PYTHONPATH}:PATH_TO_DIR/loss-of-plasticity
-```
 
 ### Running the Experiment 
 
@@ -36,7 +26,7 @@ The script takes three arguments:
     increase in length as new classes are added. 
 
 With this in mind, you can run the experiment for the base deep learning system using the following 
-command:
+command. A single run with the base deep-learning system takes about 6 hours on an A-100 GPU.
 
 ```sh
 python3.8 incremental_cifar_experiment.py --config ./cfg/base_deep_learning_system.json ---verbose \
@@ -57,7 +47,7 @@ which is specified in the config file and is by default the name of the `json` f
 Inside of this directory, there will be several directories corresponding to different metrics.
 Each metric directory contains files named `index-${experiment_index}.npy` corresponding to the given
 experiment index. 
-You can then use these files to reproduce the plots in the paper. 
+You can then use these files to reproduce the plots in the paper.
 
 ### Post-Run Analysis
 
@@ -78,15 +68,15 @@ With this in mind, you can run the script for the `base_deep_learning_system` by
 line:
 
 ```sh
-python post_run_analysis.py --results_dir ./results/base_deep_learning_system/
+python3.8 post_run_analysis.py --results_dir ./results/base_deep_learning_system/
 ```
 
 ### Reproducing the Plots in the Paper
 
 You can generate the plots found in the paper using the following three commands
 
 ```sh
-python python ./plots/plot_incremental_cifar_results.py --results_dir ./results/ \
+python3.8 ./plots/plot_incremental_cifar_results.py --results_dir ./results/ \
 --algorithms base_deep_learning_system,continual_backpropagation,retrained_network \
 --metric relative_accuracy_per_epoch
 ```
@@ -101,12 +91,12 @@ Whereas the following two commands generate the blue and red lines in the center
 the figure above:
 
 ```sh
-python ./plots/plot_incremental_cifar_results.py --results_dir ./results/ \
+python3.8 ./plots/plot_incremental_cifar_results.py --results_dir ./results/ \
 --algorithms base_deep_learning_system,continual_backpropagation \
 --metric next_task_dormant_units_analysis
 
 
-python ./plots/plot_incremental_cifar_results.py --results_dir ./results/ \
+python3.8 ./plots/plot_incremental_cifar_results.py --results_dir ./results/ \
 --algorithms base_deep_learning_system,continual_backpropagation \
 --metric next_task_stable_rank_analysis
 ```
@@ -115,7 +105,7 @@ Finally, to plot the raw accuracies for the base deep learning system, continual
 retrained network baseline use the following command:
 
 ```sh
-python python ./plots/plot_incremental_cifar_results.py --results_dir ./results/ \
+python3.8 ./plots/plot_incremental_cifar_results.py --results_dir ./results/ \
 --algorithms base_deep_learning_system,continual_backpropagation,retrained_network \
 --metric test_accuracy_per_epoch
 ```

lop/permuted_mnist/README.md

@@ -4,6 +4,7 @@ This repository contains the implementation of the online Permuted MNIST problem
 The first step to replicate the results is to download the data.
 
 ```sh
+cd lop/permuted_mnist
 mkdir data
 python3.8 load_mnist.py
 ```
@@ -29,7 +30,7 @@ python3.8 bp_metrics.py --cfg_file ../cfg/bp/std_net.json --metric accuracy
 ```
 
 The command above will generate the plot on the left below.
-I have found the variance among individual runs to be surprisingly small.
+The variance among individual runs is surprisingly small.
 Just one run can be used for quick prototyping.
 
 ![](plots/mnist_accuracy.png "BP on Online Permuted MNIST")

lop/rl/README.md

@@ -0,0 +1,24 @@
+# Loss of plasticity in reinforcement learning
+
+This directory contains the code to demonstrate and mitigate loss of plasticity in reinforcement learning problems from the OpenAI [Gym](https://www.gymlibrary.dev/index.html).
+The actor and critic networks are specified in [`../net/policies.py`](../nets/policies.py) and [`../net/valuesf.py`](../nets/valuefs.py) respectively.
+
+The configurations for individual experiments can be found in [`cfg`](cfg). [`cfg/ant/std.yml`](cfg/ant/std.yml) specifies the parameters for _standard PPO_ in the _Ant-v3_ environment.
+The following command can be used to perform one run for this configuration file. The `-s` parameter specifies the random seed for the experiment. 
+A single run (for 50M time-steps) on a normal laptop takes about 24 CPU-hours.
+
+```sh
+python3.8 run_ppo.py -c cfg/ant/std.json -s 0
+```
+
+Configuration files in [`cfg/ant/ns.yml`](cfg/ant/ns.yml), [`cfg/ant/l2.yml`](cfg/ant/l2.yml), and [`cfg/ant/cbp.yml`](cfg/ant/cbp.yml)specify the parameters for
+PPO with proper Adam, PPO with L2 regularization, and PPO with continual backpropagation respectively.
+
+After completing 30 runs for the four configuration files specified above, the commands below can be used to plot the left figure below.
+The generated figures will be in the [`plots`](plots) directory.
+```sh
+cd plots/
+python3.8 fig4a.py 
+```
+
+![](ant.png "Various algorithms on Ant-v3")

lop/slowly_changing_regression/README.md

@@ -5,6 +5,7 @@ previously known as the bit-flipping problem.
 The first step to replicate the results is to create the data.
 
 ```sh
+cd lop/slowly_changing_regression
 mkdir env_temp_cfg temp_cfg
 python3.8 multi_param_expr.py -c cfg/prob.json 
 ```
@@ -26,7 +27,7 @@ python3.8 multi_param_expr.py -c cfg/sgd/bp/relu.json
 
 To test continual backprop, use a different cfg file, like `cfg/sgd/cbp/relu.json`
 
-Each of the new temporary cfg files can then be used to do one run of backprop.
+Each of the new temporary cfg files can then be used to do one run of backprop. Each run takes about 15 CPU-minutes on a normal laptop.
 ```sh
 python3.8 expr.py -c temp_cfg/0.json 
 ```