Instructions are given below for each of the five steps:
- download Git software
- download Docker software
- clone GitHub respository containing the demo code into local directory
- create a Docker container locally to run the demo Python code
- run the demo Python code in the created Docker container
In order to avoid potential problems with installing Python and the needed packages to run the book's demo code on different platforms such as Mac, Windows, or Linux,
we decided to create a Docker container and to run the demo code in it.
In software engineering parlance, a container packages up code and all its dependencies into a standard unit of software so that the application can run
anywhere, as long as the container engine supports the underlying operating system.
Docker is sufficiently popular nowadays so that installing as well as running Docker on different platforms should be well supported and documented.
Personal and small business use is still free, though a sign-up for a Docker account may be required.
Git is a very popular source code management tool for version control, widely used among software professionals.
Estimated time: 15 mins.
Go to Install Git, and scroll to Install Git on [Mac OS X | Windows | Linux] section as appropriate.
Unless you have a preference, probably just pick the first method for your platform.
(For Mac:
Git for Mac Installer > pick latest link (currently git-2.33.0-intel-universal-mavericks.dmg);
if you're installing from a downloaded .dmg file, you may be blocked from opening the installation package; if you see
"macOS cannot verify that this app is free from malware", go to System Preferences > Security & Privacy > click Open Anyway for that downloaded git-xxx.pkg file.
Install just Git; you should not need to install git-credential-osxkeychain helper.)
Docker is a very popular container technology. The containers run on Docker Engine.
Estimated time: 15 mins.
Go to Get Docker, and pick Docker [Desktop for Mac | Desktop for Windows | for Linux] to do the appropriate install.
For Mac and Windows, after installing Docker Desktop, find the application icon to run the Docker app (Docker Desktop), so as to start Docker Engine;
it could take about half a minute to start; then you could minimize or close the Docker Dashboard window (double check that Docker Desktop is still running).
For Linux, install Docker Engine then start Docker.
Estimated time: 10 mins.
The commands shown in the text area below do the following; the text area also shows the Terminal console response to the commands:
- Clone with
git clone https://github.com/...the specified GitHub repository at a Terminal command line; this downloads the demo source code from that specified repository into your local computer cd DeepLearningPython35into the repository directory; this changes your directory to the directory of the downloaded demo Python source code- Verify with
git branchthat you are on the master branch of the repository; the branch you are on is marked with an asterisk (*); a repository can have many versions of the source code, each stored in its own branch - Checkout the desired branch instead of master branch, with
git checkout chap1_30-hidden-neurons-3.0-eta; that specific branch has the desired setup of demo code you want to run for this Part 1 class - Verify with
git branchagain that you are on the desired branch chap1_30-hidden-neurons-3.0-eta which is now marked with an asterisk (*)
~ $
~ $ git clone https://github.com/clkim/DeepLearningPython35.git
~ $ cd DeepLearningPython35
~/DeepLearningPython35 $ git branch
chap1_30-hidden-neurons-3.0-eta
chap2_fully-matrix-based-backpropagation-mini-batch
chap6
* master
~/DeepLearningPython35 $ git checkout chap1_30-hidden-neurons-3.0-eta
~/DeepLearningPython35 $ git branch
* chap1_30-hidden-neurons-3.0-eta
chap2_fully-matrix-based-backpropagation-mini-batch
chap6
master
~/DeepLearningPython35 $
(Skip until class) To run the desired setup of demo code, "uncomment in" or "comment out" as appropriate the code in test.py in order to specify the neural network and deep learning configuration to run.
(Skip until class) To see an example of the (flexible but somewhat hackish and minimalist) changes I made in test.py in order to run the demo
in the chap1 branch, at command line run
git diff ea229ac 6ba2425
to see the small changes committed in the branch
(red text is deleted, green text is added; hit space bar once to scroll down one page;
when we see (END) of document, enter q to quit and get back to the command line).
Acknowledgement: The repository is forked from the DeepLearningPython35 repository of Michal Daniel Dobrzanski who ported the book's code from Python 2.7 to Python 3.5 and wrote the "orchestrator" testing file test.py.
Background: We want to set up a bind mount in the container whose source is the directory in our local computer where the demo Python
source code has been cloned from GitHub, in order that the source code on our local computer would be accessible from inside the container.
We also want to install the Python package Numpy in the container we want to create.
Ensure that you have already started Docker Engine, e.g. by running Docker app (Docker Desktop) locally;
and that you have already cloned the demo Python source code from GitHub, into the DeepLearningPython35 directory,
as described above in "How to clone GitHub repository into local directory".
You must be at the DeepLearningPython35 directory, because the bind mount being set up into the container references pwd, the current directory.
cd into the directory DeepLearningPython35 if not already there.
Estimated time: 20 - 30 mins.
The commands shown in the text area below do the following; the text area also shows the Terminal console response to the commands:
- First run
docker pull continuumio/miniconda3to download the miniconda3 image which contains Python and conda, a package manager as well as an environment manager tool; miniconda can be considered a mini version of Anaconda which is a very popular data science platform - Then run
docker imagesto verify the image continuumio/miniconda3 is downloaded - Then run the given
dockercommand to create a new container layer over the downloaded image- At the interactive shell command line inside the container, we check the conda version, with
conda --version - Then create our own environment, named nndlbook for our use, with
conda create --nndlbook - (May be prompted to update to a new version of conda)
- Then we activate this new nndlbook conda environment, with
conda activate nndlbook - Next, do a confirming check that no packages are installed yet, with
conda list - And do a sanity check that we do have python installed, with
python --version - Now, we are ready to install our package, with
conda install numpy - We do a sanity check that we see the
numpypackage name, among others, withconda list - Finally, we check the python version which may be different in our nndlbook environment after installing
numpy - We exit our newly created local container, with
exit
- At the interactive shell command line inside the container, we check the conda version, with
- Back at the Terminal console, verify that we have created a local container named deeplearning, with
docker container ls --all
~/DeepLearningPython35 $
~/DeepLearningPython35 $ docker pull continuumio/miniconda3
Using default tag: latest
...
docker.io/continuumio/miniconda3:latest
~/DeepLearningPython35 $ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
continuumio/miniconda3 latest xxxxxxxxxxxx n months ago nnnMB
~/DeepLearningPython35 $ docker run -it --name deeplearning --mount type=bind,source="$(pwd)",target=/deeplearn continuumio/miniconda3
(base) root@xxx:/# conda --version
conda 4.10.3
(base) root@xxx:/# conda create --name nndlbook
Collecting package metadata (current_repodata.json): done
...
< if see ==> WARNING: A newer version of conda exists. <== >
< current version: 4.10.3 >
< latest version: 4.11.0 >
< exit the 'conda create' command by entering n at prompt to Proceed >
< then update conda by running the suggested command at prompt #: >
< conda update -n base -c defaults conda >
< answer y to proceed >
< then re-run at command prompt #: conda create --name nndlbook >
Proceed ([y]/n)? y
...
Executing transaction: done
...
(base) root@xxx:/# conda --version
conda 4.11.0
(base) root@xxx:/# conda activate nndlbook
(nndlbook) root@xxx:/# conda list
# packages in environment at /opt/conda/envs/nndlbook:
#
# Name Version Build Channel
(nndlbook) root@xxx:/# python --version
Python 3.9.5
(nndlbook) root@x:/# conda install numpy
Collecting package metadata (current_repodata.json): done
...
Proceed ([y]/n)? y
...
...
Executing transaction: done
(nndlbook) root@xxx:/# conda list
< Should see list of packages including numpy and numpy-base >
(nndlbook) root@xxx:/# python --version
Python 3.9.7
< note that we have a later Python version after installing Numpy >
(nndlbook) root@xxx:/# exit
exit
~/DeepLearningPython35 $ docker container ls --all
< Should see a container named "deeplearning" >
CONTAINER ID IMAGE COMMAND CREATED ... NAMES
xxxxxxxxxxxx continuumio/miniconda3 "/bin/bash" xxx deeplearning
Estimated time: 10 mins.
Ensure that you have already started Docker Engine, e.g. by running Docker app (Docker Desktop) locally;
and that you have already cloned the demo Python source code from GitHub, into the DeepLearningPython35 directory, as described above
in "How to clone GitHub repository into local directory".
cd into the directory DeepLearningPython35 if not already there.
You must be on the branch chap1_30-hidden-neurons-3.0-eta.
Verify with git branch (see section on "How to clone GitHub repository into local directory").
If not, do git checkout chap1_30-hidden-neurons-3.0-eta to switch to that branch, then verify with git branch.
The commands shown in the text area below do the following; the text area also shows the Terminal console response to the commands:
- First, just verify we see the newly created container named deeplearning
- At DeepLearningPython35 directory, start
dockercontainer deeplearning and specifying option to attach an interactive shell, withdocker container start -ai deeplearning- At the interactive shell command line inside the container, we
cdinto the deeplearn directory mounted into the container; when we created the container, we had bind that mount to the local DeepLearningPython35 directory, which must be already on the git branch chap1_30-hidden-neurons-3.0-eta - And verify what conda environments we have, with
conda info --env - Then we activate our own previously created conda environment nndlbook, with
conda activate nndlbook - We check the python version, with
python --version - Finally, we now run the demo code in test.py, with
python3.9 test.py; use control-c to break out of the run if desired - After the run, we exit the container, with
exit
- At the interactive shell command line inside the container, we
- Now we should be back at the Terminal console, in the DeepLearningPython35 directory
~/DeepLearningPython35 $
~/DeepLearningPython35 $ docker container ls --all
< Should see a container named "deeplearning" >
CONTAINER ID IMAGE COMMAND CREATED ... NAMES
xxxxxxxxxxxx continuumio/miniconda3 "/bin/bash" xxx deeplearning
~/DeepLearningPython35 $ docker container start -ai deeplearning
(base) root@xxx:/# cd deeplearn/
(base) root@xxx:/deeplearn# conda info --env
< Should see two environments: base and nndlbook >
(base) root@xxx:/deeplearn# conda activate nndlbook
(nndlbook) root@xxx:/deeplearn# python --version
Python 3.9.7
(nndlbook) root@xxx:/deeplearn# python3.9 test.py
Epoch 0 : 8943 / 10000
Epoch 1 : 9166 / 10000
Epoch 2 : 9267 / 10000
Epoch 3 : 9340 / 10000
Epoch 4 : 9337 / 10000
Epoch 5 : 9374 / 10000
Epoch 6 : 9386 / 10000
< On my late-2013 MacBook Pro, it takes about a minute to do Epoch 1 to Epoch 6; use control-c to break if desired >
< Each epoch run uses the training images; then neural network is evaluated on test images >
(nndlbook) root@xxx:/deeplearn# exit
exit
~/DeepLearningPython35 $
End of Running IEEE Boston Section class demo code: Introduction to Practical Neural Networks and Deep Learning (Part 1)
These scrips are updated ones from the neuralnetworksanddeeplearning.com gitHub repository in order to work with Python 3.5.2
The testing file (test.py) contains all three networks (network.py, network2.py, network3.py) from the book and it is the starting point to run (i.e. train and evaluate) them.
In test.py there are examples of networks configurations with proper comments. I did that to relate with particular chapters from the book.