This repro implements 3 ipython notebooks.
The first (Memorization) deals with the question of when a DNN simply "memorize data" or when it is able to abstract and generalize the data. Therefore, a bridge to overfitting (overfitting) is built and it is shown how overfitting and "memorization" are related, as well as ways of preventing this i.e. regularization. It shows that N.N. useally memorize the data instead of generalize. It uses the CIFAR10 Dataset and randomly destroys the output. The N.N is able to learn the random mapping.
This Jupyter Notebook Information Theory.ipynb introduces to information theory and explains the Information Bottleneck Theroy by Naftali Tishby. With their help, neural networks can be examined for their information content and conclusions can be drawn about the learning success and the functioning of the individual layers of a network.
To deepen the Information Threory, there is a step-by-step implementation in mutual_calculation.ipynb notebook that uses a tiny DNN to calculate the transinformation from the above information theory. It is kept so small that each step can be calculated manually with a pocket calculator if necessary.
https://arxiv.org/abs/1611.03530
https://arxiv.org/abs/1706.05394
https://arxiv.org/abs/1703.00810
https://www.youtube.com/watch?v=pFWiauHOFpY
https://www.quantamagazine.org/new-theory-cracks-open-the-black-box-of-deep-learning-20170921/
https://www.youtube.com/watch?v=0Ey02HT_1Ho
https://arxiv.org/abs/1611.05397