- MP_Law.m
- Computes the spectrum of data covariance matrix; includes numerical and theoretical results.
- Multi-product cases are included in BN_compare.m .
- BN_compare.m
- Comparison of multi-layer linear network, with/without batch normalization; related to results from
- H. Daneshmand, J. Kohler, F. Bach, T. Hofmann, and A. Lucchi, Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks, arXiv, Jun. 11, 2020. Accessed: Nov. 12, 2022.
- J. Bjorck, C. Gomes, B. Selman, and K. Q. Weinberger, Understanding Batch Normalization, arXiv, Nov. 30, 2018. Accessed: Dec. 26, 2022.
- Nonlinear.m
- Reproduction of the paper
- J. Pennington and P. Worah, Nonlinear random matrix theory for deep learning, in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2017.
[1] Damien Garcia (2023). Simpson's rule for numerical integration, MATLAB Central File Exchange. Retrieved April 11, 2023.
[2] J. Ge, Y.-C. Liang, Z. Bai, and G. Pan, Large-dimensional random matrix theory and its applications in deep learning and wireless communications, Random Matrices: Theory and Applications, vol. 10, p. 2230001, Oct. 2021.
[3] O. Lévêque, Week 12: Marchenko-pastur's theorem: Stieltjes transform method, 2011.
[4] M. Potters and J.-P. Bouchaud, A First Course in Random Matrix Theory: for Physicists, Engineers and Data Scientists, Cambridge University Press, 1 ed., Nov. 2020.
[5] H. Daneshmand, J. Kohler, F. Bach, T. Hofmann, and A. Lucchi, Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks, arXiv, Jun. 11, 2020. Accessed: Nov. 12, 2022.
[6] J. Bjorck, C. Gomes, B. Selman, and K. Q. Weinberger, Understanding Batch Normalization, arXiv, Nov. 30, 2018. Accessed: Dec. 26, 2022.
[7] H. Daneshmand, A. Joudaki, and F. Bach, Batch Normalization Orthogonalizes Representations in Deep Random Networks, June 2021.
[8] S. Ioffe and C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Mar. 2015.
[9] J. Pennington and P. Worah, Nonlinear random matrix theory for deep learning, in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2017.