Random matrix theory and deep networks

Marchěnko-Pastur Law

• MP_Law.m
• Computes the spectrum of data covariance matrix; includes numerical and theoretical results.
• Multi-product cases are included in BN_compare.m .

Batch normalization

• BN_compare.m
• Comparison of multi-layer linear network, with/without batch normalization; related to results from
  • H. Daneshmand, J. Kohler, F. Bach, T. Hofmann, and A. Lucchi, Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks, arXiv, Jun. 11, 2020. Accessed: Nov. 12, 2022.
  • J. Bjorck, C. Gomes, B. Selman, and K. Q. Weinberger, Understanding Batch Normalization, arXiv, Nov. 30, 2018. Accessed: Dec. 26, 2022.

Nonlinear activation function

• Nonlinear.m
• Reproduction of the paper
  • J. Pennington and P. Worah, Nonlinear random matrix theory for deep learning, in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2017.

Reference

[1] Damien Garcia (2023). Simpson's rule for numerical integration, MATLAB Central File Exchange. Retrieved April 11, 2023.

[2] J. Ge, Y.-C. Liang, Z. Bai, and G. Pan, Large-dimensional random matrix theory and its applications in deep learning and wireless communications, Random Matrices: Theory and Applications, vol. 10, p. 2230001, Oct. 2021.

[3] O. Lévêque, Week 12: Marchenko-pastur's theorem: Stieltjes transform method, 2011.

[4] M. Potters and J.-P. Bouchaud, A First Course in Random Matrix Theory: for Physicists, Engineers and Data Scientists, Cambridge University Press, 1 ed., Nov. 2020.

[5] H. Daneshmand, J. Kohler, F. Bach, T. Hofmann, and A. Lucchi, Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks, arXiv, Jun. 11, 2020. Accessed: Nov. 12, 2022.

[6] J. Bjorck, C. Gomes, B. Selman, and K. Q. Weinberger, Understanding Batch Normalization, arXiv, Nov. 30, 2018. Accessed: Dec. 26, 2022.

[7] H. Daneshmand, A. Joudaki, and F. Bach, Batch Normalization Orthogonalizes Representations in Deep Random Networks, June 2021.

[8] S. Ioffe and C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Mar. 2015.

[9] J. Pennington and P. Worah, Nonlinear random matrix theory for deep learning, in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2017.