A unifying approach to SGD in two-layers nets

Left: evolution of student's weights overlapping with two neurons teacher, in classical regime.
Right: evolution of the student overlapping distribution with a single neuron teacher, in the mean-field regime.

Structure

• committee_learning/: Python package containing all the code both for simulation and ODEs integration.
• how_to_classic-limit.ipynb: notebook with an example of usage for classic limit.
• how_to_mean-field.ipynb: notebook with an example of usage for mean-field limit.
• computation-database/: folder for previously generated data.

Installation

# Clone the repo (with submodules!)
git clone --recurse-submodules https://github.com/IdePHICS/DimensionlessDynamicsSGD
cd DimensionlessDynamicsSGD/
# Install Python requirements
pip install -r requirements
# Install committee_learning package (it requires g++)
pip install -e committee_learning/

Reference

Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, Bruno Loureiro From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks, 2023 https://arxiv.org/abs/2305.18502.