2021-07-01-agarwal21a.md

title	abstract	layout	series	publisher	issn	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_author	author	date	address	container-title	volume	genre	issued	pdf	extras
Acceleration via Fractal Learning Rate Schedules	In practical applications of iterative first-order optimization, the learning rate schedule remains notoriously difficult to understand and expensive to tune. We demonstrate the presence of these subtleties even in the innocuous case when the objective is a convex quadratic. We reinterpret an iterative algorithm from the numerical analysis literature as what we call the Chebyshev learning rate schedule for accelerating vanilla gradient descent, and show that the problem of mitigating instability leads to a fractal ordering of step sizes. We provide some experiments to challenge conventional beliefs about stable learning rates in deep learning: the fractal schedule enables training to converge with locally unstable updates which make negative progress on the objective.	inproceedings	Proceedings of Machine Learning Research	PMLR	2640-3498	agarwal21a	0	Acceleration via Fractal Learning Rate Schedules	87	99	87-99	87	false	Agarwal, Naman and Goel, Surbhi and Zhang, Cyril	given family Naman Agarwal given family Surbhi Goel given family Cyril Zhang	2021-07-01		Proceedings of the 38th International Conference on Machine Learning	139	inproceedings	date-parts 2021 7 1	http://proceedings.mlr.press/v139/agarwal21a/agarwal21a.pdf	label link Supplementary PDF http://proceedings.mlr.press/v139/agarwal21a/agarwal21a-supp.pdf