CannotWaitForTheseOptimisers

A collection of experimental optimizers implemented according to the Optimisers.jl interface. We intend to use this package as a testing ground for new optimization algorithms, and then possibly get them incorporated into the main Optimisers.jl package. As such, please do not expect much stability from this package.

Installation

pkg> add Optimisers
pkg> add https://github.com/MurrellGroup/CannotWaitForTheseOptimisers.jl

Usage

using CannotWaitForTheseOptimisers, Optimisers

Description

This package currently includes attempts at implementing:

• Muon which performs an orthogonalization step before parameter update, and seems excellent for training transformers.
• Apollo which tracks low rank moments using a random projection, reducing the memory footprint of the optimizer.
• NormGrowthCap which prevents the norm of the parameters from growing too quickly.