v0.0.10

Fixed

• Expose bias flag for feedforwards, same default as Timm [#220]
• Update eps value for layernormm, same default as torch [#221]
• PreNorm bugfix, only one input was normalized [#233]

Added

• Add DeepNet (DeepNorm) residual path and init [#227]

v0.0.9

Added

• Compositional Attention [#41]
• Experimental Ragged attention [#189]
• Mixture of Experts [#181]
• BlockSparseTensor [#202]
• nd-tensor support for triton softmax [#210]

Fixed

• bugfix Favor, single feature map [#183]
• sanity check blocksparse settings [#207]
• fixed some pickability [#204]

v0.0.8

Fixed

• Much faster fused dropout
• Fused dropout repeatability

Added

• Embedding weight tying option

v0.0.7

Fixed

• Dropout setting not properly passed in many attentions

v0.0.6

Fixed

• Fix self attention optimization not being triggered, broken residual path [#119]
• Improve speed by not using contiguous Tensors when not needed [#119]

Added

• Attention mask wrapper [#113]
• ViT comparison benchmark [#117]

v0.0.5

fixing the 0.0.4 pip package, next release will be better in that we'll try to expose pre-built binaries

v0.0.4

• Fixing causality not being respected by the scaled dot product attention
• Fixing Favor causal trainability
• Enabling FusedLayerNorm by default if Triton is available
• Fixing Favor with fp16

v0.03

[0.0.3] - 2021-11-01

Fixed

• Nystrom causal attention [#75]

v0.0.2

[0.0.2] - 2021-11-01

Fixed

• More robust blocksparse [#24]

Added

• Rotary embeddings [#32]
• More flexible layernorm [#50]
• More flexible blockfactory config (key deduplication)