Skip to content

Commit

Permalink
Fix typo
Browse files Browse the repository at this point in the history
Co-authored-by: Shane A <[email protected]>
  • Loading branch information
Muennighoff and 2015aroras authored Aug 21, 2024
1 parent fdc1021 commit ed82181
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion olmo/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -670,7 +670,7 @@ def build(cls, layer_id: int, config: ModelConfig, cache: BufferCache) -> OLMoBl

class OLMoEBlock(OLMoBlock):
"""
This is a a transformer MoE block where the output is computed as ``MoE(LN(x + Attention(LN(x))))``
This is a transformer MoE block where the output is computed as ``MoE(LN(x + Attention(LN(x))))``
(plus another skip connection).
"""

Expand Down

0 comments on commit ed82181

Please sign in to comment.