Skip to content

Commit

Permalink
README.md: Update new dims.
Browse files Browse the repository at this point in the history
  • Loading branch information
mashu committed Dec 4, 2024
1 parent 6827017 commit bf7f549
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,10 @@ x_with_pos = pe(x)

# Rotary Position Embeddings
rope = RoPE(512, 1024) # head_dim=512, max_length=1024
x = randn(Float32, 512, 2, 100, 32) # (head_dim, heads, seq_len, batch)
x = randn(Float32, 512, 100, 2*32) # (head_dim, seq_len, (nhead*batch_size))
x_with_pos = rope(x)
```
For a complete example of **RoPEMultiHeadAttention** implementation, please visit documentation. To keep dependencies minimal, trainable parameters must be specified.

## Contributing

Expand Down

0 comments on commit bf7f549

Please sign in to comment.