New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug with mask? #4

Open

dingjiansw101 opened this issue Jul 15, 2024 · 0 comments

dingjiansw101 commented Jul 15, 2024

attentions/attentions.py

Line 286 in b1dd65b

mask = mask.unsqueeze(1).repeat(1, self.num_heads, 1, 1) # BxNxQ_LENxK_LEN

It seems the mask is not correct. Since there is a permute of query, key, and value. The mask should also has a permute.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment