Add valid_seq_lengths to fusedsdpa - port from 1.18.0 (#509)

Add valid_seq_lengths to fusedsdpa - port from 1.18.0 https://github.com/HabanaAI/vllm-fork/blob/v1.18.0/vllm/attention/backends/habana_attn.py#L209
HabanaAI · Nov 18, 2024 · 0011e75 · 0011e75
1 parent 96467d8
commit 0011e75
Showing 1 changed file with 1 addition and 0 deletions.
diff --git a/vllm/attention/backends/hpu_attn.py b/vllm/attention/backends/hpu_attn.py
@@ -221,6 +221,7 @@ def forward(
                     matmul_qk_op=self.matmul_qk,
                     softmax_op=self.softmax,
                     matmul_av_op=self.matmul_av,
+                    valid_seq_lengths=attn_metadata.seq_lens_tensor,
                 )
             else:
                 # TODO: enable FusedSDPA