Support for transformer / attention

[pommedeterresautee]

Hi, I have seen a now closed pr in this repo to support attention mechanism with cutlass and a discussion on this subject in flash attention repo.

Do you have plan to offer composable templates to cover key parts of the transformer computation graph in cutlass? (In a more performant way than PyTorch for instance)

Or do you think this lib targets lower level computation and it's up to the final user to build this kind of high level layer?

Obviously some pieces already exist like batched and / or grouped gemm, but then, not sure if that's possible to use it combined with gemm b2b fusion to limit global memory bottleneck, or if you can apply easily different epilogues in group gemm to perform several projections in parallel (to limit the nb of kernel to launch), if a residual connection may be modeled in a performant way on cutlass, if that's hard to add layernorm, etc.

kind of related, the presentation Use CUTLASS to Fuse Multiple GEMMs to Extreme Performance looks really^n great (according to its slides at least), but, starting slide 27, the audio would really help and the only one existing is in chinese without any option for a translation. Do you know if some notes or an English version exists? [https://www.nvidia.com/en-us/on-demand/session/gtcspring22-s41606/]

[hwu36]

Do you have plan to offer composable templates to cover key parts of the transformer computation graph in cutlass?

Yes.

Do you know if some notes or an English version exists?

no english. sorry. maybe google translate can help.