About the attention function

[TitleZ99]

Thanks for this wonderful work. I have a question as the picture shown. When sr_ratio>1,you will do the conv first ,then plus original v and do attention function last. But when sr_ratio=1,you do the attention function first and then plus the v' after the conv. I am wondering why.

_{Sent from PPHub}

[TitleZ99]