prologue fusion in main loop

[ginowu]

Hello
For the 25_ampere_fprop_mainloop_fusion example, I understand the shared memory is used for storing data of the prologue operators(scale & biasadd)' inputs, if we fuse the scale&bias ops into the Conv' epilogue phase, we should can achieve the same goal of Conv+Scale+Bias fusion. So for the fusion in main loop, do we prefer to fuse in prologue instead of epilogue? Is there any merits to do so?

Thanks
Haipeng

[hwu36]

If you can do fusion in the epilogue, do it because it is usually free. Mainloop fusion inevitably hurts the performance. Example 25 is for the case that you cannot do epilogue fusion for whatever reason.

[ginowu]

Yes, if register usage is not increased then it's free. But current epilogue fusion example has one problem: there is long latency loading epilogue inputs from global memory to registers. I am considering if we could load them into shared memory in advance, then the duration of epilogue phase would be much shorter. So came to example 25 for why we did it the prologue instead of epilogue.

If we want to load epilogue inputs in advance to shared memory, do we have to add new Shm write/read iterator for epilogue inputs?

Thanks

[hwu36]

you can preload into the shared memory if you want. writing iterators or not are both okay. you can just write a simple load function.