MiniBatchProfiling

[paulstapor]

Using the idea of gradient flow together with an implicit ODE solver can give approximative profiles. This can in turn be used with mini-batching, maybe when combined with some kind of memory idea as implemented in adaptive SGD optimizers.
Something like this should be tested and implemented soon, as this might make approximative profiles possible for very, very large models with very, very much data...