Memory optimisation

[ddundo]

After discussing with @acse-ej321 on the meeting now, I thought it would be good to have all memory-optimisation efforts and ideas gathered here at one place.

---

Memory usage benchmark

• Add a time-dependent goal-based adaptation demo (e.g. Burgers or bubble-shear, which are in the works now). From it, make a benchmark case to track memory usage which would be compared to with every memory optimisation-related PR

Reducing general mesh memory usage

• See Reducing memory usage animate#112
  • It seems like using subprocess avoids these memory issues (potential memory leak?)
• When we do MeshSeq.fixed_point_iteration, iteration $k$ generates $\mathcal{H}^{(k)}$ mesh sequence, but the saved solutions in MeshSeq.solutions are still on $\mathcal{H}^{(k-1)}$ meshes. So we are holding double the amount of meshes in memory, which is costly
  • We could/should add a final solve_forward call to all fixed_point_iterations, so that the final solutions are computed on the final adapted meshes

Hessian-based

• With Yielding in get_solver #182, I think that (a lot of) memory issues are now gone since we are no longer taping

Goal-based

• Generally, increasing the number of subintervals should reduce memory issues since we are clearing the tape more frequently
  • However, this increases the number of meshes which are using a lot of memory on their own. So it's not a good solution
• With Compute error indicators after each subinterval #212, error indicators are now computed at the end of each subinterval
  • This allows us to get rid of forward_old, adjoint, and adjoint_next fields after each subinterval, since they are no longer needed

[ddundo]

Bubble-shear, with memray. So actually most of memory is being used up in the adaptor function, and not so much in the solvers.

[ddundo]

@acse-ej321 based on the above memory profiling and what I mentioned in mesh-adaptation/animate#112 (comment), I think that we should take care of this on the Animate side. It looks to me like a lot of the memory used is not being released as we'd expect it to - until the process actually ends.

Could you try doing something like this in your adaptor function:

def adaptor(mesh_seq, solutions, indicators):
    # checkpoint all solutions and indicators
    with CheckpointFile("chk.h5", "w") as afile:
        for i in range(len(solutions["u"]["forward"])):
            for j in range(len(solutions["u"]["forward"][i])):
                afile.save_function(solutions["u"]["forward"][i][j], name=f"u_{i}_{j}")
                afile.save_function(indicators["u"][i][j], name=f"ind_u_{i}_{j}")
                # etc.

    # call the adapt script
    subprocess.call(['python', 'adapt_script.py'])
    
    # load the adapted meshes
    with CheckpointFile("adapted_meshes.h5", "r") as afile:
        for i in range(len(mesh_seq)):
            mesh_seq[i] = afile.load_mesh(f"adapted_mesh_{i}")

where the adapt_script.py would load the solutions and indicators from chk.h5, compute the metrics and adapt the meshes, and then save the adapted meshes to adapted_meshes.h5.

If that indeed frees up this huge amount of memory that we see being used up, I can code up a solution to this in Animate :)

[ddundo]

And if even this uses too much memory at once, it might be required to do each metric computation in a subprocess, instead of all metric computations at once inside adapt_script.py