Symmetry in bubble correction

[utf]

Hi, I have some questions about using FE_BUBBLE=1 (note, I have already enabled it using -D_FE_BUBBLE).

1. Is symmetry used by default when calculating the bubble correction to the SCPH solution?
2. What is the primary way to converge the dense q-point grid? Is it using the &kpoint section or using KMESH_SCPH?

I'm finding the bubble correction is very slow and uses a lot of memory even for a FCC cubic system. For example using a &kpoint mesh of 30 30 30 the code warns it needs 300GB of memory. This seems like a lot of memory for a simple system and much less memory is needed to calculate the lattice thermal conductivity, say in phono3py.

Do you have any other tips for using the bubble correction?

[ttadano]

1. Yes. The current implementation already makes use of symmetry (space group, permutation).
2. Please use the &kpoint section. KMESH_SCPH is used only in the main iterative loop of an SCPH calculation. All post process calculations are performed with the k point information in the &kpoint section.

As you noticed, calculation of the bubble free energy is computationally demanding. It is much more expensive than calculation of the imaginary part of the bubble self-energy, which is necessary for computing thermal conductivity.

Regarding the large RAM requirement, the code first prepares all necessary temperature-dependent information (e.g., phonon frequencies, polarization vectors) on a uniform k-grid (specified in the &kpoint section) before starting a calculation of the bubble free energy. Thus, the more temperature points you have, the larger the RAM requirement.

I have some suggestions that might help.

RAM requirement

This issue may be avoided by reducing the temperature points of an SCPH calculation and k-point density. Please try

1. First, run an SCPH calculation with a relatively dense temperature grid and k points with FE_BUBBLE = 0. This should be feasible.
2. Next, restart the SCPH calculation with FE_BUBBLE = 1 as well as a coarser k-grid and fewer temperature points.
3. If above procedure does not help, please reduce MPI processes per node.

Tips

• It is possible to restart the SCPH calculation with different TMIN, TMAX, and DT as long as the temperature points in the restart run is a subset of that of the initial run.
• In the restart mode (RESTART_SCPH = 1), the code reads SCPH dynamical matrix from the PREFIX.scph_dymat file. For Step 2 above, I would suggest changing PREFIX to avoid overwriting the previous results. To restart the SCPH with a new PREFIX, please create a symbolic link, for example, as

   ln -s PREFIX.scph_dymat PREFIX_new.scph_dymat
  

Convergence

Since the bubble free energy is usually much smaller than the QHA or SCP terms, I think (SCP term with a dense k point) + (Bubble correction with a coarse k point) give a reasonably converged value of the total free energy. For example, using 30 30 30 k mesh for the SCP term (FE_BUBBLE = 0) and 10 10 10 k mesh for the bubble term (FE_BUBBLE = 1) would be hopefully OK. (Of course, convergence check should be done carefully for each case.)

In my experience, the convergence of the bubble free energy tends to be faster than that of the thermal conductivity.

[utf]

Thank you very much for the suggestions. I hadn't realised that the bubble correction was more expensive than calculating the imaginary part of the self-energy. But it is good to hear that convergence is faster for the bubble correction.

The tips on mining the RAM requirement and performing a subset of temperatures are very useful. I am trying them on my system now. I also wonder if the bubble correction is a smooth function with temperature, in that case I can interpolate it to a denser temperature mesh  – I will give this a go.

Thank you for developing this fantastic code. The features are very impressive!

[utf]

Can I also ask, have you benchmarked different parallelisation strategies for the bubble correction? For example, is it best to combine openMP threads with MPI processes. Or just stick to MPI only?

[ttadano]

I haven't done a performance check, but I think a pure MPI run is more efficient in terms of wall time.

[apandit2021]

Hi,

I am also trying to calculate the bubble self-energy (so continuing in this discussion thread). I used the effective second-order harmonic force constants from the SCPH calculations and the third order force constants to evaluate the bubble-self energy (following the previous discussion with Dr. Tadano). But I am not using FE_BUBBLE = 1, or -D_FE_BUBBLE tags as discussed above. To make sure could you please see my input file below and let me know if I need any corrections?

&general
PREFIX = scph_bubble
MODE = RTA
FC2XML = harmonic-2-300K.xml
FCSXML =cubic_Opt.xml
NKD = 2
KD = * *
NONANALYTIC = 2
BORNINFO = *.born
EMIN = -10; EMAX = 400 ; DELTA_E = 0.1
TMIN = 0; TMAX = 700
DT = 50
/

&scph
SELF_OFFDIAG = 0
MAXITER = 8000
MIXALPHA = 0.05
KMESH_INTERPOLATE = 5 5 1
KMESH_SCPH = 10 10 2
TOL_SCPH = 1.0e-8
/

&cell

50.00 # factor in Bohr unit
..........

/

&kpoint
2
80 80 1
/

&analysis
KS_INPUT = mode_TA12.in
SELF_W = 1
#SPS = 1
/

and mode_TA12.in looks as
4
0.0 0.0 0.0 1
0.5 0.0 0.0 1
0.0 0.0 0.0 2
0.5 0.0 0.0 2

Best,
Abhiyan

[ttadano]

This thread is about the bubble free-energy, not the bubble self-energy. Please create a new discussion.

[utf]

By the way @ttadano, I ended up doing a parallelism check on a supercomputer.

I tested the SCPH and Bubble routines separately. I was trying to understand how to balance the number of MPI processes and OMP_NUM_THREADS. The supercomputer has 40 cores per node and can use hyper-threading. I ran test calculations on 80 cores (2 nodes), with and without hyper-threading. I set OMP_NUM_THREADS to the values shown on the plots and set the MPI processes to 80 / OMP_NUM_THREADS for non-hyper-threading and 160 / OMP_NUM_THREADS with hyper-threading. The results show that:

1. The SCPH routines work best with many OpenMP threads. The optimum for my system was OMP_NUM_THREADS=20 with hyperthreading. Note the calculations often crashed with OMP_NUM_THREADS=1 or 2.
2. The Bubble routines work best with small numbers of OpenMP threads (large number of MPI processes). The optimum for my system was OMP_NUM_THREADS=4 with hyperthreading.
3. If your supercomputer supports hyper-threading it can make a big difference to run times.

So it seems like the best performance is gained by running the SCPH part with FE_BUBBLE=0 using large number of threads. And then turning on FE_BUBBLE=1 and restart from the dynmat and use OMP_NUM_THREADS=4 to obtain the thermal properties.

[ttadano]

Thanks for the detailed check of parallelization performance. They are very helpful.

The SCPH routines work best with many OpenMP threads. The optimum for my system was OMP_NUM_THREADS=20 with hyperthreading. Note the calculations often crashed with OMP_NUM_THREADS=1 or 2.

Yes, this is because the main iteration loop of the SCPH is done with a single MPI. Probably, the crashes with OMP_NUM_THREADS=1 or 2 are related to insufficient RAM.
I will think about improving the parallel performance of the SCPH routine.