litebird_sim and map-making with 1/f noise covariance

[anand-avinash]

I am opening this discussion to ask for inputs on the issues related to using the 1/f noise covariance in map-making with the output of litebird_sim.

As many of you know, I am working on a map-making framework for litebird: BrahMap. In the map-maker, I have two layers of code:

1. Layer 1: The generic GLS solver. It simply takes a big array of TOD (and pointings) and performs the map-making. This generic GLS solver knows nothing about how litebird_sim distributes and stores the TODs in memory.
2. Layer 2: An interface between litebird_sim and the generic GLS solver. It takes multiple TODs distributed across multiple observations and MPI processes, puts them together in a single TOD array and passes them to the generic GLS solver.

It is the second layer where I am facing the issues with litebird_sim's data distribution scheme. Consider for example a simulation with 2 detectors with following parameters to distribute the TODs in memory:

• 2 MPI processes
• 1 observation per detector
• 2 time blocks
• 1 detector block

The corresponding data distribution can be visualized with the following diagram.

The red blocks represents the TOD from detector $a$ and the blue blocks represent the TOD from detector $b$.

With this distribution, the TOD of single detector is partitioned into 2 blocks in time (indicated by bold letters in the diagram). For a total simulation length from time $t_0$ to $t_2$, each block will contain the following time chunks of the TOD

• block 1: $t_0$ - $t_1$
• block 2: $t_1$ - $t_2$

Note that on rank 0, sim.observations[0].tod will return det_a_block1 + det_b_block1 and on rank 1, it will return det_a_block2 + det_b_block2.

To perform the map-making with GLS, I need to compute the inverse noise covariance. When I do the map-making with TODs from multiple detectors, the way I arrange the TOD blocks in layer 2, will determine the structure of noise covariance that will be used in layer 1.

Let's consider the two cases for arranging the TODs:

Case 1:
I arrange the TOD blocks detector-wise (not necessarily on single MPI process), like in the diagram below and supply it to the layer 1.

The noise covariance will appear as following, where the bold letters stand for a block matrix:

$$ N = \left( \begin{array}{cc|cc} \mathbf{N}^a_{1,1} & \mathbf{N}^a_{1,2} & \mathbf{0} & \mathbf{0} \\ \mathbf{N}^a_{2,1} & \mathbf{N}^a_{2,2} & \mathbf{0} & \mathbf{0} \\ \hline \mathbf{0} & \mathbf{0} & \mathbf{N}^b_{1,1} & \mathbf{N}^b_{1,2} \\ \mathbf{0} & \mathbf{0} & \mathbf{N}^b_{2,1} & \mathbf{N}^b_{2,2} \\ \end{array} \right) $$

Case 2:
I arrange the TOD blocks in the order of MPI ranks (not necessarily on single MPI process), like in the diagram below and supply it to the layer 1:

The new noise covariance matrix will look as following, where the matrix blocks are same as defined in the previous equation

$$ \mathcal{N} = \left( \begin{array}{cc|cc} \mathbf{N}^a_{1,1} & \mathbf{0} & \mathbf{N}^a_{1,2} & \mathbf{0} \\ \mathbf{0} & \mathbf{N}^b_{1,1} & \mathbf{0} & \mathbf{N}^b_{1,2} \\ \hline \mathbf{N}^a_{2,1} & \mathbf{0} & \mathbf{N}^a_{2,2} & \mathbf{0} \\ \mathbf{0} & \mathbf{N}^b_{2,1} & \mathbf{0} & \mathbf{N}^b_{2,2} \\ \end{array} \right) $$

Now lets compare the pros and cons of the approaches in the two cases.

Case 1:
pros: The noise covariance is a 2x2 block diagonal matrix. We can invert it simply by inverting the 2x2 blocks in place.
cons: Rearranging the TOD blocks as in case 1 is a difficult task, as it will involve multiple MPI communications. The complication will increase if we increase the number of observations per detector. It is difficult to implement as well.

Case 2:
pros: This arrangement is already in place since sim.observation[0].tod on the two ranks 1 and 2 returns the blocks 1+1 and 2+2 respectively.
cons: The noise covariance matrix cannot be inverted easily. It will be even more complicated when we increase the number of detectors and TOD blocks.

If the noise covariance for each detector is a diagonal matrix, we can rearrange the chunks of tods in any order and it will still work well with the map-maker. That is what we take advantage of in the litebird_sim binner, where we use the TOD arrangement of case 2.

So, based on the pros and cons of the two cases, I am posing the question: What could be the optimal way (computationaly and implementation-wise) to interface litebird_sim with an optimal map-maker, given the data distribution scheme of litebird_sim?

Couple of thoughts:

1. We try to come up with a method to invert the new noise covariance $\mathcal{N}$ (I am testing a solution)

2. Re-arrange the TOD matrices across multiple MPI processes and observations to match the layout of case 1. If it is done in litebird_sim, it will allow future optimal map-makers to seamlessly integrate with litebird_sim. But it will be complicated and computationally expensive.

3. Save the per-detector TODs to the disk, allowing the map-maker to stack them together (similar to litebird_sim $\rightarrow$ MADAM interface). This solution is also an expensive one and there will be no seamless integration with an optimal map-maker.