The need for dynamic load balancing in global adaptivity

[IshaanDesai]

In global adaptivity, it is very often the case that there exists only a particular region of interest on the macro scale, which leads to only micro simulations in that region to be active. From a performance perspective this is highly inefficient, because it means that some processors solve a large number of micro simulations, while other processors are idle. In a recent study where we scaled the two-scale-heat-conduction case to have 128 micro simulations, we saw the effect of the load imbalance:

A dynamic load balancing technique which would redistribute the micro simulations across processors would aide to increased performance and scalability.

[IshaanDesai]

The dynamic load balancing is envisioned to work in the following steps:

1. Each rank of the Micro Manager accesses the complete macro mesh. Even though the entire mesh is accessed, only a part of the micro simulations are created.
2. Initially the Micro Manager distributes the total number of micro simulations as evenly as possible amongst all the available ranks.
3. When the load balancing is triggered, an allgather is run to collect on each rank the global number of active simulations.
4. The global number of active simulations are divided by the number of ranks to find out the required number of active simulations per rank to have a balanced load.
5. Just like the allgather on the number of active simulation, another allgather is run to get the global IDs of active simulations, and the information of on which rank these active simulations are. The IDs and the rank location is necessary to determine a communication map to redistribute the load.
6. Using the above information, a communication map is created to decide to which rank is each active simulation sent to. The logic for this is already implemented in the existing GlobalAdaptivityCalculator class.
7. If an active simulation is moved to a different rank, all the inactive simulations associated to it on its current rank are also moved to a different rank.
8. When a simulation is moved to a new rank, the old rank writes zero data as results of a micro simulation that it no longer has. Every rank only writes results for the micro simulations that it is currently hosting.