Static Mesh Refinement

Mesh Structure

The mesh structure in Athena++ is completely different from Athena. Athena used a grid structure like Berger & Colella (1989), in which rectangular finer grids can be placed anywhere in the computing domain. Although the terminology about mesh refinement is not well defined, here we call this a patch-based mesh refinement. In Athena++, we take a different approach which we call the oct-tree-block based mesh refinement. In this approach, the computational domain is split into small MeshBlocks, and each MeshBlock can be refined into smaller eight (in 3D; four in 2D, two in 1D) MeshBlocks self-similarly. This approach is less flexible but much simpler, therefore it is easier to implement and achieve a better performance.

In Athena++, all the levels use the same time step (shared time stepping). This greatly simplifies the code and makes parallelization much easier.

Setting Up Static Mesh Refinement

No reconfiguration of the code is required for using Static Mesh Refinement. Simply, write <refinement[n]> block(s) in the input file to specify regions covered by finer resolution.

<refinement1>
x1min=-0.1
x1max= 0.1
x2min=-0.1
x2max= 0.1
x3min=-0.1
x3max= 0.1
level=2

Here, the x?min and x?max set the region covered by this resolution, and level is specify the required resolution. The root level is 0, and Level n means it is 2^n times finer than the root grid. There is no need to specify all the refinement regions. The code automatically calculate the minimal set of MeshBlocks required to cover the requested regions, so that MeshBlocks contact with other MeshBlocks on the same level or one level different. Also, it automatically takes care of periodic boundaries. Therefore, a user only needs to specify the regions where to be refined.

Also do not forget to set the <meshblock> size (see Using MPI and OpenMP), otherwise the meshblock size is the same as the whole root grid. For a better flexibility (i.e., higher adaptivity, more efficient grid generation), smaller MeshBlocks are better but it is trade-off between performance and flexibility. We recommend at least 16^3 cells per MeshBlock.

Checking the Mesh Structure

Because the code automatically generates MeshBlocks, it is difficult for users to predict how many MeshBlocks exist, and how many processes are required for efficient parallel simulation. In order to see this information, run the code with the -m [nproc] option (nproc = 1 is fine at first).

> athena -i athinput.example -m 1

Then it shows a list of MeshBlocks. Then, based on the total number of MeshBlocks, you should decide how many processes should be used in order to balance the load as much as possible. By specifying nproc after the -m option, it shows the actual load balance. Note that the total number of MeshBlocks can be a weird number, because Athena++ does not solve the overlapping coarse grids where finer MeshBlocks exist. In other words, when a MeshBlock is refined, it creates 8 fine MeshBlocks and destroys 1 coarse MeshBlock, yielding +7 MeshBlocks. Do not worry about this at all, as it is anyway more efficient than solving the unneccesary overlapping coarse regions. Also note that one process can own more than one MeshBlocks, which is especially important when small MeshBlocks are used.

Note that our approach of oct-tree-block based mesh refinement tends to produce a lot of MeshBlocks. A user always should check the mesh structure before running simulations and see it is reasonable.

Visualizing the Mesh Structure

The -m option produces "meshtest.dat" in the working directory. This file contains the (logical) distribution of MeshBlocks, and can be visualized using gnuplot. You can see the mesh structure like:

gnuplot> splot "meshtest.dat" with lines

In the case of 3D, this simply shows the spatial distribution of MeshBlocks in the logical coordinates. In 2D, the third dimension is the level of refinement.

Data Output

Because mesh refinement produces a lot of MeshBlocks, we strongly recommend the HDF5 format for data output. This format is much more convenient than other serial formats because it contains a hierarchical structure within one file. Also, note that the join_vtk++ tool (in vis/vtk) does not support the data with mesh refinement for now.

Changes from the old Athena

In Athena, the refinement region is called Domain. Each Domain is divided into Grids and one Grid is corresponding to one process. In Athena++, this concept of Domain is discarded, and the computational domain (Mesh) is decomposed into MeshBlocks. The refinement level is now rather a property of each MeshBlock.