Firedrake memory + performance

[tu1620808124]

Hi,

when I use mpirun -np 2 python *.py, it seems mpirun any code,warning will appearyaksa: 1 leaked handle pool objects,for example the *.py is

from firedrake import *
mesh = UnitSquareMesh(64, 64)
V = FunctionSpace(mesh, "CG", 1)
bc = DirichletBC(V, Constant(0), [1, 2])
x,y = mesh.coordinates
f = interpolate(10*exp(-((x - 0.5)**2 \
    + (y - 0.5)**2) / 0.02), V)
g = interpolate(sin(5*x), V)

u = TrialFunction(V)
v = TestFunction(V)
a = inner(grad(u), grad(v))*dx
L = f*v*dx + g*v*ds
u = Function(V)
solve(a == L, u, bc)
outfile = File("u_poissonCG.pvd")
outfile.write(u)

it's seems a problem about my mpi, and how i can deal with it?

[wence-]

I suspect this is related to the recent MPI communicator changes from @JDBetteridge

[JDBetteridge]

@tu1620808124 I have been seeing this issue for a very long time running MPICH on Arch Linux. (Well before I refactored communicators in PyOP2)

The full message is:

[WARNING] yaksa: 2 leaked handle pool objects

I believe it is coming from MPICH itself. I see the same warning running other Python scripts and PETSc executables.
Note that this is a warning and not an error and I have been safely ignoring it.

If the above warning is causing some other issues, specifically with Firedrake, feel free to elaborate below.

[tu1620808124]

When I am dealing with a time-related problem, I use implicit time discretization, the space is the DG method, and the degree of freedom is obviously lower than 100,000. After a certain number of timesteps under multi-core, the memory will be insufficient, but no problem with single-core. I'm not sure if it's related.

[JDBetteridge]

Could you run your code under a memory profiler and share the results? Memory profiler is the suggestion from the Firedrake website.

Check you aren't keeping every timestep in memory, which would cause the memory use to grow continually. If you are still seeing issues try to create a minimal failing example to post here.

I'm still not sure it's related to the leaked handle pool objects.

[tu1620808124]

When I was using multicore, the memory usage was close to the physical limits of my computer. The more cores there are in parallel, the more memory the computation consumes, for example a simpler heat equation .

from firedrake import *
import sys
import os
#define mesh
mesh = RectangleMesh(256, 256,1.0,1.0)
V = FunctionSpace(mesh, "DG", 1)
bc = []
x,y = SpatialCoordinate(mesh)
n = FacetNormal(mesh)

ic  = project(sin(2*pi*x)*sin(2*pi*y),V)
v = TestFunction(V)
u = Function(V)
u_ = Function(V)
u.assign(ic)
u_.assign(ic)
t = 0.0
t_end = 1.0
dt = 0.1
#define the parameters of dg
sigma = 1.0
beta = 3.0
epsilon = 1.0
tc = Constant(t)

B = inner(grad(u), grad(v))*dx
J = -avg(inner(grad(u), n)) *jump(v)*dS + epsilon*avg(inner(grad(v), n)) *jump(u)*dS - inner(grad(u), n)*v*ds + epsilon*inner(grad(v), n)*u*ds
J0 = sigma/FacetArea(mesh)**beta*jump(u)*jump(v)*dS + sigma/FacetArea(mesh)**beta*u*v*ds
a_temp = B + J + J0
a_time = inner((u - u_ )/dt, v)*dx
f = (1+8*pi**2)*sin(2*pi*x)*sin(2*pi*y)*exp(tc)
F = a_temp + a_time - f*v*dx
while abs(t - t_end) > 0.1*dt:
    tc.assign(t + dt)
    solve(F == 0, u, bc)
    u_.assign(u)
    t += dt
outfile = File("sipg_result/fem_"+"exact"+".pvd")
print(t)
u_true = interpolate(sin(2*pi*x)*sin(2*pi*y)*exp(t), V)
outfile.write(u_true, u)
l2 = errornorm(u_true,u,norm_type = "L2")
l1 = errornorm(u_true,u,norm_type = "L1")
print(l2)

I'm not very familiar with memory in terms of parallelism, is this reasonable as the number of cores grows？

[wence-]

For a fixed size problem, we would expect higher total memory usage in parallel than in serial. This grows (weakly) with the number of processes you're running. Effectively the memory usage per process is a fixed cost (C) plus a variable cost that depends, approximately linearly on the size of the mesh (M). When we add more processes for a fixed size mesh, C stays basically the same, and M (per process) decreases approximately by the number of processes. So if a simulation needs C + M on 1 process, it will need 4C + M/4 on 4 processes.

[JDBetteridge]

@tu1620808124 I have taken your example and run it on my workstation and am seeing very similar numbers, so I don't think there is an issue with your installation.

A few points to help you track down your performance issues:

1. memory profiler is a great tool for tracking your memory usage, when you're using it with MPI you can use it as you have done to see total memory us. Or, if you suspect that one rank is consuming more memory than others you can run mprof run --multiprocess python script.py and the plot will show the memory consumption of each rank individually.
2. I'd take the time to read all the information on the optimising performance page. There are quite a few tools there that will help you profile your code, the flamegraph may be particularly useful.
3. You currently call solve repeatedly in a loop, which can lead to poor performance. You can change your code as follows to get better performance and reduce your memory footprint at the same time:

$ diff -u sipg_heat.py sipg_heat2.py 
--- sipg_heat.py        2022-12-12 10:14:27.315673505 +0000
+++ sipg_heat2.py       2022-12-12 10:34:33.005629677 +0000
@@ -5,7 +5,6 @@
 #define mesh
 mesh = RectangleMesh(256, 256,1.0,1.0)
 V = FunctionSpace(mesh, "DG", 1)
-bc = []
 x,y = SpatialCoordinate(mesh)
 n = FacetNormal(mesh)
 
@@ -33,9 +32,12 @@
 f = (1+8*pi**2)*sin(2*pi*x)*sin(2*pi*y)*exp(tc)
 F = a_temp + a_time - f*v*dx
 
+nlvp = NonlinearVariationalProblem(F, u)
+solver = NonlinearVariationalSolver(nlvp)
+
 while abs(t - t_end) > 0.1*dt:
     tc.assign(t + dt)
-    solve(F == 0, u, bc)
+    solver.solve()
     u_.assign(u)
     t += dt

Before:

After:

If you have any more questions I propose we convert this to a github discussion and others can chime in with suggestions.

[tu1620808124]

Thank you very much!

[lindsayad]

It would be very nice to fix these warnings as they do indicate things like missing VecDestroy, MatDestroy, etc. Perhaps the leaks from firedrake are trivial but there could be a leak from user code, say if a user were to reassign a matrix on every PCSetup, that is not. Having the leaks from firedrake itself could obfuscate whether there are issues or not in user code

[lindsayad]

Conceptually related issue at FEniCS/dolfinx#2559

[lindsayad]

It is also possible per FEniCS/dolfinx#2559 that this is truly more of a petsc4py garbage collection issue in which case we should push there to get this fixed up

[lindsayad]

Something that I find interesting is that there only seem to be leaks when running in parallel. Here are some results from -log_view when running a Poisson example in serial

Object Type          Creations   Destructions. Reports information only for process 0.                                    
                                                                                                                          
--- Event Stage 0: Main Stage                                                                                             
                                                                                                                          
           Container     6              6                                                                                 
           Index Set    59             59                                                                                 
   IS L to G Mapping    15             15                                                                                 
             Section    18             18                                                                                 
   Star Forest Graph    27             27                                                                                 
              Vector    15             15                                                                                 
              Matrix    12             12                                                                                 
      Preconditioner     1              1                                                                                 
       Krylov Solver     1              1                                                                                 
     DMKSP interface     1              1                                                                                 
                SNES     1              1                                                                                 
              DMSNES     1              1                                                                                 
    GraphPartitioner     2              2                                                                                 
    Distributed Mesh    10             10                                                                                 
            DM Label    17             17                                                                                 
     Discrete System    12             12                                                                                 
           Weak Form    12             12 

vs. parallel

Object Type          Creations   Destructions. Reports information only for process 0.                                    
                                                                                                                          
--- Event Stage 0: Main Stage                                                                                             
                                                                                                                          
           Container     8              5                                                                                 
           Index Set   170            149                                                                                 
   IS L to G Mapping    18             15                                                                                 
             Section    70             61                                                                                 
   Star Forest Graph    75             62                                                                                 
              Vector    21              5                                                                                 
              Matrix    14             10                                                                                 
      Preconditioner     1              0                                                                                 
       Krylov Solver     1              0                                                                                 
     DMKSP interface     1              0                                                                                 
                SNES     1              0                                                                                 
              DMSNES     1              0                                                                                 
    GraphPartitioner     4              3                                                                                 
    Distributed Mesh    14              6                                                                                 
            DM Label    47             34                                                                                 
     Discrete System    22             14                                                                                 
           Weak Form    22             14          

[lindsayad]

Opened #3629