Suggested optimization levels to build GEOSX and thirdPartyLibs

[drmichaelt7777]

I was wondering if there is any experience with building GEOSX and thirdPartyLibs with higher optimization levels for GCC 10.2.0

So far, -O3 -Ofast, or -O2 -fno-fast-math lead to run-time errors :
FP exceptions at BommerAMG and abort
non-convergence and abort
SIGSEGV
and so on,

Are there any suggestions on the optimization levels that would enable GEOSX run successfully?

thanks

With GCC 10.2.0 and -O3 -Ofast we get with ctest -V :

...
96% tests passed, 7 tests failed out of 164

Total Test time (real) = 919.39 sec

The following tests FAILED:
63 - testTensorOpsEigen (Failed)
64 - testTensorOpsInverseOneArg (Failed)
65 - testTensorOpsInverseTwoArgs (Failed)
68 - testTensorOpsSymInverseTwoArgs (Failed)
102 - testLinearAlgebra_ExternalSolvers (Failed)
135 - testTriaxial_elasticIsotropicPressureDependent (Failed)
163 - testReservoirCompositionalMultiphaseMSWells (Failed)

[klevzoff]

We do not recommend -Ofast and relatives since it disregards IEEE754 compliance and can lead to non-reproducible results across platforms. We do build and test the code with -O3, including including on GCC 10.3. -march=native -mtune=native should also be fine as long as you run the code on the same CPU type you built it on.

Would you mind rebuilding without -Ofast (including thirdPartyLibraries repo) and posting or attaching the full output of failing unit tests, if any? Otherwise, we have to find a way to reproduce the errors. For each of the errors you get (FPE, SIGSEGV, etc.) can you provide:

• (an example of) an input file on which the crash occurs
• a stack trace, if one was printed?

It may also help to post you host-config and the output of ldd geosx to see which other libraries get linked (e.g. BLAS, etc.), if possible.

[drmichaelt7777]

Hi Sergey,

Thanks for the prompt response!

Are there any specific options that need to be enabled in the .cmake files regarding FPE handling? I am not explicitly setting any of these.

I have also set the Warnings are Errors option to false.

So, I have tried several combinations of optimization flags with GCC 10.2.0 building the "Release", including

• -O3 -Ofast
• -O3 -Ofast -fno-fast-math
• -O3 -fno-fast-math
• -O1 -fno-fast-math and
• -fno-fast-math (plain no other options)

BTW, I am builds using IntelMPI OneAPI (2021.06) and OpenMPI HPC-X v2.11. I tried an OpenMP only build but things failed so I did not pursue this further. Incidentally I tried to build with the latest Intel OneAPI C++ compilers (2021.06) but several locations do not compile.

I will collect the ctest -V failed cases output for the above to identify the source of failures

Thanks!
Michael

[drmichaelt7777]

GCC 10.2.0 + OpenMPI HPC-X v2.11 C++ options "-fno-fast-math "

$ ctest -V :

100% tests passed, 0 tests failed out of 164
Total Test time (real) = 276.23 sec

[drmichaelt7777]

GCC 10.2.0 + OpenMPI HPC-X v2.11 C++ options "-O3 -Ofast -fno-fast-math "

ctest -V

96% tests passed, 6 tests failed out of 163

Total Test time (real) = 281.37 sec

The following tests FAILED:
62 - testTensorOpsEigen (Failed)
63 - testTensorOpsInverseOneArg (Failed)
64 - testTensorOpsInverseTwoArgs (Failed)
67 - testTensorOpsSymInverseTwoArgs (Failed)
101 - testLinearAlgebra_ExternalSolvers (Failed)
162 - testReservoirCompositionalMultiphaseMSWells
build-CPU-OPT3-Hypre-GCC_10.2.0-ompi_hpcx-release.zip

[drmichaelt7777]

GCC 10.2.0 + OpenMPI HPC-X v2.11 C++ options "-O1 -fno-fast-math

100% tests passed, 0 tests failed out of 163

Total Test time (real) = 284.70 sec

[drmichaelt7777]

BTW, does GEOSX use heavily BLAS functions? Is there a real need for vendor optimized BLAS?

Does GEOSX compile with latest Intel OneAPI C++ compilers? I am seeing that some source fail to compile.

Thanks!

[drmichaelt7777]

GCC 10.2.0 + OpenMPI HPC-X v2.11 C++ options "-O2 -fno-fast-math"

100% tests passed, 0 tests failed out of 164