Bad linear solver performance when switching oil and water order in SPE10 #2672

paveltomin · 2023-09-08T19:07:48Z

Take https://github.com/GEOS-DEV/MAELSTROM/tree/master/usecases/francois/SPE10/flow
Switch order { oil, water } -> { water, oil }
Run and observe:

Time: 0.00e+00 s, dt: 10000 s, Cycle: 0
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  0
    ( Rflow ) = ( 2.37e+03 ) ;     ( R ) = ( 2.37e+03 ) ; 
{ 3 }
		Linear Solver | NotConverged | Iterations: 200 | Final Rel Res: 0.167039 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.600905 s | Solve Time: 33.2732 s
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  1
    ( Rflow ) = ( 2.34e+03 ) ;     ( R ) = ( 2.34e+03 ) ; 
    Last LinSolve(iter,res) = ( 200, 1.67e-01 ) ; 
{ 3 }
		Linear Solver | NotConverged | Iterations: 200 | Final Rel Res: 0.151071 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.59512 s | Solve Time: 32.0907 s
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  2
    ( Rflow ) = ( 2.32e+03 ) ;     ( R ) = ( 2.32e+03 ) ; 
    Last LinSolve(iter,res) = ( 200, 1.51e-01 ) ; 
{ 3 }
		Linear Solver | NotConverged | Iterations: 200 | Final Rel Res: 0.149277 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.550563 s | Solve Time: 31.2461 s
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  3
    ( Rflow ) = ( 2.30e+03 ) ;     ( R ) = ( 2.30e+03 ) ; 
    Last LinSolve(iter,res) = ( 200, 1.49e-01 ) ; 
{ 3 }
		Linear Solver | NotConverged | Iterations: 200 | Final Rel Res: 0.112426 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.534897 s | Solve Time: 33.8116 s
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  4
    ( Rflow ) = ( 2.28e+03 ) ;     ( R ) = ( 2.28e+03 ) ; 
    Last LinSolve(iter,res) = ( 200, 1.12e-01 ) ; 
{ 3 }
		Linear Solver | NotConverged | Iterations: 200 | Final Rel Res: 0.0946187 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.732886 s | Solve Time: 45.3103 s
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  5
    ( Rflow ) = ( 2.25e+03 ) ;     ( R ) = ( 2.25e+03 ) ; 
    Last LinSolve(iter,res) = ( 200, 9.46e-02 ) ;

Set initial water to pure zero like that:

    <FieldSpecification
      name="initialComposition_oil"
      initialCondition="1"
      setNames="{ all }"
      objectPath="ElementRegions/region/block"
      fieldName="globalCompFraction"
      component="0"
      scale="1.0"/>
    <FieldSpecification
      name="initialComposition_water"
      initialCondition="1"
      setNames="{ all }"
      objectPath="ElementRegions/region/block"
      fieldName="globalCompFraction"
      component="1"
      scale="0.0"/>

Observe crash:

Time: 0.00e+00 s, dt: 10000 s, Cycle: 0
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  0
    ( Rflow ) = ( 2.37e+03 ) ;     ( R ) = ( 2.37e+03 ) ; 
{ 3 }
Received signal 8: Floating point exception
Floating point exception:

** StackTrace of 14 frames **
Frame 0: /lib64/libc.so.6 
Frame 1: hypre_FlexGMRESSolve 
Frame 2: geos::HypreSolver::doSolve(geos::HypreVector const&, geos::HypreVector&) const 
Frame 3: geos::HypreSolver::solve(geos::HypreVector const&, geos::HypreVector&) const 
Frame 4: geos::SolverBase::solveLinearSystem(geos::DofManager const&, geos::HypreMatrix&, geos::HypreVector&, geos::HypreVector&) 
Frame 5: geos::SolverBase::solveNonlinearSystem(double const&, double const&, int, geos::DomainPartition&) 
Frame 6: geos::SolverBase::nonlinearImplicitStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 7: geos::SolverBase::solverStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 8: geos::SolverBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 9: geos::EventBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 10: geos::EventManager::run(geos::DomainPartition&) 
Frame 11: geos::GeosxState::run() 
Frame 12: main 
Frame 13: __libc_start_main 
Frame 14: /data/rpo_ptls/GEOSX/residual_flash/x86_64/RHEL7/GEOS/0.2.0--pt_residual-flash/install-CPU-OPTO2-Hypre-GCC_10.2.0-ompi_hpcx-OMP-relwithdebinfo/bin/geos 
=====

The text was updated successfully, but these errors were encountered:

paveltomin · 2023-09-08T19:08:01Z

CC @oseikuffuor1

castelletto1 · 2023-10-24T21:39:48Z

Let's revise the behavior when the TrueIMPES approach is in place.

paveltomin · 2024-01-08T21:05:48Z

Seems to improve a lot with latest GEOS and Hypre:

GEOS version: 0.2.0 (develop, sha1: 39e75de52)
...
  - hypre version: v2.30.0-8-g61fa167d7 (master)

Time: 0.00e+00 s, dt: 10000 s, Cycle: 0

    Attempt:  0, ConfigurationIter:  0, NewtonIter:  0
        ( Rflow ) = ( 2.37e+03 )        ( R ) = ( 2.37e+03 )
        MGR preconditioner: numComponentsPerField = [3]
        Linear Solver | Success | Iterations: 8 | Final Rel Res: 0.000823991 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.617116 s | Solve Time: 0.493542 s
        compflow: Max pressure change: 13789975.317 Pa (before scaling)
        compflow: Max component density change: 3631.760 kg/m3 (before scaling)
        compflow: Global solution scaling factor = 0.042962838962992016
        compflow: Max phase volume fraction change = 0.1546
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  1
        ( Rflow ) = ( 2.27e+03 )        ( R ) = ( 2.27e+03 )
        Last LinSolve(iter,res) = (   8, 8.24e-04 )
        MGR preconditioner: numComponentsPerField = [3]
        Linear Solver | Success | Iterations: 7 | Final Rel Res: 0.000672764 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.758325 s | Solve Time: 0.663731 s
        compflow: Max pressure change: 13197536.080 Pa (before scaling)
        compflow: Max component density change: 3455.103 kg/m3 (before scaling)
        compflow: Global solution scaling factor = 0.04721011807865644
        compflow: Max phase volume fraction change = 0.1616
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  2
        ( Rflow ) = ( 2.16e+03 )        ( R ) = ( 2.16e+03 )
        Last LinSolve(iter,res) = (   7, 6.73e-04 )
        MGR preconditioner: numComponentsPerField = [3]
        Linear Solver | Success | Iterations: 6 | Final Rel Res: 0.000554765 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.534814 s | Solve Time: 0.465981 s
        compflow: Max pressure change: 12574468.701 Pa (before scaling)
        compflow: Max component density change: 3118.941 kg/m3 (before scaling)
        compflow: Global solution scaling factor = 0.05467350494019756
        compflow: Max phase volume fraction change = 0.1690
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  3
        ( Rflow ) = ( 2.04e+03 )        ( R ) = ( 2.04e+03 )
        Last LinSolve(iter,res) = (   6, 5.55e-04 )
        MGR preconditioner: numComponentsPerField = [3]
        Linear Solver | Success | Iterations: 6 | Final Rel Res: 0.000530665 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.602726 s | Solve Time: 0.380616 s
        compflow: Max pressure change: 11886984.564 Pa (before scaling)
        compflow: Max component density change: 2417.589 kg/m3 (before scaling)
        compflow: Global solution scaling factor = 0.07373856414068213
        compflow: Max phase volume fraction change = 0.1766
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  4
        ( Rflow ) = ( 1.89e+03 )        ( R ) = ( 1.89e+03 )
        Last LinSolve(iter,res) = (   6, 5.31e-04 )
        MGR preconditioner: numComponentsPerField = [3]
        Linear Solver | Success | Iterations: 7 | Final Rel Res: 0.000795164 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.513657 s | Solve Time: 0.565541 s
        compflow: Max pressure change: 11010449.231 Pa (before scaling)
        compflow: Max component density change: 1483.864 kg/m3 (before scaling)
        compflow: Global solution scaling factor = 0.11069438106184616
        compflow: Max phase volume fraction change = 0.1687
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  5
        ( Rflow ) = ( 1.68e+03 )        ( R ) = ( 1.68e+03 )
        Last LinSolve(iter,res) = (   7, 7.95e-04 )
        MGR preconditioner: numComponentsPerField = [3]
        Linear Solver | Success | Iterations: 8 | Final Rel Res: 0.00047289 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.693059 s | Solve Time: 0.59985 s
        compflow: Max pressure change: 9791663.917 Pa (before scaling)
        compflow: Max component density change: 422.047 kg/m3 (before scaling)
        compflow: Global solution scaling factor = 0.42376371188368994
        compflow: Max phase volume fraction change = 0.1152
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  6
        ( Rflow ) = ( 9.65e+02 )        ( R ) = ( 9.65e+02 )
        Last LinSolve(iter,res) = (   8, 4.73e-04 )
        MGR preconditioner: numComponentsPerField = [3]
        Linear Solver | Success | Iterations: 12 | Final Rel Res: 0.000842897 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.595183 s | Solve Time: 0.854187 s
        compflow: Max pressure change: 5642317.626 Pa (before scaling)
        compflow: Max component density change: 392.468 kg/m3 (before scaling)
        compflow: Global solution scaling factor = 0.4101994375643316
        compflow: Max phase volume fraction change = 0.1490
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  7
        ( Rflow ) = ( 5.69e+02 )        ( R ) = ( 5.69e+02 )
        Last LinSolve(iter,res) = (  12, 8.43e-04 )
        MGR preconditioner: numComponentsPerField = [3]
        Linear Solver | Success | Iterations: 15 | Final Rel Res: 0.000681896 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.588453 s | Solve Time: 1.10204 s
        compflow: Max pressure change: 3327841.281 Pa (before scaling)
        compflow: Max component density change: 226.974 kg/m3 (before scaling)
        compflow: Global solution scaling factor = 0.7902769676632144
        compflow: Max phase volume fraction change = 0.2299
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  8
        ( Rflow ) = ( 1.19e+02 )        ( R ) = ( 1.19e+02 )
        Last LinSolve(iter,res) = (  15, 6.82e-04 )
        MGR preconditioner: numComponentsPerField = [3]
        Linear Solver | Success | Iterations: 21 | Final Rel Res: 0.000718918 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.56804 s | Solve Time: 1.55028 s
        compflow: Max pressure change: 697928.157 Pa (before scaling)
        compflow: Max component density change: 121.028 kg/m3 (before scaling)
        compflow: Global solution scaling factor = 1
        compflow: Max phase volume fraction change = 0.1552
    Attempt:  0, ConfigurationIter:  0, NewtonIter:  9
        ( Rflow ) = ( 2.27e+00 )        ( R ) = ( 2.27e+00 )
        Last LinSolve(iter,res) = (  21, 7.19e-04 )
        MGR preconditioner: numComponentsPerField = [3]
        Linear Solver | Success | Iterations: 37 | Final Rel Res: 0.000979006 | Make Restrictor Time: 0 | Compute Auu Time: 0 | SC Filter Time: 0 | Setup Time: 0.592545 s | Solve Time: 2.96233 s
        compflow: Max pressure change: 8885.745 Pa (before scaling)
        compflow: Max component density change: 48.297 kg/m3 (before scaling)
        compflow: Global solution scaling factor = 1
        compflow: Max phase volume fraction change = 0.0479

paveltomin · 2024-01-08T21:08:20Z

Crash when initial water = 0 is still there

Time: 0.00e+00 s, dt: 10000 s, Cycle: 0

    Attempt:  0, ConfigurationIter:  0, NewtonIter:  0
        ( Rflow ) = ( 2.37e+03 )        ( R ) = ( 2.37e+03 )
        MGR preconditioner: numComponentsPerField = [3]
Received signal 8: Floating point exception
Floating point exception:

** StackTrace of 14 frames **
Frame 0: /lib64/libc.so.6 
Frame 1: hypre_FlexGMRESSolve 
Frame 2: geos::HypreSolver::doSolve(geos::HypreVector const&, geos::HypreVector&) const 
Frame 3: geos::HypreSolver::solve(geos::HypreVector const&, geos::HypreVector&) const 
Frame 4: geos::SolverBase::solveLinearSystem(geos::DofManager const&, geos::HypreMatrix&, geos::HypreVector&, geos::HypreVector&) 
Frame 5: geos::SolverBase::solveNonlinearSystem(double const&, double const&, int, geos::DomainPartition&) 
Frame 6: geos::SolverBase::nonlinearImplicitStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 7: geos::SolverBase::solverStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 8: geos::SolverBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 9: geos::EventBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 10: geos::EventManager::run(geos::DomainPartition&) 
Frame 11: geos::GeosxState::run() 
Frame 12: main 
Frame 13: __libc_start_main 
Frame 14: /data/rpo_ptls/GEOSX/residual_flash/GEOS/GEOS/build-CPU-OPTO2-Hypre-GCC_10.2.0-ompi_hpcx-OMP-relwithdebinfo/bin/geosx 
=====

paveltomin · 2024-01-10T21:12:45Z

Crash when initial water = 0 is still there

Time: 0.00e+00 s, dt: 10000 s, Cycle: 0

    Attempt:  0, ConfigurationIter:  0, NewtonIter:  0
        ( Rflow ) = ( 2.37e+03 )        ( R ) = ( 2.37e+03 )
        MGR preconditioner: numComponentsPerField = [3]
Received signal 8: Floating point exception
Floating point exception:

** StackTrace of 14 frames **
Frame 0: /lib64/libc.so.6 
Frame 1: hypre_FlexGMRESSolve 
Frame 2: geos::HypreSolver::doSolve(geos::HypreVector const&, geos::HypreVector&) const 
Frame 3: geos::HypreSolver::solve(geos::HypreVector const&, geos::HypreVector&) const 
Frame 4: geos::SolverBase::solveLinearSystem(geos::DofManager const&, geos::HypreMatrix&, geos::HypreVector&, geos::HypreVector&) 
Frame 5: geos::SolverBase::solveNonlinearSystem(double const&, double const&, int, geos::DomainPartition&) 
Frame 6: geos::SolverBase::nonlinearImplicitStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 7: geos::SolverBase::solverStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 8: geos::SolverBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 9: geos::EventBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 10: geos::EventManager::run(geos::DomainPartition&) 
Frame 11: geos::GeosxState::run() 
Frame 12: main 
Frame 13: __libc_start_main 
Frame 14: /data/rpo_ptls/GEOSX/residual_flash/GEOS/GEOS/build-CPU-OPTO2-Hypre-GCC_10.2.0-ompi_hpcx-OMP-relwithdebinfo/bin/geosx 
=====

The root cause of the crash has to be addressed on GEOS side, while on Hypre side it would be good to get a more informative error message, not it stops with

***** ERROR
***** LOCATION: /data/rpo_ptls/GEOSX/residual_flash/GEOS/GEOS/src/coreComponents/linearAlgebra/interfaces/hypre/HyprePreconditioner.cpp:395
***** Controlling expression (should be false): ierr != 0
***** Rank 4: Error in call to m_precond->setup( m_precond->ptr, precondMat.unwrapped(), nullptr, nullptr )
Expected ierr == 0
  ierr = 1
  0 = 0

which does not give a good idea for GEOS developer about what is wrong.

victorapm · 2024-01-13T04:01:49Z

while on Hypre side it would be good to get a more informative error message

Since hypre returned a non-zero error code here, GEOS should call HYPRE_PrintErrorMessages to get a description of the error coming from hypre.

https://github.com/hypre-space/hypre/blob/57bfb26e268ddf003668c5d0b5938ae258922a83/src/utilities/error.c#L241

paveltomin · 2024-01-13T15:15:19Z

while on Hypre side it would be good to get a more informative error message

Since hypre returned a non-zero error code here, GEOS should call HYPRE_PrintErrorMessages to get a description of the error coming from hypre.

https://github.com/hypre-space/hypre/blob/57bfb26e268ddf003668c5d0b5938ae258922a83/src/utilities/error.c#L241

Thanks, I will try to implement that

victorapm · 2024-01-13T16:35:07Z

Let me know when you set up the branch. I was just looking at this again and realized we will need a few more tweaks to enable error message printing, but I can take care of these additional tweaks in your branch.

paveltomin · 2024-01-15T21:12:35Z

Let me know when you set up the branch. I was just looking at this again and realized we will need a few more tweaks to enable error message printing, but I can take care of these additional tweaks in your branch.

drafted here #2935, please have a look

paveltomin · 2024-01-17T18:45:44Z

Runs fine with #2928 + #2938

paveltomin added type: bug Something isn't working type: new A new issue has been created and requires attention labels Sep 8, 2023

paveltomin assigned francoishamon, castelletto1 and victorapm Sep 8, 2023

TotoGaz removed the type: new A new issue has been created and requires attention label Nov 2, 2023

paveltomin self-assigned this Jan 10, 2024

This was linked to pull requests Jan 17, 2024

Improve linear solver for compositional flow runs #2928

Merged

Print Hypre preconditioner setup error #2935

Merged

Apply min comp dens treatment for initial conditions #2938

Merged

CusiniM closed this as completed in #2938 Jan 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bad linear solver performance when switching oil and water order in SPE10 #2672

Bad linear solver performance when switching oil and water order in SPE10 #2672

paveltomin commented Sep 8, 2023

paveltomin commented Sep 8, 2023

castelletto1 commented Oct 24, 2023

paveltomin commented Jan 8, 2024 •

edited

Loading

paveltomin commented Jan 8, 2024

paveltomin commented Jan 10, 2024

victorapm commented Jan 13, 2024

paveltomin commented Jan 13, 2024

victorapm commented Jan 13, 2024

paveltomin commented Jan 15, 2024

paveltomin commented Jan 17, 2024

Bad linear solver performance when switching oil and water order in SPE10 #2672

Bad linear solver performance when switching oil and water order in SPE10 #2672

Comments

paveltomin commented Sep 8, 2023

paveltomin commented Sep 8, 2023

castelletto1 commented Oct 24, 2023

paveltomin commented Jan 8, 2024 • edited Loading

paveltomin commented Jan 8, 2024

paveltomin commented Jan 10, 2024

victorapm commented Jan 13, 2024

paveltomin commented Jan 13, 2024

victorapm commented Jan 13, 2024

paveltomin commented Jan 15, 2024

paveltomin commented Jan 17, 2024

paveltomin commented Jan 8, 2024 •

edited

Loading