Skip to content

Commit

Permalink
Refactor end-of-run MPI/Routing logic
Browse files Browse the repository at this point in the history
The inter-mixed conditions of MPI and Routing being active were confusing.

Additionally, running routing after calling MPI_Finalize is
inadvisable. Per the MPICH documentation,

> The number of processes running after this routine is called is
> undefined; it is best not to perform much more than a return rc
> after calling MPI_Finalize.
  • Loading branch information
PhilMiller committed Jan 3, 2024
1 parent 86a8443 commit afa7de5
Showing 1 changed file with 21 additions and 14 deletions.
35 changes: 21 additions & 14 deletions src/NGen.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,9 @@ bool is_subdivided_hydrofabric_wanted = false;
std::string PARTITION_PATH = "";
int mpi_rank;
int mpi_num_procs;
#endif
#else // NGEN_MPI_ACTIVE
int mpi_rank = 0;
#endif // NGEN_MPI_ACTIVE

#include <Layer.hpp>
#include <SurfaceLayer.hpp>
Expand Down Expand Up @@ -534,18 +536,24 @@ int main(int argc, char *argv[]) {
}

} //done time
std::cout<<"Finished "<<manager->Simulation_Time_Object->get_total_output_times()<<" timesteps."<<std::endl;

#if NGEN_MPI_ACTIVE
MPI_Barrier(MPI_COMM_WORLD);
#endif

#ifdef NGEN_ROUTING_ACTIVE
if (mpi_rank == 0)
{
std::cout << "Finished " << manager->Simulation_Time_Object->get_total_output_times() << " timesteps." << std::endl;
}

#ifdef NGEN_MPI_ACTIVE
//Syncronization here. MPI barrier. If rank == 0, do routing

#ifdef NGEN_MPI_ACTIVE
MPI_Barrier(MPI_COMM_WORLD);
MPI_Finalize();
if( mpi_rank == 0 )
#endif

#ifdef NGEN_ROUTING_ACTIVE
if (mpi_rank == 0)
{ // Run t-route from single process
#endif //NGEN_MPI_ACTIVE
if(manager->get_using_routing()) {
//Note: Currently, delta_time is set in the t-route yaml configuration file, and the
//number_of_timesteps is determined from the total number of nexus outputs in t-route.
Expand All @@ -557,13 +565,12 @@ int main(int argc, char *argv[]) {

router->route(number_of_timesteps, delta_time);
}
#ifdef NGEN_MPI_ACTIVE
}
#endif //NGEN_MPI_ACTIVE
#else
#ifdef NGEN_MPI_ACTIVE
#endif

#ifdef NGEN_MPI_ACTIVE
MPI_Finalize();
#endif //NGEN_MPI_ACTIVE
#endif // NGEN_ROUTING_ACTIVE
#endif

return 0;
}

0 comments on commit afa7de5

Please sign in to comment.