-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bypass APF/SCOREC #68
Merged
+1,620
−1,261
Merged
Changes from all commits
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
15030bc
Add insphere calculator, add sparse alltoallv
davschneller 0ade085
Replace APF for SimModSuite
davschneller 151c921
Allow larger vertex arrays
davschneller cef998a
Make Serial Mesh files work without APF
davschneller 86c0f52
Improve insphere calculation
davschneller be03f73
Add cstddef includes
davschneller a6cd259
Fix boundaries
davschneller c5d9194
Remove APF serial mesh file
davschneller ba84b61
Make clang-format happy
davschneller a159100
Adjust for building without SCOREC
davschneller f06da88
Remove FIDAP support
davschneller 12135d6
More SCOREC removal
davschneller fe74e88
Larger MPI
davschneller 9725ed8
Clang-format
davschneller f5c838f
Remove FIDAP source file
davschneller bd3398c
Update NetCDF reader
davschneller 72678b4
Include fix
davschneller 17e66c7
Add APF to new data converter
davschneller c7367e0
Allow Simmod with and without APF
davschneller 4fd6b82
Add mesh data check
davschneller 4ed6a1a
MPI_Init_thread
davschneller 5613804
Fixing call simmodeler twice
davschneller 66bd465
Fix virtual destructors
davschneller 59b404b
Propagate boundary offset
davschneller 73d3149
Avoid looking for unused libraries
davschneller 7b2f4c7
Merge remote-tracking branch 'origin/master' into davschneller/bypass…
davschneller 5dd700d
Remove group if
davschneller a41c373
Fix typos in the Netcdf reader
davschneller 5e96dda
Implement Large-range MPI for scatter, more boundary conditions
davschneller 81317a2
Add missing header
davschneller 88d9533
Fix position increment
davschneller dd32ce7
Update clang-format pipeline
davschneller 9906523
Add boundary format as attribute
davschneller 5e9758a
Formatting again, fix clang-format version
davschneller 8fb55e3
Remove old CI pipelines
davschneller File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
#ifndef PUMGEN_AUX_DISTRIBUTOR_H_ | ||
#define PUMGEN_AUX_DISTRIBUTOR_H_ | ||
|
||
#include <cstdlib> | ||
|
||
// no namespace for now | ||
|
||
constexpr std::size_t getChunksize(std::size_t total, int rank, int size) { | ||
return (total / size) + (total % size > rank); | ||
} | ||
|
||
#endif // PUMGEN_AUX_DISTRIBUTOR_H_ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,165 @@ | ||
#include "InsphereCalculator.h" | ||
#include "third_party/MPITraits.h" | ||
#include <algorithm> | ||
#include <array> | ||
#include <cmath> | ||
#include <iterator> | ||
#include <mpi.h> | ||
#include <unordered_map> | ||
|
||
#include "MPIConvenience.h" | ||
|
||
std::vector<double> calculateInsphere(const std::vector<std::size_t>& connectivity, | ||
const std::vector<double>& geometry, MPI_Comm comm) { | ||
int commsize; | ||
int commrank; | ||
|
||
MPI_Comm_size(comm, &commsize); | ||
MPI_Comm_rank(comm, &commrank); | ||
|
||
MPI_Datatype vertexType; | ||
|
||
MPI_Type_contiguous(3, MPI_DOUBLE, &vertexType); | ||
MPI_Type_commit(&vertexType); | ||
|
||
std::vector<std::size_t> outrequests(commsize); | ||
std::vector<std::size_t> inrequests(commsize); | ||
|
||
std::size_t localVertices = geometry.size() / 3; | ||
|
||
std::vector<std::size_t> vertexDist(commsize + 1); | ||
|
||
MPI_Allgather(&localVertices, 1, tndm::mpi_type_t<std::size_t>(), vertexDist.data() + 1, 1, | ||
tndm::mpi_type_t<std::size_t>(), comm); | ||
|
||
for (std::size_t i = 0; i < commsize; ++i) { | ||
vertexDist[i + 1] += vertexDist[i]; | ||
} | ||
|
||
std::vector<std::unordered_map<std::size_t, std::size_t>> outidxmap(commsize); | ||
|
||
for (const auto& vertex : connectivity) { | ||
auto itPosition = std::upper_bound(vertexDist.begin(), vertexDist.end(), vertex); | ||
auto position = std::distance(vertexDist.begin(), itPosition) - 1; | ||
auto localVertex = vertex - vertexDist[position]; | ||
|
||
// only transfer each vertex coordinate once, and only if it's not already on the same rank | ||
if (position != commrank && | ||
outidxmap[position].find(localVertex) == outidxmap[position].end()) { | ||
outidxmap[position][localVertex] = outrequests[position]; | ||
++outrequests[position]; | ||
} | ||
} | ||
|
||
MPI_Alltoall(outrequests.data(), 1, tndm::mpi_type_t<std::size_t>(), inrequests.data(), 1, | ||
tndm::mpi_type_t<std::size_t>(), comm); | ||
|
||
std::vector<std::size_t> outdisp(commsize + 1); | ||
std::vector<std::size_t> indisp(commsize + 1); | ||
for (std::size_t i = 1; i < commsize + 1; ++i) { | ||
outdisp[i] = outdisp[i - 1] + outrequests[i - 1]; | ||
indisp[i] = indisp[i - 1] + inrequests[i - 1]; | ||
} | ||
|
||
std::vector<std::size_t> inidx(indisp[commsize]); | ||
|
||
{ | ||
std::vector<std::size_t> outidx(outdisp[commsize]); | ||
|
||
std::vector<std::size_t> counter(commsize); | ||
|
||
for (std::size_t i = 0; i < commsize; ++i) { | ||
for (const auto& [localVertex, j] : outidxmap[i]) { | ||
outidx[j + outdisp[i]] = localVertex; | ||
} | ||
} | ||
|
||
// transfer indices | ||
|
||
sparseAlltoallv(outidx.data(), outrequests.data(), outdisp.data(), | ||
tndm::mpi_type_t<std::size_t>(), inidx.data(), inrequests.data(), indisp.data(), | ||
tndm::mpi_type_t<std::size_t>(), comm); | ||
} | ||
|
||
std::vector<double> outvertices(3 * connectivity.size()); | ||
|
||
{ | ||
std::vector<double> invertices(3 * indisp[commsize]); | ||
|
||
for (std::size_t i = 0; i < inidx.size(); ++i) { | ||
for (int j = 0; j < 3; ++j) { | ||
invertices[3 * i + j] = geometry[3 * inidx[i] + j]; | ||
} | ||
} | ||
|
||
// transfer vertices | ||
|
||
sparseAlltoallv(invertices.data(), inrequests.data(), indisp.data(), vertexType, | ||
outvertices.data(), outrequests.data(), outdisp.data(), vertexType, comm); | ||
} | ||
|
||
std::vector<std::size_t> counter(commsize); | ||
std::vector<double> inspheres(connectivity.size() / 4); | ||
|
||
for (std::size_t i = 0; i < connectivity.size() / 4; ++i) { | ||
std::array<std::array<double, 3>, 4> vertices; | ||
for (int j = 0; j < 4; ++j) { | ||
auto vertex = connectivity[i * 4 + j]; | ||
auto itPosition = std::upper_bound(vertexDist.begin(), vertexDist.end(), vertex); | ||
auto position = std::distance(vertexDist.begin(), itPosition) - 1; | ||
auto localVertex = vertex - vertexDist[position]; | ||
if (position == commrank) { | ||
vertices[j][0] = geometry[localVertex * 3 + 0]; | ||
vertices[j][1] = geometry[localVertex * 3 + 1]; | ||
vertices[j][2] = geometry[localVertex * 3 + 2]; | ||
} else { | ||
auto transferidx = outidxmap[position][localVertex] + outdisp[position]; | ||
vertices[j][0] = outvertices[transferidx * 3 + 0]; | ||
vertices[j][1] = outvertices[transferidx * 3 + 1]; | ||
vertices[j][2] = outvertices[transferidx * 3 + 2]; | ||
} | ||
} | ||
double a11 = vertices[1][0] - vertices[0][0]; | ||
double a12 = vertices[1][1] - vertices[0][1]; | ||
double a13 = vertices[1][2] - vertices[0][2]; | ||
double a21 = vertices[2][0] - vertices[0][0]; | ||
double a22 = vertices[2][1] - vertices[0][1]; | ||
double a23 = vertices[2][2] - vertices[0][2]; | ||
double a31 = vertices[3][0] - vertices[0][0]; | ||
double a32 = vertices[3][1] - vertices[0][1]; | ||
double a33 = vertices[3][2] - vertices[0][2]; | ||
double b11 = vertices[2][0] - vertices[1][0]; | ||
double b12 = vertices[2][1] - vertices[1][1]; | ||
double b13 = vertices[2][2] - vertices[1][2]; | ||
double b21 = vertices[3][0] - vertices[1][0]; | ||
double b22 = vertices[3][1] - vertices[1][1]; | ||
double b23 = vertices[3][2] - vertices[1][2]; | ||
double det = a11 * a22 * a33 + a12 * a23 * a31 + a13 * a21 * a32 - a13 * a22 * a31 - | ||
a12 * a21 * a33 - a11 * a23 * a32; | ||
double gram = std::abs(det); | ||
|
||
double a1a2x1 = a12 * a23 - a13 * a22; | ||
double a1a2x2 = a13 * a21 - a11 * a23; | ||
double a1a2x3 = a11 * a22 - a12 * a21; | ||
double a1a3x1 = a12 * a33 - a13 * a32; | ||
double a1a3x2 = a13 * a31 - a11 * a33; | ||
double a1a3x3 = a11 * a32 - a12 * a31; | ||
double a2a3x1 = a22 * a33 - a23 * a32; | ||
double a2a3x2 = a23 * a31 - a21 * a33; | ||
double a2a3x3 = a21 * a32 - a22 * a31; | ||
double b1b2x1 = b12 * b23 - b13 * b22; | ||
double b1b2x2 = b13 * b21 - b11 * b23; | ||
double b1b2x3 = b11 * b22 - b12 * b21; | ||
|
||
double faces = std::sqrt(a1a2x1 * a1a2x1 + a1a2x2 * a1a2x2 + a1a2x3 * a1a2x3) + | ||
std::sqrt(a1a3x1 * a1a3x1 + a1a3x2 * a1a3x2 + a1a3x3 * a1a3x3) + | ||
std::sqrt(a2a3x1 * a2a3x1 + a2a3x2 * a2a3x2 + a2a3x3 * a2a3x3) + | ||
std::sqrt(b1b2x1 * b1b2x1 + b1b2x2 * b1b2x2 + b1b2x3 * b1b2x3); | ||
|
||
inspheres[i] = gram / faces; | ||
} | ||
|
||
MPI_Type_free(&vertexType); | ||
|
||
return inspheres; | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
#ifndef PUMGEN_AUX_INSPHERE_CALCULATOR_H_ | ||
#define PUMGEN_AUX_INSPHERE_CALCULATOR_H_ | ||
|
||
#include <cstddef> | ||
#include <mpi.h> | ||
#include <vector> | ||
|
||
// assumes a contiguous distribution of all vertices over all processes | ||
|
||
std::vector<double> calculateInsphere(const std::vector<std::size_t>& connectivity, | ||
const std::vector<double>& geometry, MPI_Comm comm); | ||
|
||
#endif // PUMGEN_AUX_INSPHERE_CALCULATOR_H_ |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess you don't use the one from seissol:
https://github.com/SeisSol/SeisSol/blob/dd018b3398258a23ec2a33c74bd7f31b503dcca6/src/Initializer/time_stepping/GlobalTimestep.cpp#L22-L32
because you want to avoid another dependency?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup—that was the main thing. Though it probably wouldn't be a big one anyways... I.e. it's thinkable I'd say.
IIRC I copied the calculation from there.