Enabling Scalasca square tool to enrich profile with trace information

[mathaefele]

Hi,

I am used to use Scalasca with C or F90 codes and I was pretty happy with the functionality of looking at profiles in cube that contain computed metrics like synchronisation time, late sender, early receivers, etc... These metrics are typically computed from a trace analysis.

I managed to get a first profile, to post-process it with square to get the summary.cubex. I could also get a trace I can visualize alone, but I could not enrich the summary.cubex with trace information to get the trace.cubex file. And I could not find any documentation on it. Am I missing something here ?

Thanks for your help.
Mat

[AndreasGocht]

Hey,

Unfortunately, I do not have any experience with Scalasca. You might want to write to [email protected].

Moreover, I want to mention that I can not support Score-P Python anymore, as I changed my affiliation and no longer have access to Vampir. However, @bertwesarg said that they take over.

Best,

Andreas

[bertwesarg]

The Scalasca trace analyzer is capable of analyzing MPI in combination of OpenMP or Pthreads (not sure if the latter combination is valid though). If a Python trace fits into the supported combination is not known to me. As it looks like a Scalsca issue, I kindly direct you to the team at https://www.scalasca.org/.

If you still think this is a Score-P Python binding issue, please describe how you recorded the profile/trace and what you think went wrong.

[mathaefele]

We made a test with a python pure MPI application and the profiling post processing tool from Scalasca works. It manages to convert the profile.cubex file into a summary.cubex file where MPI related communications and synchronisations are separated from computations in the metrics.

However, when enabling tracing, the same tool is able to enrich the profile with metrics like late sender or early receiver that can further split the time spent in MPI between real communication and application imbalance. And that's a key feature when optimising a parallel application. Would be great to have it here ;)

I guess something is missing on the scalasca side. I did not realise this project was so disconnected from the scalasca project. I ping them. Thanks

[bertwesarg]

Scalasca provides two "analysis" tools, a) square which processes only the profiling result from Score-P (.cubex) and b) scout which processes the tracing result from Score-P (.otf2).

[maximilian-tech]

This might be a Score-P issue related to the tracing of MPI_Mprobe + MPI_Mrecv.

[bertwesarg]

This might be a Score-P issue related to the tracing of MPI_Mprobe + MPI_Mrecv.

can you give some more information, why do you think this might be the issue here

[maximilian-tech]

I have no experience with Scalasca, but a semantically broken otf2 trace ( having no Recv to a Send, because Mrecv, which arise often in mpi4py, does not create a correspondig MPI_RECV event) might lead to not being able to derive metrics like "real communication and application imbalance". Althoug Scalasca should probably throw an error(?).