Fix out of bounds memory accesses in RAJAPerf suite

[johnbowen42]

Multiple benchmarks including (DEL_DOT_VEC_2D, EDGE3D, VOL3D, PRESSURE) are segfaulting with a variation of

Callback: Queue 0x15553ca00000 aborting with error : HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION: The agent attempted to access memory beyond the largest legal address. code: 0x29

I'm still identifying all the failing benchmarks, triaging, and working on a fix

[ggeorgakoudis]

Could be related to BlockDim, GridDim specialization in the runtime and caching. Try with the latest main which merges #87

[johnbowen42]

This issue is not fixed by #87

[koparasy]

I am suggesting the following steps:

1. Ensure the store cache is deleted
2. Run then bencmark with setting environment variables:

ENV_PROTEUS_USE_STORED_CACHE=0 ENV_PROTEUS_SET_LAUNCH_BOUNDS=0 ENV_PROTEUS_SPECIALIZE_ARGS=0 ENV_PROTEUS_SPECIALIZE_DIMS=0 <cli>

if it doesn't fail start adding optimizations. 1 by 1. We somehow corrupt either the dynamic information or the module itself.

[johnbowen42]

I was able to verify that enabling ENV_PROTEUS_SPECIALIZE_DIMS is causing this

[johnbowen42]

I think when this is fixed it could be nice to add an integration test CI pipeline that does the following

1. Clone our fork of RAJAPerf and init submodules
2. Build in release
3. Run with --features forall, which runs all benchmarks of RAJA::forall

[koparasy]

Can you please verify once more that you are using/linking with the correct proteus version. Can you double check that the cache hash includes block and grid dimensions: https://github.com/Olympus-HPC/proteus/blob/main/lib/JitEngineDevice.hpp#L450 ?

[johnbowen42]

I'm using the latest commit (54dbb1f) as a submodule. I will check the cache hash, but this bug exists with and without cache enabled

[johnbowen42]

Disabling https://github.com/Olympus-HPC/proteus/blob/54dbb1fbc55d99a7bf9c6f6ce96e059b1dac5ed6/lib/JitEngineDevice.hpp#L202 fixes the issue for the ENERGY benchmark, but not all the benchmarks

[koparasy]

I can replicate the issue(s). I think I have a fix for in bugfix/specialize-dims.

I need though to have a clear head to check how we name things. I will do that tomorrow. I will add some tests as well.

[ggeorgakoudis]

I went through bugfix/specialize-dims. It looks like it's a simple misnaming bug. @johnbowen42 Does this branch fix your issues?

[johnbowen42]

This fixes a subset of these issues but I still am seeing many benchmarks fail

[ggeorgakoudis]

This fixes a subset of these issues but I still am seeing many benchmarks fail

Hmm, that means there's another underlying issue. My advice is to run with ENV_PROTEUS_SPECIALIZE_DIMS=0 for now until we find out what's wrong. @johnbowen42 and @koparasy, I'm tagging you both.

[koparasy]

ENV_PROTEUS_SPECIALIZE_DIMS=0 this does not fix all the errors (We had this discussion with @johnbowen42 privately).

[ggeorgakoudis]

Can you fill me in? Here or in slack?

[johnbowen42]

Update:

• All reduction benchmarks are failing
• All forall benchmarks except INT_PREDICT run without error
• Some atomic benchmarks are hanging

[johnbowen42]

see #96, #97, #98