Move to Alpaka Memory + Alpaka 0.9 + CMSSW Alpaka Caching Allocator

[GNiendorf]

This PR moves the memory management for the codebase over to Alpaka. It also removes unused memory functions in a few files, and gets rid of the unused PrintUtil files in SDL (along with a couple of unused print functions). Lastly, by including Alpaka in the main makefile it moves some of the required Alpaka statements from Event.cuh to Constants.cuh. See timing and performance plots below. Note that the timing is slower because the Alpaka caching allocator is not yet in place.

[GNiendorf]

Timing (Ignore it being slower - this is without the alpaka caching allocator!)

Validation Plots - Here
Comparison to Master (pre-DNN) - Here

[GNiendorf]

I fixed a small bug with the last commit where segmentsInGPU was not being deleted when the Event destructor was called at the end of a run. It led to a mismatch in the number of cudaMallocs vs cudaFrees.

[GNiendorf]

Putting this back as a Draft PR since moving over all of the files to Alpaka memory shouldn't take long.

Here's the current timing and validation plots:

Validation Plots - Here
Comparison to Master - Here

[GNiendorf]

Timing (Now on CGPU-1! So not able to compare to previous timing yet)

[GNiendorf]

I got things to work on lnx7188 again. Here are the current timing results (keep in mind the caching allocator is still bugging out so take these timings with a grain of salt for now. Notice how the TC timing goes crazy with the caching allocator because there is a super long cuda stream sync being called for no apparent reason. Also the caching allocator version throws a runtime error during ntuple writing.)

Relevant Master Timing (Only CUDA w/ CUDA Caching Allocator)

This PR, All Alpaka, No Caching Allocator

This PR, All Alpaka, with Alpaka Caching Allocator

Current Alpaka Branch (With Alpaka Kernels but CUDA memory and CUDA caching allocator)

[slava77]

Notice how the TC timing goes crazy with the caching allocator because there is a super long cuda stream sync being called for no apparent reason

do you happen to know when it's called? before or after the TC kernel or related alloc/read/writes

[GNiendorf]

Notice how the TC timing goes crazy with the caching allocator because there is a super long cuda stream sync being called for no apparent reason

do you happen to know when it's called? before or after the TC kernel or related alloc/read/writes

It's hanging the removeDupQuintupletsInGPUBeforeTC kernel. I have to look into it more.

[GNiendorf]

Fixed it. New timing, looks great!:

Another timing run. This is the first time we can see the full effect of the caching allocator.

[GNiendorf]

Ntuple writing runtime error was not fixed by moving to a newer version of the caching allocator unfortunately. Looking into it more, but I may open the PR for review even if I can't find a solution since it only affects ntuple writing with the caching allocator turned on.

[GNiendorf]

Good to go now @VourMa.

Timings below are averages over 10 runs and are shown next to their corresponding standard uncertainties.

Timing - lnx7188

Timing - cgpu1

Validation Plots - Here
Comparison to Relevant Master - Here

[GNiendorf]

Turning off the half precision code allowed the compiler to find two unused variables in one of the smaller kernels.

[GNiendorf]

After speaking with @VourMa over skype we've agreed to merge this PR first into the Alpaka branch, rebase to the current master, and then do a final review of the Alpaka branch before merging it into the master branch.

[VourMa]

First set of comments is there - to be continued...

I have a general question: What are the files under code/alpaka_interface supposed to do? Are they helper functions for the Alpaka memory management? If so, could you add a small description of what's their purpose?

[VourMa]

Please note that this file is obsolete, replaced by setup_ucsd.sh - not sure if this was meant to be fixed in the rebase or it was missed.

Also, has the whole setup been tested in any of the UCSD machines?

Also, I see that the setup_lnx7188.sh has switched to el8. Could the two files be reconciled once again?

[VourMa]

Since part of the Alpaka migration is the cleanup of the cpu code, is this file (and its corresponding implementation) being used anywhere or can they be removed (instead of moved)?

[VourMa]

Sorry, wrong PR, copying to the proper one...