Questions about GPU workflow in gem5

[Ronchi1997]

I am a beginner studying the GPU model in gem5. I use the following command to run it:

docker run -u $UID:$GID --volume $(pwd):$(pwd) -w $(pwd) ghcr.io/gem5/gcn-gpu:latest ./gem5/build/VEGA_X86/gem5.opt gem5/configs/example/apu_se.py -n 3 -c gem5-resources/src/gpu/square/bin/square

The -c parameter specifies the bin/square file, which is an executable file and is not directly readable by humans.
I believe the GPU has two types of input: machine instructions and command processor packets. How does a square binary file get transformed into these two parts? And how wavefront(warp) created?

Could anyone help me explain these? Thank you very much.

[mattsinc]

• @v-ramadas , @abmerop for reference

Hi @Ronchi1997, yes -c is specifying the binary.

I'm not sure any of your other questions are needed for using gem5, because they are more related to how GPGPU programming languages compile code. But the HSA/etc. packets that are sent to the command processor (CP)/packet processor (PP) are essentially how the GPGPU programming language encodes the information (e.g., run this kernel, copy this memory, etc.) to the GPU. The CP then takes these packets and performs the appropriate action on the GPU (e.g., running the kernel the packet encodes). The kernel objects specifically point to the binary that HIP (AMD's GPGPU programming language) generates (the "machine instructions"). This part is done through LLVM -- I don't know the exact details of how this works, but again you don't need to know this for running workloads in gem5 unless you want to modify the compiler itself. The wavefront size is fixed in AMD GPUs (currently 64 threads/wavefront in what gem5 supports). You don't control this -- the CP will take the information from your kernel object and convert it into N wavefronts, where N is ceil(number of threads in kernel / 64).

You may also find these slides useful: https://www.gem5.org/assets/files/isca2024-tutorial/05-gpu.pdf (e.g., slides 30-33)

[Ronchi1997]

Thanks a lot for your comment. The slides is really useful!

The kernel objects specifically point to the binary that HIP (AMD's GPGPU programming language) generates (the "machine instructions"). This part is done through LLVM -- I don't know the exact details of how this works, but again you don't need to know this for running workloads in gem5 unless you want to modify the compiler itself.

In my understanding, the compiled kernel instructions should be somewhere in memories (or SQC) and we use pc in wavefront to point to it.
I'm diving into the code in src/gpu-compute dir, such as fetch_unit.cc/hh and fetch_stage.cc/hh, but I haven't found where the wavefront pc is specified. So when and where is the pc changed in wavefront? (what I mean changed here is initialization)

[abmerop]

The details can get complicated so I'll build on the answer from @mattsinc and point to some gem5 locations.

One of the AQL packets described on slide 32 is a dispatch packet. This is defined in the HSA Runtime Programmer's Reference in Section 2.6.5.7. The "kernel_object" field is the address of a "code object." The code object is defined in gem5 in gpu-compute/kernel_code.hh based on the LLVM description in the LLVM AMDGPU User Guide. The GPU Command Processor calculates the start of the kernel code and creates an HSA task with the initial starting PC.

Next some dispatch magic happens which is a whole other topic. Eventually, individual wavefronts are started and the compute unit sets the initial PC to start the wavefront. I suspect this last is what you are directly asking about. The previous paragraph describes where that value comes from.

[mattsinc]

@mjkpolo : do you have a pointer handy to what line in the fetch_unit/fetch_stage code the GPU PC is grabbed?

[mjkpolo]

Looks like this is where the PC is grabbed

[Ronchi1997]

Looks like this is where the PC is grabbed

Hi, @mjkpolo :
I think at that point, the PC had already been set. The function nextFetchAddr() is using the PC to calculate next fetch address. What I actually wanted to know was where the PC is initialized. This has already been answered by @abmerop , but thank you anyway.