Skip to content

Commit

Permalink
Integrating AIR with the ROCm runtime (#367)
Browse files Browse the repository at this point in the history 
* Removing gitmodules

* Updating the documentation to just mentio the VCK5000 and how to point to the ROCr installation

* Adding ROCr integration into the compiler runtime

* Converting all tests to use ROCr

* clang-format

* Install ROCm runtime in github CI

* Removing building libelf by source
  • Loading branch information
@eddierichter-amd
eddierichter-amd authored Dec 20, 2023
1 parent a6eab83 commit 7a222a8
Show file tree
Hide file tree
Showing 79 changed files with 3,817 additions and 3,182 deletions.
33 changes: 33 additions & 0 deletions .github/workflows/buildAndTest.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,24 @@ jobs:
fetch-depth: 2
submodules: "true"

- name: Install necessary build tools for the ROCm runtime
run: |
sudo apt install libelf-dev elfutils libdwarf-dev
sudo mkdir --parents --mode=0755 /etc/apt/keyrings
wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | \
gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null
for ver in 5.3.3 5.4.3 5.5.1 5.6; do
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/$ver focal main" \
| sudo tee --append /etc/apt/sources.list.d/rocm.list
done
echo -e 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600' \
| sudo tee /etc/apt/preferences.d/rocm-pin-600
sudo apt update
sudo apt install rocm-hip-runtime-dev5.6.0
- name: Install libboost
run: sudo apt-get install -y libboost-all-dev

- name: Install pip packages
run: sudo pip install psutil pybind11 numpy

Expand Down Expand Up @@ -66,12 +84,23 @@ jobs:
run: utils/clone-mlir-aie.sh
shell: bash

- name: Get ROCm-air-platforms
id: clone-rocm-air-platforms
run: utils/clone-rocm-air-platforms.sh
shell: bash

- name: Rebuild and Install libxaie
run: utils/github-clone-build-libxaie.sh

- name: Rebuild and Install mlir-aie
run: utils/github-build-mlir-aie.sh

- name: Build ROCt
run: ROCm-air-platforms/utils/clone-build-roct.sh

- name: Build ROCr
run: ROCm-air-platforms/utils/clone-build-rocr.sh

# Build the repo test target in debug mode to build and test.
- name: Build and test (Assert)
run: |
Expand All @@ -87,6 +116,8 @@ jobs:
-DLLVM_DIR=../llvm/install/lib/cmake/llvm/ \
-DAIE_DIR=`pwd`/../mlir-aie/install/lib/cmake/aie/ \
-DLibXAIE_ROOT=`pwd`/../aienginev2/install \
-Dhsa-runtime64_DIR=`pwd`/../rocm/lib/cmake/hsa-runtime64/ \
-Dhsakmt_DIR=`pwd`/../rocm/lib/cmake/hsakmt/ \
-DAIR_RUNTIME_TARGETS:STRING="x86_64" \
-Dx86_64_TOOLCHAIN_FILE=`pwd`/../cmake/modules/toolchain_x86_64.cmake \
-DLLVM_EXTERNAL_LIT=`pwd`/../llvm/build/bin/llvm-lit \
Expand Down Expand Up @@ -114,6 +145,8 @@ jobs:
-DLLVM_DIR=../llvm/install/lib/cmake/llvm/ \
-DAIE_DIR=`pwd`/../mlir-aie/install/lib/cmake/aie/ \
-DLibXAIE_ROOT=`pwd`/../aienginev2/install \
-Dhsa-runtime64_DIR=`pwd`/../rocm/lib/cmake/hsa-runtime64/ \
-Dhsakmt_DIR=`pwd`/../rocm/lib/cmake/hsakmt/ \
-DAIR_RUNTIME_TARGETS:STRING="x86_64" \
-Dx86_64_TOOLCHAIN_FILE=`pwd`/../cmake/modules/toolchain_x86_64.cmake \
-DLLVM_EXTERNAL_LIT=`pwd`/../llvm/build/bin/llvm-lit \
Expand Down
3 changes: 0 additions & 3 deletions .gitmodules
View file

This file was deleted.

2 changes: 2 additions & 0 deletions CMakeLists.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,8 @@ foreach(target ${AIR_RUNTIME_TARGETS})
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
-DLibXAIE_ROOT=${LibXAIE_ROOT}
-Dhsa-runtime64_DIR=${hsa-runtime64_DIR}
-Dhsakmt_DIR=${hsakmt_DIR}
-DAIE_DIR=${AIE_DIR}
BUILD_ALWAYS true
STEP_TARGETS clean build install test
Expand Down
130 changes: 34 additions & 96 deletions docs/building.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Building AIR Tools and Runtime

## Running on the VCK5000

The first step required for running on the VCK5000 is building the hardware platform and loading it on your device. We provide a platform repository titled [ROCm-air-platforms](https://github.com/Xilinx/ROCm-air-platforms) which contains instructions on how to compile and load the platform, driver, and firmware on your VCK5000. Then, the AIR tools must be installed by following the [Building external projects on X86](#building-external-projects-on-X86) and [Environment setup](#environment-setup) instructions. Then, the [AIR PCIe driver](https://github.com/Xilinx/ROCm-air-platforms/tree/main/driver) from [ROCm-air-platforms](https://github.com/Xilinx/ROCm-air-platforms) must be compiled and installed. Now you are ready to run tests using AIR on the VCK5000! For example, go to [Test 13](test/13_mb_add_one) and run `make` followed by `sudo ./test.elf` to run the application.


## Prerequisites

**mlir-air** is built and tested on Ubuntu 20.04 with the following packages installed:
Expand All @@ -10,6 +15,7 @@
python 3.8.x
ninja 1.10.0
Xilinx Vitis 2021.2
libelf
```

In addition the following packages maybe useful:
Expand All @@ -21,7 +27,7 @@ In addition the following packages maybe useful:
Xilinx aienginev2 library from https://github.com/Xilinx/embeddedsw
```
Vivado and Vitis are used to build the platforms for available Xilinx cards with AIEs (VCK190 and VCK5000). Vitis also contains the AIE single core compiler xchesscc. The aienginev2 library is a driver backend used to configure the AIE array and used in building the AIR runtime. The aietools single core compiler and aienginev2 library are required to build binaries that run on AIE hardware.
Vivado and Vitis are used to build the platforms for available Xilinx cards with AIEs (VCK5000). Vitis also contains the AIE single core compiler xchesscc. The aienginev2 library is a driver backend used to configure the AIE array and used in building the AIR runtime. The aietools single core compiler and aienginev2 library are required to build binaries that run on AIE hardware.

NOTE: using the Vitis recommended settings64.sh script to set up your environement can cause tool conflicts. Setup your environment in the following order for aietools and Vitis:

Expand All @@ -34,7 +40,20 @@ NOTE: that you do not need the above additional Xilinx packages to make use of t
Building mlir-air requires several other open source packages:
- [mlir](https://github.com/llvm/llvm-project/tree/main/mlir)
- [mlir-aie](https://github.com/Xilinx/mlir-aie)
- [Xilinx cmakeModules](https://github.com/Xilinx/cmakeModules)
- [Xilinx cmakeModules](https://github.com/Xilinx/cmakeModules). Note that this is installed as a submodule of mlir-aie.

In order to build and run on PCIe cards, you first have to build and install the aienginev2 library. We chose to install the library in /opt/xaiengine but it is not required for the tools to be installed there. Just ensure that when building mlir-aie and mlir-air, that you point to the directory in which the aienginev2 library was installed.

```
git clone https://github.com/jnider/aie-rt
cd aie-rt
git checkout joel-aie
cd driver/src
make -f Makefile.Linux CFLAGS="-D__AIELINUX__ -D__AIESYSFS__ -D__AIEAMDAIR__"
sudo cp -r ../include /opt/aiengine/
sudo cp libxaiengine.so* /opt/xaiengine/lib/
export LD_LIBRARY_PATH=/opt/xaiengine/lib:${LD_LIBRARY_PATH}
```

## Building external projects on X86

Expand All @@ -46,127 +65,46 @@ First, run utils/setup_python_packages.sh to setup the prerequisite python packa
source utils/setup_python_packages.sh
```

Next, clone and build LLVM, with the ability to target AArch64 as a cross-compiler, and with MLIR enabled. In addition, we make some common build optimizations to use a linker ('lld' or 'gold') other than 'ld' (which tends to be quite slow on large link jobs) and to link against libLLVM.so and libClang so. You may find that other options are also useful. Note that due to changing MLIR APIs, only a particular revision is expected to work.

To clone llvm and cmakeModules, see utils/clone-llvm.sh for the correct commithash. We point LLVM and subsequent tools to a common installation directory.
Next, clone and build LLVM, with MLIR enabled. In addition, we make some common build optimizations to use a linker ('lld' or 'gold') other than 'ld' (which tends to be quite slow on large link jobs) and to link against libLLVM.so and libClang so. You may find that other options are also useful. Note that due to changing MLIR APIs, only a particular revision is expected to work.

```
cd utils
./clone-llvm.sh
./build-llvm-local.sh llvm build ../../install
```
Run the following to clone and build llvm:

Next, clone and build MLIR-AIE with paths to llvm, aienginev2, and cmakeModules repositories. Again, we use a common installation directory. Note that in the following commands, we assume that the aienginev2 library is installed in /opt/xaiengine as directed in the `Building on x86 with runtime for PCIe` section. If the aienginev2 library was installed elsewhere, be sure that the 4th argument to build mlir-aie points to that location.
```
./clone-mlir-aie.sh
./build-mlir-aie-local.sh llvm mlir-aie/cmake/modulesXilinx /opt/xaiengine mlir-aie build ../../install
cd mlir-air
./utils/clone-llvm.sh
./utils/build-llvm-local.sh llvm
```

The MLIR-AIE tools will be able to generate binaries targeting AIEngines.

Finally, build the MLIR-AIR tools for your desired use case:

- Building on x86
- Building on x86 for deployment on a PCIe platform with AIEs
- Building on x86 for deployment on an embedded platform with AIEs
- Building on ARM for deployment on an embedded platform with AIEs

## Building on x86

Use the following command to build the AIR tools to compile on x86:
Next, clone and build MLIR-AIE with paths to llvm, aienginev2, and cmakeModules repositories. Note that in the following commands, we assume that the aienginev2 library is installed in /opt/xaiengine as directed in the `Prerequisites` section. If the aienginev2 library was installed elsewhere, be sure that the 4th argument to build mlir-aie points to that location.

```
./build-mlir-air.sh $SYSROOT /full/path/to/mlir-air/utils/llvm /full/path/to/mlir-air/utils/cmakeModules /full/path/to/mlir-air/utils/mlir-aie build install
./utils/clone-mlir-aie.sh
./utils/build-mlir-aie-local.sh llvm mlir-aie/cmake/modulesXilinx /opt/xaiengine mlir-aie
```

## Building on x86 with runtime for PCIe

In order to build and run on PCIe cards, you first have to build and install the aienginev2 library. We chose to install the library in /opt/xaiengine but it is not required for the tools to be installed there. Just ensure that when building mlir-aie and mlir-air, that you point to the directory in which the aienginev2 library was installed.
Use the following command to build the AIR tools to compile on x86 for PCIe cards (VCK5000). Make sure that ${ROCM\_ROOT} is pointing to your local ROCm install:

```
git clone https://github.com/jnider/aie-rt
cd aie-rt
git checkout joel-aie
cd driver/src
make -f Makefile.Linux CFLAGS="-D__AIELINUX__ -D__AIESYSFS__ -D__AIEAMDAIR__"
sudo cp -r ../include /opt/xaiengine/
sudo cp libxaiengine.so* /opt/xaiengine/lib/
export LD_LIBRARY_PATH=/opt/xaiengine/lib:${LD_LIBRARY_PATH}
./utils/build-mlir-air-pcie.sh llvm/ mlir-aie/cmake/modulesXilinx/ mlir-aie/ /opt/xaiengine ${ROCM_ROOT}/lib/cmake/hsa-runtime64/ ${ROCM_ROOT}/lib/cmake/hsakmt/
```

Use the following command to build the AIR tools to compile on x86 for PCIe cards (VCK5000):
The PCIe AIR runtime requires the use of the [AIR PCIe kernel driver](https://github.com/Xilinx/ROCm-air-platforms/tree/main/driver). The driver directory in the [ROCm-air-platforms](https://github.com/Xilinx/ROCm-air-platforms) repository contains documentation on how to compile and load the AIR PCIe kernel driver. Run the following script to clone the ROCm-air-platforms repository:

```
./utils/build-mlir-air-pcie.sh utils/llvm/ utils/mlir-aie/cmake/modulesXilinx/ utils/mlir-aie/ /opt/xaiengine
./utils/clone-rocm-air-platforms.sh
```

The PCIe AIR runtime requires the use of the [AIR PCIe kernel driver](driver). The driver directory contains documentation on how to compile and load the AIR PCIe kernel driver.

Note that building a sysroot and the cross-compilation instructions are not necessary when running on x86 with the PCIe runtime.

## Environment setup

Set up your environment to use the tools you just built with the following commands:

```
export PATH=/path/to/mlir-air/install/bin:${PATH}
export PYTHONPATH=/path/to/mlir-air/install/python:${PYTHONPATH}
export LD_LIBRARY_PATH=/path/to/mlir-air/install/lib:/opt/xaiengine/lib:${LD_LIBRARY_PATH}
```

Note that if you are running on x86 with the PCIe runtime, the following path should be added to your path rather than `/path/to/mlir-air/install/bin`.
```
export LD_LIBRARY_PATH=/path/to/install/mlir-air/lib:/opt/xaiengine/lib:${LD_LIBRARY_PATH}
export PATH=/path/to/mlir-air/install-pcie/bin:${PATH}
```

## Building hardware platforms

The instructions for building the hardware platform designs are found in the mlir-air/platforms directory:

- [xilinx_vck190_air](platforms/xilinx_vck190_air)
- [xilinx_vck5000_air](platforms/xilinx_vck5000_air)
- [xilinx_vck5000_air_scale_out](platforms/xilinx_vck5000_air_scale_out)

## Building a Sysroot

Since the MLIR-AIE tools are cross-compiling, in order to actually compile code, we need a 'sysroot' directory,
containing an ARM rootfs. This rootfs must match what will be available in the runtime environment.
Note that copying the rootfs is often insufficient, since many root file systems include absolute links.
Absolute symbolic links can be converted to relative symbolic links using [symlinks](https://github.com/brandt/symlinks).

```sh
cd /
sudo symlinks -rc .
```
Following the VCK190 platform build steps will also create a sysroot.

## Compiling Runtime for AArch64 (partial cross-compile)

Use the following command to build the AIR tools to build on x86 and cross-compile host code for ARM:

```
???
```

## Compiling Tools and Runtime for AArch64 (full cross-compile)

Use the following command to cross-compile the AIR tools for ARM:

```
cd utils
./cross-build-llvm.sh /full/path/to/mlir-air/install-aarch64
./cross-build-mlir-aie.sh ../cmake/modules/toolchain_crosscomp_aarch64.cmake $SYSROOT /full/path/to/mlir-air/utils/cmakeModules /full/path/to/mlir-air/utils/llvm /full/path/to/mlir-air/install-aarch64
./cross-build-mlir-air.sh ../cmake/modules/toolchain_crosscomp_aarch64.cmake $SYSROOT /full/path/to/mlir-air/utils/cmakeModules /full/path/to/mlir-air/utils/llvm /full/path/to/mlir-air/utils/mlir-aie /full/path/to/mlir-air/install-aarch64
cd ..
tar -cvf air_tools.tar.gz install-aarch64
```

## Running on the VCK5000

The first step required for running on the VCK5000 is building the hardware platform and loading it on your device. [Here](platforms/xilinx_vck5000_air) we provide documentation on how to do that. Then, the AIR tools must be installed by running the [Building external projects on X86](#building-external-projects-on-X86) followed by the [Building on x86 with runtime for PCIe](#building-on-x86-with-runtime-for-PCIe) and [Environment setup](#environment-setup). Then, the [AIR PCIe driver](driver) must be compiled and installed. Now you are ready to run tests using AIR on the VCK5000! For example, go to [Test 13](test/13_mb_add_one) and run `make` followed by `sudo ./test.elf` to run the application.



-----

<p align="center">Copyright&copy; 2022 Advanced Micro Devices, Inc.</p>
5 changes: 4 additions & 1 deletion examples/airbert/airbert.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,10 @@ def forward(self, x, q, k, v):
jit_module.forward(x.numpy(), q.numpy(), k.numpy(), v.numpy()))
print(qkv)

# Unloading the air backend
del airbackend

if torch.allclose(qkv_ref, qkv):
print("PASS!")
else:
print("failed.")
print("failed.")
5 changes: 3 additions & 2 deletions examples/weather_stencil_air/Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,10 @@ i32_chess:

i32.elf:
aiecc.py aie_$(b).mlir \
-I/opt/xaiengine/include -L/opt/xaiengine/lib -Wl,-R/opt/xaiengine/lib -I$(ACDC_AIE)/runtime_lib/x86_64/test_lib/include -I$(ACDC_AIR)/runtime_lib/airhost/include -L$(ACDC_AIR)/runtime_lib/airhost $(ACDC_AIE)/runtime_lib/x86_64/test_lib/src/test_library.cpp ./test_$(b).cpp -Wl,--whole-archive -lairhost -Wl,--no-whole-archive -lstdc++ -ldl -o test_$(b).elf
../../elfutils/libelf/libelf.a \
-I/opt/xaiengine/include -L/opt/xaiengine/lib -Wl,-R/opt/xaiengine/lib -I$(ACDC_AIE)/runtime_lib/x86_64/test_lib/include -I$(ACDC_AIR)/runtime_lib/airhost/include -L$(ACDC_AIR)/runtime_lib/airhost $(ACDC_AIE)/runtime_lib/x86_64/test_lib/src/test_library.cpp -I${ROCM_ROOT}/include -Wl,-rpath,${ROCM_ROOT}/lib ${ROCM_ROOT}/lib/libhsa-runtime64.so.1.9.0 ./test_$(b).cpp -Wl,--whole-archive -lairhost -Wl,--no-whole-archive -lstdc++ -ldl -o test_$(b).elf

clean:
rm -rf work *mem acdc_project core_* *.elf *.o *.mlir.prj chesswork*
rm -rf work *mem acdc_project core_* *.elf *.mlir.prj chesswork*


Loading

0 comments on commit 7a222a8

Please sign in to comment.