cmake fails when cross compiling with mbedtls

[kevinduk]

I am trying to cross-compile amazon-kinesis-video-streams-webrtc-sdk-c with mbedTLS but the cmake command fails.

I do not know if I have made a mistake or if it just doesn't work. I am running on Ubuntu 20.04.

I have set CC and CXX for the cross-compiler and want mbedTLS and other dependencies to be built. I have run this command:

cmake -DBUILD_STATIC_LIBS=TRUE -DBUILD_WEBRTC_SAMPLES=ON -DBUILD_LIBSRTP_HOST_PLATFORM=x86_64 -DUSE_OPENSSL=OFF -DUSE_MBEDTLS=ON ..

The mbedTLS libraries are built correctly.

But when cmake is building libwebsockets there is a compile error:

/home/kevind/code/amazon-kinesis-video-streams-webrtc-sdk-c-H8XXX/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/libwebsockets/build/src/project_libwebsockets/include/libwebsockets.h:287:25: fatal error: mbedtls/ssl.h: No such file or directory #include <mbedtls/ssl.h> ^ compilation terminated.

This suggests that the include path is not been passed in the libwebsockets build args.

I also see these messages which shows cmake is picking up mbedtls headers from the Ubuntu installed mbedtls instead of mbedtls built by cmake (which could be a different version).

Compiling with SSL support MBEDTLS include dir: /usr/include MBEDTLS libraries: /home/kevind/code/amazon-kinesis-video-streams-webrtc-sdk-c-H8XXX/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libmbedtls.a;/home/kevind/code/amazon-kinesis-video-streams-webrtc-sdk-c-H8XXX/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libmbedx509.a;/home/kevind/code/amazon-kinesis-video-streams-webrtc-sdk-c-H8XXX/amazon-kinesis-video-streams-webrtc-sdk-c/open-source/lib/libmbedcrypto.a

Does anyone know if my cmake command is incorrect, or should I create an Issue?

[niyatim23]

Hi @kevinduk, we have some cross compilation builds in our CI that work fine. Can you refer to these if it helps

amazon-kinesis-video-streams-webrtc-sdk-c/.github/workflows/ci.yml

Lines 526 to 579 in bfa6667

	arm64-cross-compilation:
	runs-on: ubuntu-20.04
	env:
	CC: aarch64-linux-gnu-gcc
	CXX: aarch64-linux-gnu-g++
	steps:
	- name: Install dependencies
	run: |
	sudo apt clean && sudo apt update
	sudo apt-get -y install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu binutils-aarch64-linux-gnu
	- name: Clone repository
	uses: actions/checkout@v3
	- name: Build Repository
	run: |
	sudo sh -c 'echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6'
	mkdir build && cd build
	cmake .. -DBUILD_OPENSSL=TRUE -DBUILD_OPENSSL_PLATFORM=linux-generic64 -DBUILD_LIBSRTP_HOST_PLATFORM=x86_64-unknown-linux-gnu -DBUILD_LIBSRTP_DESTINATION_PLATFORM=arm-unknown-linux-uclibcgnueabi
	make
	linux-aarch64-cross-compilation:
	runs-on: ubuntu-20.04
	env:
	CC: aarch64-linux-gnu-gcc
	CXX: aarch64-linux-gnu-g++
	steps:
	- name: Install dependencies
	run: |
	sudo apt clean && sudo apt update
	sudo apt-get -y install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu binutils-aarch64-linux-gnu
	- name: Clone repository
	uses: actions/checkout@v3
	- name: Build Repository
	run: |
	sudo sh -c 'echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6'
	mkdir build && cd build
	cmake .. -DBUILD_OPENSSL=TRUE -DBUILD_OPENSSL_PLATFORM=linux-aarch64 -DBUILD_LIBSRTP_HOST_PLATFORM=x86_64-unknown-linux-gnu -DBUILD_LIBSRTP_DESTINATION_PLATFORM=arm-unknown-linux-uclibcgnueabi
	make
	arm32-cross-compilation:
	runs-on: ubuntu-20.04
	env:
	CC: arm-linux-gnueabi-gcc
	CXX: arm-linux-gnueabi-g++
	steps:
	- name: Install dependencies
	run: |
	sudo apt clean && sudo apt update
	sudo apt-get -y install gcc-arm-linux-gnueabi g++-arm-linux-gnueabi binutils-arm-linux-gnueabi
	- name: Clone repository
	uses: actions/checkout@v3
	- name: Build Repository
	run: |
	sudo sh -c 'echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6'
	mkdir build && cd build
	cmake .. -DBUILD_OPENSSL=TRUE -DBUILD_OPENSSL_PLATFORM=linux-generic32 -DBUILD_LIBSRTP_HOST_PLATFORM=x86_64-unknown-linux-gnu -DBUILD_LIBSRTP_DESTINATION_PLATFORM=arm-unknown-linux-uclibcgnueabi
	make

[niyatim23]

Does building libwebsockets with mbedtls standalone work for you?

[kevinduk]

If you mean doing a fresh cloning of the libwebsockets repo and building with mbedtls there then I haven't tried. If you can give me the cmake command line for libwebsockets and mbedtls then I can try it.

[niyatim23]

Yes, I meant that. I haven't tried it myself but you may be able to find the steps specific to your environment in the CI for the libwebsockets repository.

Additionally, our CI does have an example for mbedtls. Can you check if that works for you?

amazon-kinesis-video-streams-webrtc-sdk-c/.github/workflows/ci.yml

Lines 325 to 391 in bfa6667

	mbedtls-ubuntu-gcc-11:
	runs-on: ubuntu-latest
	env:
	AWS_KVS_LOG_LEVEL: 2
	permissions:
	id-token: write
	contents: read
	steps:
	- name: Clone repository
	uses: actions/checkout@v3
	- name: Configure AWS Credentials
	uses: aws-actions/configure-aws-credentials@v2
	with:
	role-to-assume: ${{ secrets.AWS_ROLE_TO_ASSUME }}
	aws-region: ${{ secrets.AWS_REGION }}
	- name: Install deps
	run: |
	sudo apt clean && sudo apt update
	sudo sh -c 'echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6'
	sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
	sudo add-apt-repository 'deb http://archive.ubuntu.com/ubuntu/ jammy main'
	sudo add-apt-repository 'deb http://archive.ubuntu.com/ubuntu/ jammy universe'
	sudo apt-get -q update
	sudo apt-get -y install libcurl4-openssl-dev
	- name: Build repository
	run: |
	mkdir build && cd build
	cmake .. -DBUILD_TEST=TRUE -DUSE_OPENSSL=OFF -DUSE_MBEDTLS=ON
	make
	ulimit -c unlimited -S
	- name: Run tests
	run: |
	cd build
	timeout --signal=SIGABRT 60m ./tst/webrtc_client_test
	mbedtls-ubuntu-clang:
	runs-on: ubuntu-20.04
	env:
	CC: clang
	CXX: clang++
	AWS_KVS_LOG_LEVEL: 2
	permissions:
	id-token: write
	contents: read
	steps:
	- name: Clone repository
	uses: actions/checkout@v3
	- name: Configure AWS Credentials
	uses: aws-actions/configure-aws-credentials@v2
	with:
	role-to-assume: ${{ secrets.AWS_ROLE_TO_ASSUME }}
	aws-region: ${{ secrets.AWS_REGION }}
	- name: Install dependencies
	run: |
	sudo apt clean && sudo apt update
	sudo apt-get -y install clang
	sudo apt-get -y install libcurl4-openssl-dev
	- name: Build repository
	run: |
	sudo sh -c 'echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6'
	mkdir build && cd build
	cmake .. -DBUILD_TEST=TRUE -DUSE_OPENSSL=OFF -DUSE_MBEDTLS=ON
	make
	ulimit -c unlimited -S
	- name: Run tests
	run: |
	cd build
	timeout --signal=SIGABRT 60m ./tst/webrtc_client_test

[kevinduk]

Additionally, our CI does have an example for mbedtls.

Unfortunately, the examples are not cross compiled. It is the combination of cross-compile and mbedtls that is the problem.

[niyatim23]

Understood, can you please share the compiler you're using? Also, are there any other changes except the cross-compilation with mbedtls? I'm going to try to install and build it myself on an Ubuntu instance

[disa6302]

@kevinduk ,

Can you confirm what CC and CXX are being set to?
In addition to this, can you also confirm which SDK version you are using and if there are any changes to the dependencies version you made locally?

[kevinduk]

PR #1923 has added cross compile examples.

To cross compile on x86_64 host for arm32 target using mbedtls static libraries:

sudo apt-get -y install gcc-arm-linux-gnueabi g++-arm-linux-gnueabi binutils-arm-linux-gnueabi
export CC=arm-linux-gnueabi-gcc
export CXX=arm-linux-gnueabi-gc++
cmake .. -DBUILD_STATIC_LIBS=ON -DUSE_OPENSSL=OFF -DUSE_MBEDTLS=ON -DBUILD_LIBSRTP_HOST_PLATFORM=x86_64-unknown-linux-gnu -DBUILD_LIBSRTP_DESTINATION_PLATFORM=arm-unknown-linux-uclibcgnueabi
make

[mehmetcy84]

@kevinduk what change has resolved the issue for you? I am also trying to build the Kinesis C SDK for iOS with mbedtls enabled, and the libwebsocket build throws the same error for me:
libwebsockets.h:287:25: fatal error: mbedtls/ssl.h: No such file or directory #include <mbedtls/ssl.h>

When I check the open-source/lib folder, I see that mbedtls is built fine for iOS device (arm64).