WIP: esp-qcom-image: convert to OE-core uki.bbclass #138

koenkooi · 2025-01-23T13:12:31Z

This new class uses the OE-core uki.bbclass infrastructure to assemble an
ESP image with systemd-boot + UKI. It tries to closely mimic the
behaviour of the WIC UKI plugin, but since WIC doesn't supported full
disk, partition-less images, this needs its own bbclass.

Systemd-boot is installed through package management, the UKI is picked
up from deploy. A systemd-boot config and boot entry config are
generated based on the BSP metadata.

Signed-off-by: Koen Kooi [email protected]
Fixes qualcomm-linux/meta-qcom#707

lumag · 2025-01-23T13:19:09Z

classes/uki-esp-image.bbclass

+	if [ -n "${KERNEL_CMDLINE_EXTRA}" ]; then
+	    UKI_CMDLINE="$UKI_CMDLINE ${KERNEL_CMDLINE_EXTRA}"
+	fi
+	echo "options $UKI_CMDLINE" >> ${IMAGE_ROOTFS}${ESPFOLDER}/loader/entries/boot.conf


Does this override or extend the command line in the UKI?

From a build perspective, it will extend what is passed into UKI_CMDLINE. From a boot perspective, the order seems to be:

The provided cmdline in boot.conf replaces

The provided cmdline inside the UKI replaces

A compiled-in cmdline in the *Image

The systemd-boot documentation is ambiguous, the most definitive discussions seems to be here: 1. A compiled-in cmdline in the *Image

This behavior depends if secure boot is enabled as well.

lumag · 2025-01-23T13:21:15Z

classes/uki-esp-image.bbclass

+		install -m 0644 ${DEPLOY_DIR_IMAGE}/${ESPINITRD} ${IMAGE_ROOTFS}${ESPFOLDER}
+		echo "initrd /${ESPINITRD}" >> ${IMAGE_ROOTFS}${ESPFOLDER}/loader/entries/boot.conf
+	fi
+	echo "devicetree /${KERNEL_DEVICETREE}" >> ${IMAGE_ROOTFS}${ESPFOLDER}/loader/entries/boot.conf


I generally feel that specifying extra DT, initrd or kernel command line defeats the purpose of the UKI in some way. Ideally it should be a single and the only file that is being loaded by the bootloader. For example, loader.conf can not be signed.

I agree, but the aim for this PR is to match what the WIC plugin does, so we have an actual, bootable example to show why this is unsuited for the use cases we want to support. Having said that, this should be the highest quality implementation that matches that plugin!

my 2c would be to drop the initrd and devicetree from the beginning (and maybe the commandline as it is so unobvios). In the end, the major obstacle with WIC is in the image format, not in some other property.

Yeah, I think it is better to just support type 2 entries instead (https://uapi-group.org/specifications/specs/boot_loader_specification/#type-2-efi-unified-kernel-images), with everything included as part of the UKI.

koenkooi · 2025-01-23T13:41:46Z

This PR is at the stage where it boots on the RB3 board with the vision kit attached and the bbclass doesn't make my eyes bleed. It does need a bit of cleanup and much, much better error checking and logging.

ndechesne · 2025-01-23T16:39:50Z

I am confused as to why we need ufi-esp-image class at all. we are now generating an EFI boot loader spec entry which we were not using before. why do we need it now? We do not use WIC since we manage our partitions/GPT separately, and the ESP image/partition (VFAT) is built by esp-qcom-image anyway. I must be missing something.

koenkooi · 2025-01-24T07:44:08Z

I am confused as to why we need ufi-esp-image class at all. we are now generating an EFI boot loader spec entry which we were not using before. why do we need it now? We do not use WIC since we manage our partitions/GPT separately, and the ESP image/partition (VFAT) is built by esp-qcom-image anyway. I must be missing something.

We are currently using a home-grown way of creating UKIs, OE-core recently added a bbclass to do that. We want to see if we can use the OE-core bbclass instead of our recipe, the OE-core bbclass doesn't package the UKI, it only puts it in deploy, which is why we need this PR to get the UKI into our boot partition.

If OE-core gains a way to package the generated UKI, we won't need this anymore.

lumag · 2025-01-24T08:29:55Z

I think it might make sense to squash both commits. It will make code flow more obvious.

ndechesne · 2025-01-24T08:31:54Z

I am confused as to why we need ufi-esp-image class at all. we are now generating an EFI boot loader spec entry which we were not using before. why do we need it now? We do not use WIC since we manage our partitions/GPT separately, and the ESP image/partition (VFAT) is built by esp-qcom-image anyway. I must be missing something.

We are currently using a home-grown way of creating UKIs, OE-core recently added a bbclass to do that. We want to see if we can use the OE-core bbclass instead of our recipe, the OE-core bbclass doesn't package the UKI, it only puts it in deploy, which is why we need this PR to get the UKI into our boot partition.

If OE-core gains a way to package the generated UKI, we won't need this anymore.

I understand our home grown uki method, and the oe core class. But I did not understand why we needed another 'plumbing' class (uki-esp-image). Is it such a bad thing that the UKI is deployed? that seems to make sense. Why do we need to package the UKI?

lumag · 2025-01-24T08:34:51Z

If UKI is packaged, then building the ESP image becomes as easy as selecting packages to install. Fetching UKI from deploy dir is more fun, it is necessary to find the image, etc.

ndechesne · 2025-01-24T08:38:22Z

recipes-kernel/images/esp-qcom-image.bb

@@ -1,14 +1,20 @@
 DESCRIPTION = "EFI System Partition Image to boot Qualcomm boards"

 PACKAGE_INSTALL = " \
-    linux-qcom-uki \


I also noticed that in our current UKI (we build with linux-qcom-uki) we do not set an initrd, while I think there is one by default with the upstream uki class. It's a boot behavior change that we need to be aware of. We might want an initrd, but we should not make this change without an explicit commit/discussion.

Initrd or initramfs? They are subtly different in how the / mountpoint is pivoted or not.

we use none of them right now.

We might want to use initrd with meta-qcom-distro, but here I don't think it is needed / used.

koenkooi · 2025-01-24T09:13:20Z

I am confused as to why we need ufi-esp-image class at all. we are now generating an EFI boot loader spec entry which we were not using before. why do we need it now? We do not use WIC since we manage our partitions/GPT separately, and the ESP image/partition (VFAT) is built by esp-qcom-image anyway. I must be missing something.

We are currently using a home-grown way of creating UKIs, OE-core recently added a bbclass to do that. We want to see if we can use the OE-core bbclass instead of our recipe, the OE-core bbclass doesn't package the UKI, it only puts it in deploy, which is why we need this PR to get the UKI into our boot partition.
If OE-core gains a way to package the generated UKI, we won't need this anymore.

I understand our home grown uki method, and the oe core class. But I did not understand why we needed another 'plumbing' class (uki-esp-image). Is it such a bad thing that the UKI is deployed? that seems to make sense. Why do we need to package the UKI?

The OE-core class puts the UKI in deploy, nothing more, it doesn't package it nor does it put it in staging. Aside from WIC, there is no way to include it in an image without plumbing as we have in this PR.
And we can't use WIC for our ESP images, since WIC forces a partition table in every image it creates.

koenkooi · 2025-01-24T09:15:26Z

I think it might make sense to squash both commits. It will make code flow more obvious.

Agreed, the split is mostly an artefact from the meta-qcom/meta-qcom-hwe split, I'll squash appropriately before removing the WIP status.

koenkooi · 2025-01-24T09:34:54Z

This version of systemd-boot doesn't like type 2:

POST Time      [ 2664] OS Loader
/usr/src/debug/systemd-boot/257.1/src/boot/boot.c:2902@run: No loader found. Configuration files in \loader\entries\*.conf are needed.
ERROR: C40000002:V03051003 I0 5A50AA81-C3AE-4608-A0E3-41A2E69BAF94

$ cat build/buildhistory/images/qcs6490_rb3gen2_core_kit/glibc/esp-qcom-image/files-in-image.txt
drwxr-xr-x root       root             4096 ./EFI
drwxr-xr-x root       root             4096 ./EFI/BOOT
-rwxr-xr-x root       root           120320 ./EFI/BOOT/bootaa64.efi
drwxr-xr-x 4663338    200              4096 ./ESP
drwxr-xr-x 4663338    200              4096 ./ESP/Linux
-rwxr-xr-x 4663338    200          53436416 ./ESP/Linux/linux-qcs6490-rb3gen2-core-kit.efi
drwxr-xr-x 4663338    200              4096 ./loader
drwxr-xr-x 4663338    200              4096 ./loader/entries
-rw-r--r-- 4663338    200                10 ./loader/loader.conf

It doesn't say anything amount missing PE sections, but that is something I'm going to check: whether .cmdline .osrel are present or not.

EDIT: oops, that 'ESP/Linux' is not right!

ndechesne · 2025-01-24T09:41:18Z

drwxr-xr-x 4663338 200 4096 ./loader
drwxr-xr-x 4663338 200 4096 ./loader/entries
-rw-r--r-- 4663338 200 10 ./loader/loader.conf

We do not have these files in the UKI we generate so far. Not sure it makes a difference. just pointing it out.

koenkooi · 2025-01-24T09:43:30Z

After fixing the ESP vs EFI folder typo:

POST Time      [ 2625] OS Loader
    Poky (Yocto Project Reference Distro) 5.1 (styhead)                                     Boot in 5 s

And that fails to boot:

Bad main_symbols in ufdt_overlay_do_fixups
Failed to apply devie tree
HypDtFixupEventNotifyFunc: Dtb apply overlay skipped
ERROR: DisableDisplay disabling,Status=0
SdccDxeFixupEventNotifyFunc: non-removable node not found
Added qcom,platform-parts-info successfully in BoardInfoDxe
Failed to get SKU information. Unsupported

data fault: @ 90CB534 PC at 0x9CEAE700, FAR 0x2D656C747465732D, iss 0x4

recovery attempted for : 1000 us
ESR 0x96000004: ec 0x25, il 0x2000000, iss 0x4
iframe 0x83629670:

ndechesne · 2025-01-24T09:55:43Z

In our current builds, we do not add a DTB in the UKI, we rely on the DTB from the firmware. it looks like you are trying to use an upstream DTB here, and the firmware is failing to do its fixups.

lumag · 2025-01-24T10:03:28Z

So, it seems, there are two issues:

Firmware crash on the non-supported DTB
Upstream DT being non-supported

BTW: is the EFI registering EFI_DT_FIXUP_PROTOCOL or does it perform patching in some other way? It really must be a standard protocol

This class uses the OE-core uki.bbclass infrastructure to assemble an ESP image with systemd-boot + UKI. It tried to closely mimic the behaviour of the WIC UKI plugin, but since WIC doesn't supported full disk, partition-less images, this needs its own bbclass. Systemd-boot is installed through package management, the UKI is picked up from deploy. A systemd-boot config and boot entry config are generated based on the BSP metadata. Signed-off-by: Koen Kooi <[email protected]>

Replace homegrown linux-qcom-uki with uki.bbclass. Signed-off-by: Koen Kooi <[email protected]>

Signed-off-by: Koen Kooi <[email protected]>

koenkooi · 2025-01-24T14:55:17Z

In our current builds, we do not add a DTB in the UKI, we rely on the DTB from the firmware. it looks like you are trying to use an upstream DTB here, and the firmware is failing to do its fixups.

Unsetting KERNEL_DEVICETREE makes the kernel boot again, but it fails to mount the rootfs on UFS. I suspect the cmdline isn't quite right.

ndechesne · 2025-01-24T14:58:28Z

In our current builds, we do not add a DTB in the UKI, we rely on the DTB from the firmware. it looks like you are trying to use an upstream DTB here, and the firmware is failing to do its fixups.

Unsetting KERNEL_DEVICETREE makes the kernel boot again, but it fails to mount the rootfs on UFS. I suspect the cmdline isn't quite right.

I think that's the initramfs which you have in this PR, which we don't. Or did you remove it already?

koenkooi · 2025-01-24T14:59:49Z

The log for the EFI+systemd-boot+linux bit:

UEFI Total : 1142 ms
POST Time      [ 2679] OS Loader
EFI stub: Booting Linux Kernel...
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
Start EBS        [ 2986]
BDS: LogFs sync skipped, Unsupported
App Log Flush : 0 ms
ScmArmV8ExitBootServicesHandler, Status = 0x0.
Gunyah based bootup
Exit EBS        [ 3014] UEFI End
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
[    0.000000] Linux version 6.12.0-yoctodev-standard-00090-gf95a6f9e826a (oe-user@oe-host) (aarch64-poky-linux-gcc (GCC) 14.2.0, GNU ld (GNU Binutils) 2.43.1.20241111) qualcomm-linux/meta-qcom-hwe#1 SMP PREEMPT Tue Dec 17 01:35:43 UTC 2024
[    0.000000] KASLR enabled
[    0.000000] Machine model: Qualcomm Technologies, Inc. Robotics RB3gen2
[    0.000000] efi: EFI v2.7 by Qualcomm Technologies, Inc.
[    0.000000] efi: RTPROP=0x9ed9be18 MEMATTR=0x9c88c018 ESRT=0x9c9b1898 INITRD=0x9c885c98 RNG=0x9c880018 MEMRESERVE=0x9c885b18

And the cmdline it is using:

[    0.000000] Kernel command line: rootwait root=LABEL=root console=ttyMSM0,115200

koenkooi · 2025-01-24T15:12:10Z

In our current builds, we do not add a DTB in the UKI, we rely on the DTB from the firmware. it looks like you are trying to use an upstream DTB here, and the firmware is failing to do its fixups.

Unsetting KERNEL_DEVICETREE makes the kernel boot again, but it fails to mount the rootfs on UFS. I suspect the cmdline isn't quite right.

I think that's the initramfs which you have in this PR, which we don't. Or did you remove it already?

I did not, or rather, cannot, without patching uki.bbclass in OE-core. I'm looking at that right now.

ricardosalveti · 2025-01-24T15:53:12Z

So, it seems, there are two issues:

Firmware crash on the non-supported DTB

Upstream DT being non-supported

This is a bit unexpected because the dtb used is already coming from upstream, but loaded via the dtb partition instead. Should work the same when loaded by firmware or loaded by UEFI.

BTW: is the EFI registering EFI_DT_FIXUP_PROTOCOL or does it perform patching in some other way? It really must be a standard protocol

Yeah, IIRC it is using EFI_DT_FIXUP_PROTOCOL.

ricardosalveti · 2025-01-24T15:55:06Z

And the cmdline it is using:

[    0.000000] Kernel command line: rootwait root=LABEL=root console=ttyMSM0,115200

QCOM_BOOTIMG_ROOTFS ?= "PARTLABEL=rootfs"

What we have ^

Signed-off-by: Koen Kooi <[email protected]>

koenkooi · 2025-01-27T12:57:58Z

And the cmdline it is using:
[    0.000000] Kernel command line: rootwait root=LABEL=root console=ttyMSM0,115200
QCOM_BOOTIMG_ROOTFS ?= "PARTLABEL=rootfs"

What we have ^

Yeah, setting UKI_CMDLINE="root=${QCOM_BOOTIMG_ROOTFS} rw rootwait" in the image recipe made it boot.

koenkooi · 2025-01-27T13:02:49Z

So, it seems, there are two issues:

Firmware crash on the non-supported DTB

Upstream DT being non-supported

This is a bit unexpected because the dtb used is already coming from upstream, but loaded via the dtb partition instead. Should work the same when loaded by firmware or loaded by UEFI.

So far I've encountered the following situations:

Dtb present in ESP, UKI + dtb loaded by systemd-boot from an <entry>.conf: works
Dtb present in UKI only, type 2 boot: breaks
Dtb missing from both UKI and ESP: works

ricardosalveti · 2025-01-27T19:13:08Z

Dtb present in UKI only, type 2 boot: breaks

Yeah, this is unexpected, needs validation with the 1.3 fw.

ricardosalveti · 2025-01-27T19:13:22Z

Please move this PR to https://github.com/qualcomm-linux/meta-qcom

koenkooi · 2025-01-28T09:18:41Z

Moved to qualcomm-linux/meta-qcom#722

koenkooi self-assigned this Jan 23, 2025

koenkooi requested review from quic-vkraleti, ricardosalveti, sbanerjee-quic and ndechesne as code owners January 23, 2025 13:12

lumag reviewed Jan 23, 2025

View reviewed changes

koenkooi marked this pull request as draft January 23, 2025 15:02

ndechesne reviewed Jan 24, 2025

View reviewed changes

koenkooi added 7 commits January 24, 2025 15:53

esp-qcom-image: convert to use OE-core UKI infrastructure

729f6bb

Replace homegrown linux-qcom-uki with uki.bbclass. Signed-off-by: Koen Kooi <[email protected]>

SQUASH ME: uki-esp-image: convert to type#2

ef37c09

Signed-off-by: Koen Kooi <[email protected]>

SQUASH ME: fix UKI location

f6c3e55

Signed-off-by: Koen Kooi <[email protected]>

SQUASH ME: uki-esp-image: s/ESP/EFI/

18c8360

Signed-off-by: Koen Kooi <[email protected]>

SQUASH ME: remove systemd-boot configs

c1d5827

Signed-off-by: Koen Kooi <[email protected]>

SQUASH ME: esp-qcom-image: unset KERNEL_DEVICETREE

67818f6

Signed-off-by: Koen Kooi <[email protected]>

koenkooi force-pushed the koen/UKI branch from 0838740 to 67818f6 Compare January 24, 2025 14:54

esp-qcom-image: provide custom kernel cmdline

2c1b663

Signed-off-by: Koen Kooi <[email protected]>

koenkooi mentioned this pull request Jan 28, 2025

WIP: esp-qcom-image: convert to OE-core uki.bbclass qualcomm-linux/meta-qcom#722

Open

koenkooi closed this Jan 28, 2025

WIP: esp-qcom-image: convert to OE-core uki.bbclass #138

WIP: esp-qcom-image: convert to OE-core uki.bbclass #138

Conversation

koenkooi commented Jan 23, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koenkooi commented Jan 23, 2025

ndechesne commented Jan 23, 2025

koenkooi commented Jan 24, 2025

lumag commented Jan 24, 2025

ndechesne commented Jan 24, 2025

lumag commented Jan 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koenkooi commented Jan 24, 2025

koenkooi commented Jan 24, 2025

koenkooi commented Jan 24, 2025 • edited Loading

ndechesne commented Jan 24, 2025

koenkooi commented Jan 24, 2025

ndechesne commented Jan 24, 2025

lumag commented Jan 24, 2025

koenkooi commented Jan 24, 2025

ndechesne commented Jan 24, 2025

koenkooi commented Jan 24, 2025

koenkooi commented Jan 24, 2025

ricardosalveti commented Jan 24, 2025

ricardosalveti commented Jan 24, 2025

koenkooi commented Jan 27, 2025

koenkooi commented Jan 27, 2025

ricardosalveti commented Jan 27, 2025

ricardosalveti commented Jan 27, 2025

koenkooi commented Jan 28, 2025

koenkooi commented Jan 24, 2025 •

edited

Loading