Jetson TX2 NX stuck sometimes on boot

[BenediktRadl]

Hi,

i have a Jetson TX2 NX on a Jeston Xavier nx devkit.
I'm building tegrademo-mender distro of the latest version of the Dunfell branch and noticed that the Jeston sometimes stuck on boot. It happens every 10-200 times. Here a log of a failed boot attempt:
bootlog.txt

Did someone encounter the same problem or have an idea of how to fix it? Any help is much appreciated.
Note: it seems that it is related to the mender distro. I haven't seen that issue with 'tegrademo'

[dwalkes]

There was a thread complaining about random rollbacks on TX2 a few months ago here which looks a bit different than what you are mentioning.

I know I've gotten through 100 consecutive upgrades/reboots using https://github.com/mendersoftware/meta-mender-community/tree/kirkstone/meta-mender-tegra/scripts/test in the past so if that script fails consistently you might try bisecting the changes on the dunfell branch to look at where things broke.

Do you have an example log of a successful boot on the same configuration? Are there any clues about where in the boot process it's getting stuck?

[BenediktRadl]

thank you for the answer.

my problem is not related to a mender update. I have done my test on a freshly via usb flashed TX2 NX. For my test, I added a start script that does a "reboot" after 20 seconds. After some cycles, the camera stuck during a boot. If I then do a hardware reset the camera starts again without switching the boot partition.

Here are the log messages of a successful boot process: bootlog_good.txt

I would guess that the problem is somehow related to the EMMC or the partition layout. Especially because it does not occur in the distro without mender (tegrademo).

Do you have an idea how I could debug it to get more information?

[dwalkes]

my problem is not related to a mender update.

OK good to know

For my test, I added a start script that does a "reboot" after 20 seconds.

You can use the scripts in https://github.com/mendersoftware/meta-mender-community/tree/kirkstone/meta-mender-tegra/scripts/test to do a rolling reboot over SSH without needing to modify the image. This allows you to bisect back to older commits and find out which one(s) are/aren't booting successfully.

I would guess that the problem is somehow related to the EMMC or the partition layout. Especially because it does not occur in the distro without mender (tegrademo).

One thing I notice in the difference between success and failed boot is that in the failed case the partition prints come out later. I remember some issues with this and rootwait, and the logic in https://github.com/OE4T/meta-tegra/blob/dfcfe5fd16162d58b328f58945b7b790dc4caa1f/recipes-core/initrdscripts/tegra-minimal-init/platform-preboot-cboot.sh. I also noticed in both traces the boot_slot_suffix on the command line is empty. There's a somewhat relevant discussion at #42 and some references to cboot startup prints regarding init scripts which might help point you in the right direction.

[BenediktRadl]

Sorry for the late reply and thank to for the hint to the test scripts.

I think the empty boot_slot_suffix is normal when using slot A, only slot B has a suffix.

I have now switched to the kirkstone-l4t-r32.7.x branch (that is the one I want to use). I'm currently trying to find an old state where the issue doesn't exist. The oldest version I have tested so far is from 2022-01-07 15:15:08 (d4aeab9), there it failed after 121 cycles. Do you remember roughly at which date you did your tests?

[dwalkes]

Do you remember roughly at which date you did your tests?

I was thinking I'd find it referenced on kirkstone pull requests at https://github.com/mendersoftware/meta-mender-community/pulls?q=is%3Apr+tegra but I'm not seeing it referenced anywhere, unfortunately. The last reference I can find is at #113 (comment) after migrating to tegra-boot-tools-updater which would be https://github.com/OE4T/tegra-demo-distro/tree/b1367b42fd609ac357ea45bf8a5faadc2ad77439 I think.