initrd-flash sometimes is not detecting nvme0n1

[isaacporras]

Describe the bug
When trying to flash my Orin NX nvme with the command:

sudo ./doflash.sh --no-flash && sudo $(. boardvars.sh) ./initrd-flash

I get the following log:

Board ID(3767) version(300) sku(0001) revision(M.3)
Starting at 2023-11-13T11:34:57-06:00
Machine:       mycustommachine
Rootfs device: nvme0n1p1
Found Jetson device in recovery mode at USB 3-1
== Step 1: Signing binaries at 2023-11-13T11:34:57-06:00 ==
== Step 2: Boot Jetson via RCM at 2023-11-13T11:35:11-06:00 ==
Found Jetson device in recovery mode at USB 3-1
== Step 3: Sending flash sequence commands at 2023-11-13T11:35:14-06:00 ==
Waiting for USB storage device flashpkg from a4380a09..........[/dev/sdc]
Device size in blocks: 262144
Unmounted /dev/sdc.
== Step 4: Writing partitions on external storage device at 2023-11-13T11:35:52-06:00 ==
Waiting for USB storage device nvme0n1 from a4380a09......[got flashpkg when expecting nvme0n1]
ERR: could not find nvme0n1
== Step 5: Waiting for final status from device at 2023-11-13T11:36:09-06:00 ==
Waiting for USB storage device flashpkg from a4380a09...[/dev/sdc]
Unmounted /dev/sdc.
Final status: FAILED
Successfully finished at 2023-11-13T11:36:09-06:00
Host-side log:              log.initrd-flash.2023-11-13-11.34.57
Device-side logs stored in: device-logs-2023-11-13-11.34.57

This error happens like 9 out of 10 times when trying to flash the board. It sometimes succeed but the usual is it fails.

To flash the SOM, I previously ran:

sudo ./doflash.sh

Here are the logs:
export-nvme0n1.log
bootloader.log
log.initrd-flash.2023-11-13-11.34.57.txt

[madisongh]

The kernel on the Orin is either not detecting the NVME controller/drive at all, or it is timing out waiting for the drive to show up. Are you using an NVIDIA dev kit carrier board, or some custom carrier? What NVME drive are you using, and have you tried swapping in a different one to see if you get different results?

For debugging something like this, connecting to the serial console on the Orin would help in seeing what's going on during the early boot process, too.

[irodriguez-veridas]

@isaacporras Have you managed to solve this? I'm also facing this problem

[SKYLEO98]

Hello, this issue comes from your m2 SSD interface. You have to make sure that your SSD interface is NVME, not SATA. I encountered the same issue when I plugged in a SATA-interface SSD. Then, I swapped with a new NVME SSD, and it worked like a charm.

[irodriguez-veridas]

Hello, this issue comes from your m2 SSD interface. You have to make sure that your SSD interface is NVME, not SATA. I encountered the same issue when I plugged in a SATA-interface SSD. Then, I swapped with a new NVME SSD, and it worked like a charm.

Thank you for the suggestion @SKYLEO98. I'm using a WD PC SN740 SSD and it's stated to be an NVMe device. I think it's different problem in my case.

[SKYLEO98]

Hello, this issue comes from your m2 SSD interface. You have to make sure that your SSD interface is NVME, not SATA. I encountered the same issue when I plugged in a SATA-interface SSD. Then, I swapped with a new NVME SSD, and it worked like a charm.

Thank you for the suggestion @SKYLEO98. I'm using a [WD PC SN740 SSD] (https://www.westerndigital.com/es-es/products/internal-drives/pc-sn740-ssd?sku=SDDPNQD-256G), and it's stated to be an NVMe device. I think it's different problem in my case.

What form factor are you using for WD PC SN740 SSD? 2242 or 2230?

[irodriguez-veridas]

What form factor are you using for WD PC SN740 SSD? 2242 or 2230?

2280. It's connector 10 (M.2 Slot (Key-M, Type 2280) PCIe 3.0 x4) in the image: https://developer.nvidia.com/embedded/learn/jetson-orin-nano-devkit-user-guide/hardware_spec.html

[ichergui]

Hi @isaacporras
Any update about this issue ? or We can close it

[isaacporras]

In my case, this was related to the carrier board that broke unexpectedly. Replacing the carrier board fixed my issue.

[canyonbliss]

I have experienced this issue with a ARK Carrier board with an Orin NX. It happens sporadically. Although this is speculation, a coworker had it happen, and switching computers seemed to resolve the issue.

[isaacporras]

I experienced it again. I power off the whole board, re powered it, it didn't worked. Switch the whole SOM and NVME to another new carrier board, and it worked. Then switched back to the old carrier board, and it started to worked again. It seems like either the NVME or SOM get's into a bad state.

[dwalkes]

See comment here regarding what's probably the next debugging step

For debugging something like this, connecting to the serial console on the Orin would help in seeing what's going on during the early boot process, too.

Comparing a serial boot trace in working/non working cases on the same hardware might help narrow down the issue.