NVMe timeout when booting from eMMC but not SD

[satwell]

I just got an NVMe adapter for my Pinebook Pro and have installed it with a Kingston A2000 250GB NVMe SSD.  This is an SSD that others have reported success with on the wiki.

The boot process stalls for 60s and then I get a timeout error like this:

Code:

[    3.742566] nvme nvme0: pci function 0000:01:00.0
[    3.742612] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[   64.480323] nvme nvme0: I/O 12 QID 0 timeout, disable controller
[   64.600233] nvme nvme0: Device shutdown incomplete; abort shutdown
[   64.640667] nvme nvme0: Identify Controller failed (881)
[   64.641472] nvme nvme0: Removing after probe failure status: -5

And then it boots normally, but none of the /dev/nvme0* devices are available.  This is on Debian Bullseye installed with Daniel Thompson's installer and kernel.  Kernel 5.5 and 5.7 both have this same behavior.

Here's the weird part.  I only have this problem booting from eMMC.  If I boot from an SD card with basically the same Bullseye install and the same kernel, the NVMe drive works perfectly.  So it doesn't seem to be bad hardware or a simple compatibility issue.

I'm a little stumped.

[satwell]

Looks like this may be some sort of hardware compatibility issue after all.  I removed the Kingston A2000 SSD and replaced it with a 250GB Western Digital Blue SN550, and everything seems to be working fine now.  I'm successfully loading u-boot and the kernel from the eMMC and mounting root from the NVMe.

[alexeymin]

I'm seeing the same 60s delay and the same timeout with the same SSD (Kingston A2000 256 Gb) when booting from eMMC:

Code:

[   1.886899] usb 2-1.2: Manufacturer: Sonix Technology Co., Ltd.
[  31.847180] vbus_5vout: disabling
[  64.487248] nvme nvme0: I/O 0 QID 0 timeout, disable controller
[  64.607160] nvme nvme0: Device shutdown incomplete; abort shutdown
[  64.608409] nvme nvme0: Identify Controller failed (-4)
[  64.609050] nvme nvme0: Removing after probe failure status: -5
[  64.640311] Freeing unused kernel memory: 2816K
[  64.707554] Run /init as init process

P.S. I'm using posmarketOS

What's interesting though is that u-boot seems to recognize the drive fine during boot: https://imgur.com/q59Hiel



Also when booting the same posmarketOS image (or Manjaro ARM image) from SD card there are no timeouts and it can see this nvme drive just fine.

Instructions to alter kernel cmdline from this post from kernel bug report - https://bugzilla.kernel.org/show_bug.cgi?id=195039#c73 - specifically 

Code:

nvme_core.default_ps_max_latency_us=0 pci_aspm=performance

also don't help

That kernel bug report also mentions kingston firmware update that might help - https://www.kingston.com/unitedstates/us...are-update - that you can only install from Windows

Code:

Firmware Rev S5Z42109(03-30-2021):
Fixed an issue that might cause the drive to become unresponsive on Linux systems

If nothing works I guess I'll try another SSD 

[psstoev]

Hello, @alexeymin 

I'm also currently experiencing flaky boots with the Kingston A2000 NVMe. I guess that this is most likely some U-Boot issue. When I first tried this drive, I used pcm720's U-Boot build, with great success. Note the patch that increases the timeout, to give more time to the NVMe drives to appear. I even had this U-Boot flashed to the SPI and I had the /boot partition on the NVMe. Alas, for some reason this build broke the SD Card reader - SD Cards were neither bootable, not even readable when inserted after boot.

I'm currently using the last build of mrfixit's U-Boot, with /boot partition on the eMMC and the root partition on the NVMe. It boots most of the time, and drops me at the initramfs prompt when it doesn't find the NVMe. From there, by reloading the nvme module (rmmod nvme; modprobe nvme),  I'm able to continue the boot process and everything boots successfully.

I don't have much time at the moment to play around with different U-Boot builds and kernels. I'm currently using the unofficial Debian installer by Daniel Thompson, along with his 5.7 kernel. I guess that there should be some combination of U-Boot and kernel that will behave more consistently. Somebody will surely find it.

[alexeymin]

@psstoev I think you might be right. I'm not sure which one of the U-boots you mentioned I'm using. postmarketOS packages this graphical U-boot with boot menu, which I have flashed to SPI. You can see how it looks like here: https://youtu.be/hjzNr7wnBpk?t=68

So, when I boot holding Ctrl+C to enter the u-boot menu, I can boot fine both from SD card and from eMMC, and drive is detected with no timeouts. I'm thinking the reason is that U-boot menu is displayed before u-boot probes nvme drives. So when u-boot doesn't try to interact with nvme, everything looks fine.

If, however, I don't invoke the boot menu and let u-boot go its default boot sequence, during which nvme drive is probed and detected by u-boot, then the drive is not detected in Linux.

So, looks like yes, u-boot touching nvme causes some problems for Linux.

[psstoev]

I've found a stable-enough setup for my Pinebook Pro with a Kingston A2000 NVMe - I've installed the pre-release of Tow-Boot from here. Now, my NVMe drive is properly detected almost every time, with some failures after reboot. Even then, after a reboot from the Tow-boot UI (it has a menu for selecting the boot device) everything works OK. Now my Debian installation (with encrypted root, thanks to danielt's debian installer) is booting and usable again.

The only caveat is that the microSD slot is still broken, but this was also the case with pcm720's U-boot build, as I mentioned before.