NVMe timeout when booting from eMMC but not SD
#1
I just got an NVMe adapter for my Pinebook Pro and have installed it with a Kingston A2000 250GB NVMe SSD.  This is an SSD that others have reported success with on the wiki.

The boot process stalls for 60s and then I get a timeout error like this:

Code:
[    3.742566] nvme nvme0: pci function 0000:01:00.0
[    3.742612] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[   64.480323] nvme nvme0: I/O 12 QID 0 timeout, disable controller
[   64.600233] nvme nvme0: Device shutdown incomplete; abort shutdown
[   64.640667] nvme nvme0: Identify Controller failed (881)
[   64.641472] nvme nvme0: Removing after probe failure status: -5

And then it boots normally, but none of the /dev/nvme0* devices are available.  This is on Debian Bullseye installed with Daniel Thompson's installer and kernel.  Kernel 5.5 and 5.7 both have this same behavior.

Here's the weird part.  I only have this problem booting from eMMC.  If I boot from an SD card with basically the same Bullseye install and the same kernel, the NVMe drive works perfectly.  So it doesn't seem to be bad hardware or a simple compatibility issue.

I'm a little stumped.
  Reply
#2
Looks like this may be some sort of hardware compatibility issue after all.  I removed the Kingston A2000 SSD and replaced it with a 250GB Western Digital Blue SN550, and everything seems to be working fine now.  I'm successfully loading u-boot and the kernel from the eMMC and mounting root from the NVMe.
  Reply
#3
I'm seeing the same 60s delay and the same timeout with the same SSD (Kingston A2000 256 Gb) when booting from eMMC:

Code:
[   1.886899] usb 2-1.2: Manufacturer: Sonix Technology Co., Ltd.
[  31.847180] vbus_5vout: disabling
[  64.487248] nvme nvme0: I/O 0 QID 0 timeout, disable controller
[  64.607160] nvme nvme0: Device shutdown incomplete; abort shutdown
[  64.608409] nvme nvme0: Identify Controller failed (-4)
[  64.609050] nvme nvme0: Removing after probe failure status: -5
[  64.640311] Freeing unused kernel memory: 2816K
[  64.707554] Run /init as init process

P.S. I'm using posmarketOS

What's interesting though is that u-boot seems to recognize the drive fine during boot: https://imgur.com/q59Hiel

[Image: q59Hiel.jpg]

Also when booting the same posmarketOS image (or Manjaro ARM image) from SD card there are no timeouts and it can see this nvme drive just fine.

Instructions to alter kernel cmdline from this post from kernel bug report - https://bugzilla.kernel.org/show_bug.cgi?id=195039#c73 - specifically 
Code:
nvme_core.default_ps_max_latency_us=0 pci_aspm=performance
also don't help Sad

That kernel bug report also mentions kingston firmware update that might help - https://www.kingston.com/unitedstates/us...are-update - that you can only install from Windows Sad
Code:
Firmware Rev S5Z42109(03-30-2021):
Fixed an issue that might cause the drive to become unresponsive on Linux systems


If nothing works I guess I'll try another SSD  Cry
  Reply
#4
Hello, @alexeymin 

I'm also currently experiencing flaky boots with the Kingston A2000 NVMe. I guess that this is most likely some U-Boot issue. When I first tried this drive, I used pcm720's U-Boot build, with great success. Note the patch that increases the timeout, to give more time to the NVMe drives to appear. I even had this U-Boot flashed to the SPI and I had the /boot partition on the NVMe. Alas, for some reason this build broke the SD Card reader - SD Cards were neither bootable, not even readable when inserted after boot.

I'm currently using the last build of mrfixit's U-Boot, with /boot partition on the eMMC and the root partition on the NVMe. It boots most of the time, and drops me at the initramfs prompt when it doesn't find the NVMe. From there, by reloading the nvme module (rmmod nvme; modprobe nvme),  I'm able to continue the boot process and everything boots successfully.

I don't have much time at the moment to play around with different U-Boot builds and kernels. I'm currently using the unofficial Debian installer by Daniel Thompson, along with his 5.7 kernel. I guess that there should be some combination of U-Boot and kernel that will behave more consistently. Somebody will surely find it. Rolleyes
  Reply
#5
@psstoev I think you might be right. I'm not sure which one of the U-boots you mentioned I'm using. postmarketOS packages this graphical U-boot with boot menu, which I have flashed to SPI. You can see how it looks like here: https://youtu.be/hjzNr7wnBpk?t=68

So, when I boot holding Ctrl+C to enter the u-boot menu, I can boot fine both from SD card and from eMMC, and drive is detected with no timeouts. I'm thinking the reason is that U-boot menu is displayed before u-boot probes nvme drives. So when u-boot doesn't try to interact with nvme, everything looks fine.

If, however, I don't invoke the boot menu and let u-boot go its default boot sequence, during which nvme drive is probed and detected by u-boot, then the drive is not detected in Linux.

So, looks like yes, u-boot touching nvme causes some problems for Linux.
  Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Is the same DD command appropriate for writing EMMC and SD cards wpeckham 4 121 Yesterday, 09:19 PM
Last Post: KC9UDX
  Trouble booting in Gentoo tdiggitydog 11 478 07-20-2021, 10:16 PM
Last Post: moonwalkers
  Manjaro wont recognize eMMC after booting with it disabled peasant 1 404 06-27-2021, 02:39 AM
Last Post: Arwen
Bug Pinebook Pro NVME stopped working secretagentdude 3 1,141 06-12-2021, 11:50 AM
Last Post: secretagentdude
  eMMC frequently not found by Linux kernel (especially on low battery) v-man 14 6,865 05-14-2021, 01:18 PM
Last Post: djhnsn
  PBP won't boot from eMMC or SD card: a saga archaicmeander 18 3,684 02-06-2021, 07:40 PM
Last Post: wdt
  Anyone can get deep sleep fully works with NVME rootfs? plumlis 0 752 01-11-2021, 09:58 AM
Last Post: plumlis
  UBOOT - Mainline not booting from SD card / BSP OK Anarethos 1 1,084 01-10-2021, 12:33 AM
Last Post: Anarethos
  eMMC Trouble After Flashing WRONG Image balin 7 2,472 12-29-2020, 04:24 PM
Last Post: KC9UDX
  installing OS from SD to eMMC methods for flashing SD PineHead7 3 1,662 12-22-2020, 06:52 PM
Last Post: jiyong

Forum Jump:


Users browsing this thread: 1 Guest(s)