Is PCIE x4 bootable for NVMe SSD under Linux

[Bullet64]

Someone get it to work with mainline? For me only 4.4 is working.

[dukla2000]

Someone get it to work with mainline? For me only 4.4 is working.

Nope - I have 2 problems with 4.18 and so have reverted to 4.4.132 for the moment.

1. If I build an SDcard with 4.18 it boots fine if I have no eMMC nor NVMe plugged in. If I plug in the eMMC the boot hangs - it seems 4.18 cannot initialise eMMC ?
   
2. If I boot with just SDcard & NVMe 4.18 boots fine. If I move the 4.18 rootfs to /dev/nvme0n1 (as per your page) the boot hangs. I think it is because the PCI initialisation is very late in the standard 4.18 kernel and after the rootfs is mounted (comparing a boot from the SDcard to a boot from the NVMe.)
   

[Bullet64]

Someone get it to work with mainline? For me only 4.4 is working.

Nope - I have 2 problems with 4.18 and so have reverted to 4.4.132 for the moment.

1. If I build an SDcard with 4.18 it boots fine if I have no eMMC nor NVMe plugged in. If I plug in the eMMC the boot hangs - it seems 4.18 cannot initialise eMMC ?
   
2. If I boot with just SDcard & NVMe 4.18 boots fine. If I move the 4.18 rootfs to /dev/nvme0n1 (as per your page) the boot hangs. I think it is because the PCI initialisation is very late in the standard 4.18 kernel and after the rootfs is mounted (comparing a boot from the SDcard to a boot from the NVMe.)
   

1:
will test it later today

2:
will rootwait=20 fix this?

[dukla2000]

2:
will rootwait=20 fix this?

I didn't have time to try this yet: as far as I search seems rootwait doesn't take parameters. It is by default an infinite wait?
My thought was based on checking the log when booting 4.18 off an SDcard with the NVMe in place. The rootfs mounts around 4s, the PCIe initialises around 11s. It seems to me at 4.18 the rootfs is needed to complete the PCIe initialisation!

A random thought I had, and tried with zero success, was to try a dos partition table with a partition. Works fine for 4.4 but not 4.18.

[Tigger]

If the delay is what you want to try :

rootdelay= : Delay (in seconds) to pause before attempting to mount the root filesystem

rootwait : Wait (indefinitely) for root device to show up. Useful for devices that are detected asynchronously (e.g. USB and MMC devices).

You would be better of with a $1.99 Serial console from the Pine shop.
What exactly is on the eMMC ?

[dukla2000]

Tried rootdelay=20, did not help. Also tried timeout 200 which was equally futile.

I have zero knowledge on what causes what to happen during boot but clearly 4.4 gets the nvme ready very early in the process, and 4.18 does not. See the 2 logs below.

Code:

$uname -a
Linux rockpro64 4.4.132-1075-rockchip-ayufan-ga83beded8524 #1 SMP Thu Jul 26 08:22:22 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

$dmesg | grep nvme
[    3.366224] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[    3.509043]  nvme0n1: p1

$dmesg | grep pcie
[    0.504048] of_get_named_gpiod_flags: parsed 'gpio' property of node '/vcc3v3-pcie-regulator[0]' - status (0)
[    0.504098] reg-fixed-voltage vcc3v3-pcie-regulator: Looking up vin-supply from device tree
[    0.504137] vcc3v3_pcie: supplied by dc_12v
[    0.504203] vcc3v3_pcie: 3300 mV
[    0.504358] reg-fixed-voltage vcc3v3-pcie-regulator: vcc3v3_pcie supplying 3300000uV
[    2.774636] vcc3v3_pcie: disabling
[    3.078824] phy phy-pcie-phy.9: Looking up phy-supply from device tree
[    3.078828] phy phy-pcie-phy.9: Looking up phy-supply property in node /pcie-phy failed
[    3.142089] rockchip-pcie f8000000.pcie: GPIO lookup for consumer ep
[    3.142103] rockchip-pcie f8000000.pcie: using device tree for GPIO lookup
[    3.142163] of_get_named_gpiod_flags: parsed 'ep-gpios' property of node '/pcie@f8000000[0]' - status (0)
[    3.142496] rockchip-pcie f8000000.pcie: Looking up vpcie3v3-supply from device tree
[    3.142741] rockchip-pcie f8000000.pcie: Looking up vpcie1v8-supply from device tree
[    3.142753] rockchip-pcie f8000000.pcie: Looking up vpcie1v8-supply property in node /pcie@f8000000 failed
[    3.142786] rockchip-pcie f8000000.pcie: no vpcie1v8 regulator found
[    3.150011] rockchip-pcie f8000000.pcie: Looking up vpcie0v9-supply from device tree
[    3.150027] rockchip-pcie f8000000.pcie: Looking up vpcie0v9-supply property in node /pcie@f8000000 failed
[    3.150057] rockchip-pcie f8000000.pcie: no vpcie0v9 regulator found
[    3.157191] rockchip-pcie f8000000.pcie: invalid power supply
[    3.229681] PCI host bridge /pcie@f8000000 ranges:
[    3.259602] rockchip-pcie f8000000.pcie: PCI host bridge to bus 0000:00
[    3.344039] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
[    3.351408] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    3.365337] pcie_pme 0000:00:00.0:pcie01: service driver pcie_pme loaded
[    3.365681] aer 0000:00:00.0:pcie02: service driver aer loaded

$dmesg | grep EXT4-fs
[    4.022277] EXT4-fs (mmcblk0p7): mounted filesystem with writeback data mode. Opts: (null)

At 4.4 the pcie & nvme initialisation is ready before the rootfs is mounted (even though I am booting off the SDcard).

Code:

$uname -a
Linux rockpro64 4.18.0-rc8-1060-ayufan-g45828c4db727 #1 SMP PREEMPT Thu Aug 9 19:41:21 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

$dmesg ? grepp nvme
[   11.267625] nvme nvme0: pci function 0000:01:00.0
[   11.267700] nvme 0000:01:00.0: enabling device (0000 -> 0002)
[   11.403278]  nvme0n1: p1

$dmesg | grep pcie
[    1.772645] vcc3v3_pcie: supplied by dc_12v
[    3.557384] vcc3v3_pcie: disabling
[   11.190914] rockchip-pcie f8000000.pcie: no vpcie12v regulator found
[   11.190998] rockchip-pcie f8000000.pcie: no vpcie1v8 regulator found
[   11.191009] rockchip-pcie f8000000.pcie: no vpcie0v9 regulator found
[   11.258608] rockchip-pcie f8000000.pcie: host bridge /pcie@f8000000 ranges:
[   11.258634] rockchip-pcie f8000000.pcie:   MEM 0xfa000000..0xfbdfffff -> 0xfa000000
[   11.258642] rockchip-pcie f8000000.pcie:    IO 0xfbe00000..0xfbefffff -> 0xfbe00000
[   11.258909] rockchip-pcie f8000000.pcie: PCI host bridge to bus 0000:00

$dmesg | grep EXT4-fs
[    4.534168] EXT4-fs (mmcblk0p7): mounted filesystem with writeback data mode. Opts: (null)

At 4.18 the pcie & nvme initialisation only happens way after the rootfs is mounted.

What exactly is on the eMMC ?

Is a bionic minimal image - / on p7 etc. This is a separate issue I have with 4.18: if the emmc is plugged in when booting then the 4.18 kernel fails to complete its boot as it appears to be not able to initialise the emmc, and gets into infinite loop

Code:

mmc1: running CQE recovery

repeating this every 60s

[jzhang18]

dukla2000,

I saw your comments:
At 4.4 the pcie & nvme initialisation is ready before the rootfs is mounted (even though I am booting off the SDcard).
At 4.18 the pcie & nvme initialisation only happens way after the rootfs is mounted.

To my best knowledge, it means Kernel -4.4 has pcie & nvme modules built into the Linux image, and Kernel -4.18 has pcie & nvme modules as external loadable modules (drivers).

Because we want rootfs on NVMe which plugged in PCIe slot, so both pcie & nvme need to be loaded before rootfs is mounted.

If Ayufan can compile Kernel -4.18 as the same way as his compiled Kernel -4.4, (hopefully in the final release version, he will do) then everything should work.

[Tigger]

..........

You would be better of with a $1.99 Serial console from the Pine shop.

Adding to my earlier advice. See what the Serial Console can do for you here : https://www.linuxjournal.com/content/handy-u-boot-trick
Why keep on guessing, if $1.99 will tell you what's going on ?
In the serial console normally, u-boot can be interrupted by hitting Escape during boot. You can then manually check the environment, check what's up and what's not, manually fatload (FAT) or ext2load (EXT2/EXT3) the kernel into memory and execute it with bootm or bootcmd or go (Windows).
But U-boot can be configured to not allow interruption.
Doesn't have to be the Serial Console as sold by Pine. The one from Adafruit for Pi will probably be compatible : https://www.adafruit.com/product/954

[Tigger]

..........

What exactly is on the eMMC ?

Is a bionic minimal image - / on p7 etc. This is a separate issue I have with 4.18: if the emmc is plugged in when booting then the 4.18 kernel fails to complete its boot as it appears to be not able to initialise the emmc, and gets into infinite loop

Code:

mmc1: running CQE recovery

repeating this every 60s

During CQE recovery, CQE and eMMC are stopped. The CQE and eMMC queues are purged/discarded. Most likely one of the queues (or both) fail at purging, and the recovery process is retried.

Have you setup the eMMC to be bootable ?
Did you flash the image for RockPro64 (or Rock64) ? ..... (Don't fret, just asking.)

If it's the Ubuntu Bionic Bear from Ayufan that you flashed, you could check the issues at his Github account. Someone might already have reported it, and a fix given. : https://github.com/ayufan-rock64/linux-build/releases

[dukla2000]

Have you setup the eMMC to be bootable ?
Did you flash the image for RockPro64 (or Rock64) ? ..... (Don't fret, just asking.)

Yeah there are no issues with the eMMC itself. It has the 0.7.9 RockPro64 image on it and boots 4.4.132 kernel just fine, either on its own or with rootfs on the NVMe. Or indeed can boot 0.7.9 with 4.4.132 kernel from SDcard and still see the eMMC if I want (playing with the jumper during boot).

The issue is if I try boot a 4.18.0-rc8 kernel (inside 0.7.9) then the boot hangs as it cannot initialise the eMMC: this is a problem whether using an SDcard or the eMMC itself as the boot device.

Thanks for other pointers on serial console etc.