Is PCIE x4 bootable for NVMe SSD under Linux

[Tigger]

Sorry that I can't be of anymore help. I haven't got a RockPro64 yet.
[off-topic](Sales is not answering one of my questions. Although I made them aware of it. So I'm not buying yet.)[/off-topic]

[dukla2000]

At 4.4 the pcie & nvme initialisation is ready before the rootfs is mounted (even though I am booting off the SDcard).
At 4.18 the pcie & nvme initialisation only happens way after the rootfs is mounted.

To my best knowledge, it means Kernel -4.4 has pcie & nvme modules built into the Linux image, and Kernel -4.18 has pcie & nvme modules as external loadable modules (drivers).

Because we want rootfs on NVMe which plugged in PCIe slot,  so both pcie & nvme need to be loaded before rootfs is mounted.

If Ayufan can compile Kernel -4.18 as the same way as his compiled Kernel -4.4, (hopefully in the final release version, he will do) then everything should work.

I opened an issue on github - Ayufan says we must use initramfs. (Which makes sense as it is only the RockPro64 that needs these modules.)

So now to work out how to cook an enhanced initramfs. (And it is probably the same issue with eMMC not working at 4.18?)

[Tigger]

#-- If the ISO files from the following URL are live bootable to a command line or desktop with NVMe support.
#-- Make sure that the one you download has the exact same kernel version as the failing installation.
#-- https://www.ubuntu.com/download/server/arm

#-- Mount the installation that is failing to boot at : /media/live-user-name or ubuntu here/.......... (or similar). This will become the new root file system (/).
#-- Hide the ugly path in a meaning variable name, for repetitive use.
failing_target="/media/live-user-name or ubuntu here/.........." .

#-- Move into the failing target.
cd "$failing_target"
mount -t sysfs /sys sys/
mount -t proc /proc proc/
mount -o bind /dev dev/
mount --rbind /run run/
cp /etc/resolv.conf etc/resolv.conf

#-- Descend into failing (s)hell. Swaps root file systems, from live to failing.
chroot "/$failing_target" /bin/bash

#-- Ask the developer of the distro for the kernel module names, and in which file to enable them. I'm not an Ubuntu user, sorry. Arch Linux here.
#-- Or look in /proc/modules and try to find them, then use module-assistant and lsmod. But then you still need to find out how to dynamically load them.
#-- Maybe this is of use, although i doubt it : http://manpages.ubuntu.com/manpages/bion...eebsd.html

#-- Backup/Rename your current initramfs (/boot/initrd.img-*)

#-- Update initramfs against the newest/latest kernel version.
update-initramfs -u

#-- If update-initramfs fails with a read-only file system message (shouldn't). Use mkinitramfs instead.
mkinitramfs -o /boot/initrd.img-$(uname -r) #-- For the currently running kernel.

#-- Leave the chrooted environment. Returning to the live root file system.
exit

This is from Arch Linux (except for the commands to build initrd). It might be (slightly) different on Ubuntu.

[jzhang18]

That is exactly what we need to deal -- for Kernel4.18, build initramfs to include pcie and nvme 2 modules inside.
When Kernel is loaded, those modules can be instantiated immediately before re-mounting rootfs which we want to mount nvme device as rootfs.

---

There is another issue I found, lxde and mate desktop have some conflict with nvme disk.
When I use LXDE release image, after I login, the system hangs when MVMe disk is inserted (the red led on my PCIe -M.2 adapter is turned on, then off).
When I use minimal release image, MVMe disk is recognized (the red led on my PCIe -M.2 adapter is turned on and keeps on)
Based on minimal release image, I installed Ubuntu-Mate-Deskstop and LightDM. The systems works after I log in, but MVMe disk is not recognized (the red led on my PCIe -M.2 adapter is turned on, then off).

I think there are some Memory Address or IRQ conflict between GUI module and nvme module.

Could someone repeat and report the same issue, or it just happened to me?

[dukla2000]

OK - got rootfs on NVMe and 4.18 kernel:

Code:

$ uname -a
Linux rpro64.dukla.net 4.18.0-rc8-1060-ayufan-g45828c4db727 #1 SMP PREEMPT Thu Aug 9 19:41:21 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
chris@rpro64:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            982M     0  982M   0% /dev
tmpfs           199M  636K  199M   1% /run
/dev/nvme0n1p1  117G  7.4G  104G   7% /
tmpfs           994M   58M  936M   6% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           994M     0  994M   0% /sys/fs/cgroup
/dev/mmcblk0p6  112M  4.0K  112M   1% /boot/efi
tmpfs           199M  4.0K  199M   1% /run/user/1000

And yup, that NVMe is quicker than under 4.4:

Code:

$ sudo iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
    Iozone: Performance Test of File I/O
            Version $Revision: 3.429 $
        Compiled for 64 bit mode.
        Build: linux

    Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                 Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                 Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                 Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                 Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                 Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                 Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
                 Vangel Bojaxhi, Ben England, Vikentsi Lapa.

    Run began: Wed Aug 15 21:44:08 2018

    Include fsync in write timing
    O_DIRECT feature enabled
    Auto Mode
    File size set to 102400 kB
    Record Size 4 kB
    Record Size 16 kB
    Record Size 512 kB
    Record Size 1024 kB
    Record Size 16384 kB
    Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
    Output is in kBytes/sec
    Time Resolution = 0.000001 seconds.
    Processor cache size set to 1024 kBytes.
    Processor cache line size set to 32 bytes.
    File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    63115   109487   126839   120045    52068   140978                                                          
          102400      16   225493   319486   335585   347588   122025   321304                                                          
          102400     512   723276   737359   901821   973829   807431   735093                                                          
          102400    1024   720335   739367   924279  1004689   926714   735601                                                          
          102400   16384   510442   732234  1308627  1466939  1448327   734839                                                          

iozone test complete.

Will post instructions in the Tutorial sub-forum shortly.

[jzhang18]

a nice job! thanks

[dukla2000]

There is another issue I found,  lxde and mate desktop have some conflict with nvme disk.
When I use  LXDE release image, after I login, the system hangs when MVMe disk is inserted (the red led on my PCIe -M.2 adapter is turned on, then off).
When I use minimal release image,  MVMe disk is recognized (the red led on my PCIe -M.2 adapter is turned on and keeps on)
Based on minimal release image, I installed Ubuntu-Mate-Deskstop and LightDM.   The systems works after I log in, but MVMe disk is not recognized (the red led on my PCIe -M.2 adapter is turned on, then off).

I have no desire to try MATE, but my thoughts:

1) LXDE release is known to be less usable than minimal, stick to minimal.

2) If you boot a newly flashed minimal then hopefully you can see the NVMe? I suggest lsblk and sudo blkid all the time to check what is happening. If you add the Mate-Desktop and NOT lightdm presumably you login to a console prompt? Check again. Then add lightdm etc.

I do not have the Pine64 PCIe riser for my NVMe (got some other no-name) so have no insight what is correct LED behaviour, sorry.

[jzhang18]

1) LXDE release is known to be less usable than minimal, stick to minimal.

2) If you boot a newly flashed minimal then hopefully you can see the NVMe? I suggest lsblk and sudo blkid all the time to check what is happening. If you add the Mate-Desktop and NOT lightdm presumably you login to a console prompt? Check again. Then add lightdm etc.

I do not have the Pine64 PCIe riser for my NVMe (got some other no-name) so have no insight what is correct LED behaviour, sorry.

Thanks, dukla2000

It turned out my NVMe SSD is not stable.  Today I submit a RMA on the manufacture's web site. (https://www.corsair.com/us/en/)  

https://www.amazon.com/Corsair-Force-MP5...B01MQU72O9

I am also considering to buy Samsung NVMe SM961 SSD,  just like most people used here in the forum.

Nothing is related to M.2 ->PCIe adapter. the circuit seems very simple and stable.  majority of no-named adapter should work with Rockpro64 board.

After I get a new SSD, I will report it back.

---

BTW -- how about your instructions for kernel-4.18.0-rc8-1060-ayufan? eager to see it.

[dukla2000]

BTW -- how about your instructions for kernel-4.18.0-rc8-1060-ayufan?  eager to see it.

It is here (in the tutorials sub-forum).

[jzhang18]

Thanks for the quick response. it is a really nice solution!

---

I choose another approach
https://www.amazon.com/Express-Expansion...USB-C&th=1

I hooked up a regular 2.5 inch SSD through SATA6.0 to USB3 cable. I can use regular SSD as rootfs now.

The PCIe x4 provide enough current to power an USB external SSD. no extra 5v power needed.

---

jzhang18@rockpro64:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           388M  696K  387M   1% /run
/dev/sda1       116G  2.9G  108G   3% /
tmpfs           1.9G   23M  1.9G   2% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p6  112M   10K  112M   1% /boot/efi
tmpfs           388M  4.0K  388M   1% /run/user/112
tmpfs           388M   20K  388M   1% /run/user/1001