load root on NVMe and /boot on eMMC

[Vikk42]

While looking for a better way to get the kernel on my NVMe I ran across this guy who figured out the best way to load his system. Put the boot files on the boot device and the root files on the fastest device, NVMe.

I'm hoping someone here can put some flesh on these bones to make it work for us. Did I say I'm a linux/programmer novice, yes that is me. Eventually I could figure this out. Think I know how to proceed if no one else cares, but I would like expert advice too.


https://delightlylinux.wordpress.com/201...sing-nvme/

An NVMe SSD makes a superb system drive for a Linux installation, but an easy point to forget is how to make Linux boot on systems that do not support NVMe booting from the motherboard BIOS.
So, here is a quick reminder.

First, Use a secondary SSD that the motherboard BIOS does recognize
This is the key. In a recent Linux Mint 18.3 installation, I used the Samsung 960 EVO NVMe connected to the motherboard using a PCIe to NVMe adapter because the older motherboard lacked an M.2 slot.
The problem is that the motherboard did not recognize the Samsung 960 as a boot device. Sure, Linux Mint 18.3 would install to the NVMe drive fine from a LiveUSB, but there was no way to boot the system from it.
Was all lost?
Nope! Remember, this is Linux that we are talking about, and Linux offers flexible customization. When Linux boots, it first looks for the /boot directory to kickstart the process.
/boot can be located on any other boot device, so it does not have to reside on the same drive as root. We can install the main Linux system on the NVMe, but specify /boot on a different boot device that the motherboard BIOS will boot from.
I had an SSD installed in the system. The motherboard recognized the SSD as a valid boot drive, so I created a 4GB partition (more than necessary, but just in case). The rest of the drive was formatted as ext4 for regular data storage.
The system and swap partitions were located on the Samsung NVMe. The boot loader installation was set to the SSD, not the NVMe. Below is the Install dialog from a Linux Mint 18.3 installation that shows how the partitions were set up.

Linux Mint 18.3 Cinnamon installation showing how to arrange partitions so a system will boot Linux is the Linux has been installed onto an NVMe device that lacks NVMe booting.
/dev/nvme0n1 is the NVMe device divided into two partitions: btrfs for the Linux installation and all Linux directories except for /boot, and the swap space. btrfs was used here, but any other bootable filesystem is valid. Install Linux normally to the NVMe.

• /dev/nvme0n1p1     Linux root filesystem
  
  
• /dev/nvme0n1p2     Linux swap partition
  

/dev/sda in this example is the SSD. The motherboard BIOS detects this as a valid boot drive, so this is where we must place the /boot partition. All other directories in the Linux hierarchy may reside on the NVMe as you wish, but /boot is special. We need to place it on the boot drive.
The SSD was divided into two partitions: a small 4G btrfs partition used solely for booting, and the remaining space as ext4 for data storage.

• /dev/sda1   Regular storage space. Nothing special.
  
  
• /dev/sda2   /boot goes here. Small partition.
  

/dev/sdb is the LiveUSB containing the ISO, so ignore it.
The final important step is the boot loader. Specify the SSD (/dev/sda in the example), not the NVMe device, as the boot loader.

• /dev/sda   The boot loader. BIOS looks for this. Must be the SSD, not the NVMe.
  

BIOS will look for /boot located on the SSD and proceed to boot the rest of the Linux system. Specify the SSD device itself as the boot loader, not a specific partition on the SSD. (Use /dev/sda, not /dev/sda1 or /dev/sda2.)
Of course, you are not limited to an SSD as a boot device. A spare USB, SCSI, or mechanical hard drive will also work as long as BIOS recognizes it as a boot device. But, watch out! If the boot device containing /boot is missing or fails, then the system will not boot.
Linux Mint booted and ran perfectly following this design. Performance with the SSD does not decrease because the SSD is used only for the bootstrap process. Once Linux is loaded, it runs from the much faster NVMe as the fast and snappy beast Linux was meant to be.

[hmuller]

While looking for a better way to get the kernel on my NVMe I ran across this guy who figured out the best way to load his system. Put the boot files on the boot device and the root files on the fastest device, NVMe.

I'm hoping someone here can put some flesh on these bones to make it work for us. Did I say I'm a linux/programmer novice, yes that is me. Eventually I could figure this out. Think I know how to proceed if no one else cares, but I would like expert advice too.


https://delightlylinux.wordpress.com/201...sing-nvme/

An NVMe SSD makes a superb system drive for a Linux installation, but an easy point to forget is how to make Linux boot on systems that do not support NVMe booting from the motherboard BIOS.
So, here is a quick reminder.

First, Use a secondary SSD that the motherboard BIOS does recognize
This is the key. In a recent Linux Mint 18.3 installation, I used the Samsung 960 EVO NVMe connected to the motherboard using a PCIe to NVMe adapter because the older motherboard lacked an M.2 slot.
The problem is that the motherboard did not recognize the Samsung 960 as a boot device. Sure, Linux Mint 18.3 would install to the NVMe drive fine from a LiveUSB, but there was no way to boot the system from it.
Was all lost?
Nope! Remember, this is Linux that we are talking about, and Linux offers flexible customization. When Linux boots, it first looks for the /boot directory to kickstart the process.
/boot can be located on any other boot device, so it does not have to reside on the same drive as root. We can install the main Linux system on the NVMe, but specify /boot on a different boot device that the motherboard BIOS will boot from.
I had an SSD installed in the system. The motherboard recognized the SSD as a valid boot drive, so I created a 4GB partition (more than necessary, but just in case). The rest of the drive was formatted as ext4 for regular data storage.
The system and swap partitions were located on the Samsung NVMe. The boot loader installation was set to the SSD, not the NVMe. Below is the Install dialog from a Linux Mint 18.3 installation that shows how the partitions were set up.

Linux Mint 18.3 Cinnamon installation showing how to arrange partitions so a system will boot Linux is the Linux has been installed onto an NVMe device that lacks NVMe booting.
/dev/nvme0n1 is the NVMe device divided into two partitions: btrfs for the Linux installation and all Linux directories except for /boot, and the swap space. btrfs was used here, but any other bootable filesystem is valid. Install Linux normally to the NVMe.

• /dev/nvme0n1p1     Linux root filesystem
  
  
• /dev/nvme0n1p2     Linux swap partition
  

/dev/sda in this example is the SSD. The motherboard BIOS detects this as a valid boot drive, so this is where we must place the /boot partition. All other directories in the Linux hierarchy may reside on the NVMe as you wish, but /boot is special. We need to place it on the boot drive.
The SSD was divided into two partitions: a small 4G btrfs partition used solely for booting, and the remaining space as ext4 for data storage.

• /dev/sda1   Regular storage space. Nothing special.
  
  
• /dev/sda2   /boot goes here. Small partition.
  

/dev/sdb is the LiveUSB containing the ISO, so ignore it.
The final important step is the boot loader. Specify the SSD (/dev/sda in the example), not the NVMe device, as the boot loader.

• /dev/sda   The boot loader. BIOS looks for this. Must be the SSD, not the NVMe.
  

BIOS will look for /boot located on the SSD and proceed to boot the rest of the Linux system. Specify the SSD device itself as the boot loader, not a specific partition on the SSD. (Use /dev/sda, not /dev/sda1 or /dev/sda2.)
Of course, you are not limited to an SSD as a boot device. A spare USB, SCSI, or mechanical hard drive will also work as long as BIOS recognizes it as a boot device. But, watch out! If the boot device containing /boot is missing or fails, then the system will not boot.
Linux Mint booted and ran perfectly following this design. Performance with the SSD does not decrease because the SSD is used only for the bootstrap process. Once Linux is loaded, it runs from the much faster NVMe as the fast and snappy beast Linux was meant to be.

I do not yet own an embedded SBC (single board computer), but will once the Pinebook Pro goes into production. So I have not yet done this, but will once I have the hardware.

This guy has done some work on this, but was not successful at the time he published this:

https://yuncloud.ddns.net/wordpress/blog...nvme-disk/

This article most likely contains the solution to the problem:

https://unix.stackexchange.com/questions...vme-device

Later this year, I plan on tackling this and will post the solution in the forums.

[stuartiannaylor]

While looking for a better way to get the kernel on my NVMe I ran across this guy who figured out the best way to load his system. Put the boot files on the boot device and the root files on the fastest device, NVMe.

I'm hoping someone here can put some flesh on these bones to make it work for us. Did I say I'm a linux/programmer novice, yes that is me. Eventually I could figure this out. Think I know how to proceed if no one else cares, but I would like expert advice too.


https://delightlylinux.wordpress.com/201...sing-nvme/

An NVMe SSD makes a superb system drive for a Linux installation, but an easy point to forget is how to make Linux boot on systems that do not support NVMe booting from the motherboard BIOS.
So, here is a quick reminder.

First, Use a secondary SSD that the motherboard BIOS does recognize
This is the key. In a recent Linux Mint 18.3 installation, I used the Samsung 960 EVO NVMe connected to the motherboard using a PCIe to NVMe adapter because the older motherboard lacked an M.2 slot.
The problem is that the motherboard did not recognize the Samsung 960 as a boot device. Sure, Linux Mint 18.3 would install to the NVMe drive fine from a LiveUSB, but there was no way to boot the system from it.
Was all lost?
Nope! Remember, this is Linux that we are talking about, and Linux offers flexible customization. When Linux boots, it first looks for the /boot directory to kickstart the process.
/boot can be located on any other boot device, so it does not have to reside on the same drive as root. We can install the main Linux system on the NVMe, but specify /boot on a different boot device that the motherboard BIOS will boot from.
I had an SSD installed in the system. The motherboard recognized the SSD as a valid boot drive, so I created a 4GB partition (more than necessary, but just in case). The rest of the drive was formatted as ext4 for regular data storage.
The system and swap partitions were located on the Samsung NVMe. The boot loader installation was set to the SSD, not the NVMe. Below is the Install dialog from a Linux Mint 18.3 installation that shows how the partitions were set up.

Linux Mint 18.3 Cinnamon installation showing how to arrange partitions so a system will boot Linux is the Linux has been installed onto an NVMe device that lacks NVMe booting.
/dev/nvme0n1 is the NVMe device divided into two partitions: btrfs for the Linux installation and all Linux directories except for /boot, and the swap space. btrfs was used here, but any other bootable filesystem is valid. Install Linux normally to the NVMe.

• /dev/nvme0n1p1     Linux root filesystem
  
  
• /dev/nvme0n1p2     Linux swap partition
  

/dev/sda in this example is the SSD. The motherboard BIOS detects this as a valid boot drive, so this is where we must place the /boot partition. All other directories in the Linux hierarchy may reside on the NVMe as you wish, but /boot is special. We need to place it on the boot drive.
The SSD was divided into two partitions: a small 4G btrfs partition used solely for booting, and the remaining space as ext4 for data storage.

• /dev/sda1   Regular storage space. Nothing special.
  
  
• /dev/sda2   /boot goes here. Small partition.
  

/dev/sdb is the LiveUSB containing the ISO, so ignore it.
The final important step is the boot loader. Specify the SSD (/dev/sda in the example), not the NVMe device, as the boot loader.

• /dev/sda   The boot loader. BIOS looks for this. Must be the SSD, not the NVMe.
  

BIOS will look for /boot located on the SSD and proceed to boot the rest of the Linux system. Specify the SSD device itself as the boot loader, not a specific partition on the SSD. (Use /dev/sda, not /dev/sda1 or /dev/sda2.)
Of course, you are not limited to an SSD as a boot device. A spare USB, SCSI, or mechanical hard drive will also work as long as BIOS recognizes it as a boot device. But, watch out! If the boot device containing /boot is missing or fails, then the system will not boot.
Linux Mint booted and ran perfectly following this design. Performance with the SSD does not decrease because the SSD is used only for the bootstrap process. Once Linux is loaded, it runs from the much faster NVMe as the fast and snappy beast Linux was meant to be.

I do not yet own an embedded SBC (single board computer), but will once the Pinebook Pro goes into production. So I have not yet done this, but will once I have the hardware.

This guy has done some work on this, but was not successful at the time he published this:

https://yuncloud.ddns.net/wordpress/blog...nvme-disk/

This article most likely contains the solution to the problem:

https://unix.stackexchange.com/questions...vme-device

Later this year, I plan on tackling this and will post the solution in the forums.

https://forum.frank-mankel.org/topic/208...vme-platte
google translate to the rescue

[Vikk42]

While looking for a better way to get the kernel on my NVMe I ran across this guy who figured out the best way to load his system. Put the boot files on the boot device and the root files on the fastest device, NVMe.

I'm hoping someone here can put some flesh on these bones to make it work for us. Did I say I'm a linux/programmer novice, yes that is me. Eventually I could figure this out. Think I know how to proceed if no one else cares, but I would like expert advice too.


https://delightlylinux.wordpress.com/201...sing-nvme/

An NVMe SSD makes a superb system drive for a Linux installation, but an easy point to forget is how to make Linux boot on systems that do not support NVMe booting from the motherboard BIOS.
So, here is a quick reminder.

I do not yet own an embedded SBC (single board computer), but will once the Pinebook Pro goes into production. So I have not yet done this, but will once I have the hardware.

This guy has done some work on this, but was not successful at the time he published this:

https://yuncloud.ddns.net/wordpress/blog...nvme-disk/

This article most likely contains the solution to the problem:

https://unix.stackexchange.com/questions...vme-device

Later this year, I plan on tackling this and will post the solution in the forums.

https://forum.frank-mankel.org/topic/208...vme-platte
google translate to the rescue

I am aware of Frank-Mankel solution.

But as he points out it has a problem in that it will be broke by any system update. 

update
A shutdown now works fine, even after waking up via the power button.
Kernel update

Be careful , with a kernel update you have to repeat the changes in the file /boot/extlinux/extlinux.conf .

If we get off to the right start by letting the system build the file systems in the right place of boot software on boot drives and root and swap on NVMe, the problem fixes itself.

Can someone point out where U-Boot chooses to put the kernel and swap files on the system.

I guess if it was a trivial solution Frank-Mankel would have done it already?

[stuartiannaylor]

While looking for a better way to get the kernel on my NVMe I ran across this guy who figured out the best way to load his system. Put the boot files on the boot device and the root files on the fastest device, NVMe.

I'm hoping someone here can put some flesh on these bones to make it work for us. Did I say I'm a linux/programmer novice, yes that is me. Eventually I could figure this out. Think I know how to proceed if no one else cares, but I would like expert advice too.


https://delightlylinux.wordpress.com/201...sing-nvme/

An NVMe SSD makes a superb system drive for a Linux installation, but an easy point to forget is how to make Linux boot on systems that do not support NVMe booting from the motherboard BIOS.
So, here is a quick reminder.

I do not yet own an embedded SBC (single board computer), but will once the Pinebook Pro goes into production. So I have not yet done this, but will once I have the hardware.

This guy has done some work on this, but was not successful at the time he published this:

https://yuncloud.ddns.net/wordpress/blog...nvme-disk/

This article most likely contains the solution to the problem:

https://unix.stackexchange.com/questions...vme-device

Later this year, I plan on tackling this and will post the solution in the forums.

https://forum.frank-mankel.org/topic/208...vme-platte
google translate to the rescue

I am aware of Frank-Mankel solution.

But as he points out it has a problem in that it will be broke by any system update. 

update
A shutdown now works fine, even after waking up via the power button.
Kernel update

Be careful , with a kernel update you have to repeat the changes in the file /boot/extlinux/extlinux.conf .

If we get off to the right start by letting the system build the file systems in the right place of boot software on boot drives and root and swap on NVMe, the problem fixes itself.

Can someone point out where U-Boot chooses to put the kernel and swap files on the system.

I guess if it was a trivial solution Frank-Mankel would have done it already?

Its not uboot depends on distro but think its just part of the initcpio hooks if you where using Arch, initramfs if deb/buntu?
Have a look at for hints as seems to have some good tutorials that just have to substitute with specifics.
https://github.com/umiddelb/armhf/wiki/H...ARM-device
https://github.com/umiddelb/armhf/wiki/G...-U-Boot%22

I currently have a rockpi4 which can boot from spi nor to nvme ssd, so blind until the RockPro64 turns up in the next couple of weeks.
So dunno until I also try in a couple of weeks.