Mainline U-Boot with SPI, NVMe and SATA boot support

[belfastraven]

After seeing @pcm720's thread on porting the RK3399 PCIe/NVMe driver from Radxa's Rock Pi 4 U-Boot version, and this guide to building U-Boot for RockPro64 with all mainline, open source components, I've copied the aforementioned PCIe driver into mainline U-Boot, fixed a bug in the driver, created some configuration for U-Boot and a build script to create U-Boot binaries for RockPro64 with SPI and NVMe boot support.

The code is available on Github and:

• is based on mainline U-Boot
  
• doesn't use any binary-only blobs for booting
  
• with support for installation to SPI flash
  
• with support for storing the U-Boot Environment in SPI flash (configured to store it at offset 0x3f8000 bytes, 8KBytes size)
  
• can insert "partition" entries in the SPI Flash device tree for Linux so that /dev/mtdX devices are created for each area of the SPI flash layout it uses (if there is a spi-nor flash node in Linux's device tree for it to stick them on)
  
• with support for booting Linux off an NVMe device
  
• also supports the normal U-Boot features of booting from eMMC/SD/USB/Network
  

You can see the differences between current U-Boot master and this version here.

I have tested the SPI installation, SPI environment saving & loading, USB boot, network boot, and NVMe boot on my RockPro64 with a Samsung 970 EVO Plus NVMe device. I don't have any other NVMe devices to test with, and I'd like to verify this works for others, so if anyone else is interested, and would like to test this build, read on. In particular I'm uncertain if this works for other NVMe devices and possibly other SPI flash chips - my RockPro64 has a Gigadevice gd25q128 chip, but I've heard some other versions may use Winbond flash chips which I haven't been able to test with.

The build script (here) has several stages and builds several different binaries. The binaries attached to this release are:

• flash_spi.img.gz - this is an image which can be un-gzipped and written to an SD card, it starts U-Boot from the SD card and then runs this script to install U-Boot to SPI Flash. The image includes flash_spi.scr and spi_combined.img. More detailed info here.
  
• flash_spi.scr - this is the compiled script to install U-Boot to SPI flash, it's not useful except for certain specific scenarios (if you want to make your own install SD-card). It is included in flash_spi.img.gz.
  
• mmc_idbloader.img - this is the image containing the TPL+SPL of U-Boot compiled for eMMC/SD card install, it should be written to eMMC or SD card at offset 64 sectors.
  
• mmc_u-boot.itb - this is the image containing U-Boot proper compiled for eMMC/SD card install, it should be written to eMMC or SD card at offset 16384 sectors.
  
• spi_idbloader.img - this is the image containing the TPL+SPL of U-Boot compiled for SPI Flash install, it should be written to SPI flash at offset 0, but for writing I recommend using spi_combined.img instead - see below.
  
• spi_u-boot.itb - this is the image containing U-Boot proper compiled for SPI Flash install, it should be written to SPI flash at offset 0x60000 bytes, but for writing I recommend using spi_combined.img instead - see below.
  
• spi_combined.img - this contains spi_idbloader.img, padding, and then spi_u-boot.itb at offset 0x60000 bytes, this can be written to SPI flash starting at offset 0 and will write spi_idbloader.img and spi_u-boot.itb at the correct locations.
  

For testing it - you'll need to have a serial / UART console available so that you can see log messages and interact with U-Boot that way - there is no HDMI or keyboard support in U-Boot.

Also, I strongly recommend *not* trying to flash it to SPI at first, since SPI is first in the RK3399 boot order, so if you flash something there and it doesn't work, your board is useless until you temporarily disable SPI to recover it.

Instead, first as a test just write mmc_idbloader.img and mmc_u-boot.itb to a spare SD card, remove / disable eMMC and SPI if you are currently booting off them, and test the NVMe boot functionality by booting off SD card, interrupting U-boot at the "Hit any key to stop autoboot" stage, and then entering: run bootcmd_nvme0 at the U-Boot => prompt to scan the NVMe device for bootable files and boot from it. On your NVMe drive, you just need something U-Boot can boot, e.g. a root partition with a /boot/extlinux/extlinux.conf or /boot/boot.scr telling it how to load the kernel, initrd and FDT file.

Then, if testing using a spare SD card works successfully, try installing to SPI, making sure you have a way to recover by disabling SPI temporarily if needed.

Some caveats: I've seen it mentioned that the PCIe driver for RK3399 from the Rock Pi 4 U-Boot is somewhat buggy, and looking at the code it seems like some PCIe features are missing or deliberately disabled; there is no support for more than one PCI device and no support for any other type of device apart from NVMe (so no PCI SATA adapter support at the moment, sorry). I don't know enough about the RK3399 PCIe controller to be able to find any more bugs from reading the code, but they might be there.
Also, if you're not sure what "write mmc_idbloader.img to eMMC or SD card at offset 64 sectors." means or how to do that, I recommend not trying this at all until some other folks have verified that it works, and it's packaged in an easier to use format.

---

This is very exciting. I am happy to test this, as I have already messed up my flash a few times and am adept at shortings pins 23 and 25 and getting into makrom mode to fix things :-)

Running as you have suggested, from the SD card, I am getting the following: this is from a current armbian mainline build. I normally would be booting from an DS card on which I have written mainline u-boot, as armbian is still using the old rockchip based u-boot for this board. This is a Samsung pm981 nvme. It is formatted btrs? Is that the problem, or should I be checking something else?

-Boot TPL 2020.01-rc5-01969-g0bfd4738c6 (Jan 03 2020 - 15:46:42)
Channel 0: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
Channel 1: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
256B stride
256B stride
lpddr4_set_rate: change freq to 400000000 mhz 0, 1
lpddr4_set_rate: change freq to 800000000 mhz 1, 0
Trying to boot from BOOTROM
Returning to boot ROM...

U-Boot SPL 2020.01-rc5-01969-g0bfd4738c6 (Jan 03 2020 - 15:46:42 +0000)
Trying to boot from MMC1
NOTICE: BL31: v2.2(release):v2.2-277-gb1af2a51
NOTICE: BL31: Built : 15:44:18, Jan 3 2020


U-Boot 2020.01-rc5-01969-g0bfd4738c6 (Jan 03 2020 - 15:46:42 +0000)

Model: Pine64 RockPro64
DRAM: 3.9 GiB
PMIC: RK808
MMC: dwmmc@fe320000: 1, sdhci@fe330000: 0
Loading Environment from MMC... Card did not respond to voltage select!
*** Warning - No block device, using default environment

In: serial@ff1a0000
Out: serial@ff1a0000
Err: serial@ff1a0000
Model: Pine64 RockPro64
rockchip_dnl_key_pressed: adc_channel_single_shot fail!
Net: eth0: ethernet@fe300000
Hit any key to stop autoboot: 0


=> run bootcmd_nvme0

Device 0: Vendor: 0x144d Rev: EXD7101Q Prod: S444NE0K603440
Type: Hard Disk
Capacity: 244198.3 MB = 238.4 GB (500118192 x 512)
... is now current device
** Unrecognized filesystem type **

JI'm just including the log this time so that you can see that I did use the proper images....

Once again, thanks.