Mainline U-Boot with SPI, NVMe and SATA boot support

[sigmaris]

Update 2020-02-01: Made a new release based on v2020.01, with added SATA AHCI boot support, and a fix for the "rockchip_dnl_key_pressed: adc_channel_single_shot fail" issue where the Recover button wasn't being read successfully.

--
After seeing @pcm720's thread on porting the RK3399 PCIe/NVMe driver from Radxa's Rock Pi 4 U-Boot version, and this guide to building U-Boot for RockPro64 with all mainline, open source components, I've copied the aforementioned PCIe driver into mainline U-Boot, fixed a bug in the driver, created some configuration for U-Boot and a build script to create U-Boot binaries for RockPro64 with SPI and NVMe boot support.

The code is available on Github and:

• is based on mainline U-Boot
  
• doesn't use any binary-only blobs for booting
  
• with support for installation to SPI flash
  
• with support for storing the U-Boot Environment in SPI flash (configured to store it at offset 0x3f8000 bytes, 8KBytes size)
  
• can insert "partition" entries in the SPI Flash device tree for Linux so that /dev/mtdX devices are created for each area of the SPI flash layout it uses (if there is a spi-nor flash node in Linux's device tree for it to stick them on)
  
• with support for booting Linux off an NVMe device
  
• with support for booting Linux off a SATA drive attached to a PCIe SATA controller (AHCI)
  
• also supports the normal U-Boot features of booting from eMMC/SD/USB/Network
  

You can see the differences between current U-Boot master and this version here.

I have tested the SPI installation, SPI environment saving & loading, USB boot, network boot, SATA boot, and NVMe boot on my RockPro64 with a Samsung 970 EVO Plus NVMe device. I don't have any other NVMe devices to test with, and I'd like to verify this works for others, so if anyone else is interested, and would like to test this build, read on. In particular I'm uncertain if this works for other NVMe devices and possibly other SPI flash chips - my RockPro64 has a Gigadevice gd25q128 chip, but I've heard some other versions may use Winbond flash chips which I haven't been able to test with.

The build script (here) has several stages and builds several different binaries. The binaries attached to this release are:

• flash_spi.img.gz - this is an image which can be un-gzipped and written to an SD card, it starts U-Boot from the SD card and then runs this script to install U-Boot to SPI Flash. The image includes flash_spi.scr and spi_combined.img. More detailed info here.
  
• erase_spi.img.gz - this is an image similar to flash_spi.img.gz above, but instead it erases your SPI Flash using this script.
  
• flash_spi.scr - this is the compiled script to install U-Boot to SPI flash, it's not useful except for certain specific scenarios (if you want to make your own install SD-card). It is included in flash_spi.img.gz.
  
• mmc_idbloader.img - this is the image containing the TPL+SPL of U-Boot compiled for eMMC/SD card install, it should be written to eMMC or SD card at offset 64 sectors.
  
• mmc_u-boot.itb - this is the image containing U-Boot proper compiled for eMMC/SD card install, it should be written to eMMC or SD card at offset 16384 sectors.
  
• spi_idbloader.img - this is the image containing the TPL+SPL of U-Boot compiled for SPI Flash install, it should be written to SPI flash at offset 0, but for writing I recommend using spi_combined.img instead - see below.
  
• spi_u-boot.itb - this is the image containing U-Boot proper compiled for SPI Flash install, it should be written to SPI flash at offset 0x60000 bytes, but for writing I recommend using spi_combined.img instead - see below.
  
• spi_combined.img - this contains spi_idbloader.img, padding, and then spi_u-boot.itb at offset 0x60000 bytes, this can be written to SPI flash starting at offset 0 and will write spi_idbloader.img and spi_u-boot.itb at the correct locations.
  

For testing it - you'll need to have a serial / UART console available so that you can see log messages and interact with U-Boot that way - there is no HDMI or keyboard support in U-Boot.

Also, I strongly recommend *not* trying to flash it to SPI at first, since SPI is first in the RK3399 boot order, so if you flash something there and it doesn't work, your board is useless until you temporarily disable SPI to recover it.

Instead, first as a test just write mmc_idbloader.img and mmc_u-boot.itb to a spare SD card, remove / disable eMMC and SPI if you are currently booting off them, and test the NVMe boot functionality by booting off SD card, interrupting U-boot at the "Hit any key to stop autoboot" stage, and then entering: run bootcmd_nvme0 at the U-Boot => prompt to scan the NVMe device for bootable files and boot from it. On your NVMe drive, you just need something U-Boot can boot, e.g. a root partition with a /boot/extlinux/extlinux.conf or /boot/boot.scr telling it how to load the kernel, initrd and FDT file.

Then, if testing using a spare SD card works successfully, try installing to SPI, making sure you have a way to recover by disabling SPI temporarily if needed.

Some caveats: I've seen it mentioned that the PCIe driver for RK3399 from the Rock Pi 4 U-Boot is somewhat buggy, and looking at the code it seems like some PCIe features are missing or deliberately disabled; there is no support for more than one PCI device and no support for any other type of device apart from NVMe (so no PCI SATA adapter support at the moment, sorry). I don't know enough about the RK3399 PCIe controller to be able to find any more bugs from reading the code, but they might be there. Update: this is now using an updated PCIe driver from Patrick Wildt, based on OpenBSD's driver, and now does support PCI SATA adapters that implement the AHCI standard.
Also, if you're not sure what "write mmc_idbloader.img to eMMC or SD card at offset 64 sectors." means or how to do that, I recommend not trying this at all until some other folks have verified that it works, and it's packaged in an easier to use format.