Mainline U-Boot with SPI, NVMe and SATA boot support

[themyshop]

Hi,
I ported simply the latest Ayufan U-Boot Aug 26 2019 version (http://github.com/ayufan-rock64/linux-u-boot) to support of boot only from nvme (I added PCIe support from http://github.com/torvalds/linux/blob/ma...kpro64.dts and I enabled the NVMe and PCIe in defconfig: CONFIG_NVME, CONFIG_PCIE_ROCKCHIP).
Linux boot perfectly only from NVMe, but unfortunately Android not.
I tried already all Android version from http://wiki.pine64.org/index.php/ROCKPro...oid_Images. And I parted the SSD as SD card (same name, type and begin/size sectors of partitions, etc.). But nothing.

Code:

...
=> boot_android nvme 0:3
ANDROID: reboot reason: "(none)"
ANDROID: bootargs: "storagemedia=nvme androidboot.mode=normal androidboot.dtbo_idx=0 console=ttyFIQ0 androidboot.slot_suffix= androidboot.serialno=7b6f8551bce5b3b root=PARTUUID=af01642c-9b84-11e8-9b2a-234eb5e198a0 skip_initramfs"
FDT load addr 0x10f00000 size 286 KiB
ftd_addr: 22163456
Booting kernel at 0x280000 with fdt at 22163456...


## Booting Android Image at 0x00280000 ...
Kernel load addr 0x00280800 size 19081 KiB
Kernel command line: console=ttyFIQ0 androidboot.baseband=N/A androidboot.wificountrycode=US androidboot.veritymode=enforcing androidboot.hardware=rk30board androidboot.console=ttyFIQ0 firmware_class.path=/vendor/etc/firmware init=/init rootwait ro init=/init root=PARTUUID=af01642c-9b84-11e8-9b2a-234eb5e198a0 loop.max_part=7 androidboot.selinux=permissive
newbootargs: storagemedia=nvme androidboot.mode=normal androidboot.dtbo_idx=0 console=ttyFIQ0 androidboot.slot_suffix= androidboot.serialno=7b6f8551bce5b3b root=PARTUUID=af01642c-9b84-11e8-9b2a-234eb5e198a0 skip_initramfs console=ttyFIQ0 androidboot.baseband=N/A androidboot.wificountrycode=US androidboot.veritymode=enforcing androidboot.hardware=rk30board androidboot.console=ttyFIQ0 firmware_class.path=/vendor/etc/firmware init=/init rootwait ro init=/init root=PARTUUID=af01642c-9b84-11e8-9b2a-234eb5e198a0 loop.max_part=7 androidboot.selinux=permissive
"Synchronous Abort" handler, esr 0x96000021
ELR:     f7f59ad0
LR:      f7f59acc
x0 : 0000000000000527 x1 : 00000000ff1a0000
x2 : 000000000000000a x3 : 00000000ff1a0000
x4 : 00000000f5f2ae40 x5 : 00000000f7f8eced
x6 : 0000000022163456 x7 : 0000000000000008
x8 : 00000000f5f2b240 x9 : 0000000000000008
x10: 00000000f5f2aebe x11: 00000000ffffffff
x12: 00000000ffffffff x13: 0000000000000200
x14: 000000000000000e x15: 00000000ffffffff
x16: 0000000000001080 x17: 00000000204e0120
x18: 00000000f5f37df8 x19: 0000000022163456
x20: 00000000f7fb2d80 x21: 00000000f5f4b1c0
x22: 00000000f7fb2e80 x23: 00000000f7fb2e88
x24: 0000000022163456 x25: 0000000000000016
x26: 0000000000000000 x27: 00000000f7fb2000
x28: 00000000f5f43370 x29: 0000000000200090

Resetting CPU ...

Moreover, the Android doesn't start from the SD card with the original latest Ayufan U-Boot 2017.09-rockchip-ayufan-1065-g95f6152134 (Aug 26 2019 - 12:41:34 +0000) (http://github.com/ayufan-rock64/linux-u-...95f6152134) either!

Code:

...
Hit key to stop autoboot('CTRL+C'):  0
ANDROID: reboot reason: "(none)"
SecureBoot enabled, AVB verify => CONFIG_ANDROID_AVB=y?
read_is_device_unlocked() ops returned that device is UNLOCKED
avb_slot_verify.c:638: ERROR: vbmeta: Error verifying vbmeta image: OK_NOT_SIGNED
TTT1 cmdline=root=PARTUUID=$(ANDROID_SYSTEM_PARTUUID)
FDT load addr 0x10f00000 size 286 KiB
Booting IMAGE kernel at 0x00280000 with fdt at 0x8300000...


## Booting Android Image at 0x0027f800 ...
Kernel load addr 0x00280000 size 19081 KiB
## Flattened Device Tree blob at 08300000
   Booting using the fdt blob at 0x8300000
   XIP Kernel Image ... OK
TTT3 commandline=storagemedia=sd androidboot.mode=normal androidboot.dtbo_idx=0 root=PARTUUID=af01642c-9b84-11e8-9b2a-234eb5e198a0 androidboot.verifiedbootstate=orange androidboot.slot_suffix= androidboot.serialno=7b6f8551bce5b3b console=ttyFIQ0 androidboot.baseband=N/A androidboot.wificountrycode=US androidboot.veritymode=enforcing androidboot.hardware=rk30board androidboot.console=ttyFIQ0 firmware_class.path=/vendor/etc/firmware init=/init rootwait ro init=/init root=PARTUUID=af01642c-9b84-11e8-9b2a-234eb5e198a0 loop.max_part=7 androidboot.selinux=permissive
   Loading Device Tree to 00000000e9db9000, end 00000000e9dd4727 ... OK
Adding bank: 0x00200000 - 0x08400000 (size: 0x08200000)
Adding bank: 0x0a200000 - 0xf8000000 (size: 0xede00000)
Total: 7012.817 ms

Starting kernel ...


[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Initializing cgroup subsys schedtune
[    0.000000] Linux version 4.4.167 (tugy@thzy-server) (gcc version 6.3.1 20170404 (Linaro GCC 6.3-2017.05) ) #22 SMP PREEMPT Mon May 6 09:37:48 CST 2019
[    0.000000] Boot CPU: AArch64 Processor [410fd034]
[    0.000000] earlycon: Early serial console at MMIO32 0xff1a0000 (options '')
[    0.000000] bootconsole [uart0] enabled
[    0.000000] Reserved memory: failed to reserve memory for node 'stb-devinfo@00000000': base 0x0000000000000000, size 0 MiB

Maybe does anyone have an bootable Android version? Or compiled Android with PCIe and NVME support? Thank you for your help!
Or possible boot the Android from extlinux?

My suggestions yet for the NVME port:

1. increase the SPL size (prevent of .sram size error in compile) in include\configs\rk3399_common.h:

Code:

//#define CONFIG_SPL_MAX_SIZE             0x10000
#define CONFIG_SPL_MAX_SIZE             0x20000
//#define CONFIG_SPL_BSS_MAX_SIZE         0x20000
#define CONFIG_SPL_BSS_MAX_SIZE         0x40000

2. I needed to decrease (1 MB => 512 kB) the Maximum Data Transfer Size (MDTS) for my Samsung 970 PRO M.2 512GB NVMe SSD in drivers\nvme\nvme.c:

Code:

static int nvme_get_info_from_identify(struct nvme_dev *dev)
    ...
        dev->max_transfer_shift = 20;
    }

//My Hack[20190905]: max 512 kB wihout error...
    dev->max_transfer_shift = 19;
    printf("nvme_get_info_from_identify()-max_transfer_shift: %u\n", dev->max_transfer_shift)

[belfastraven]

...
=> run bootcmd_nvme0

Device 0: Vendor: 0x144d Rev: EXD7101Q Prod: S444NE0K603440      
           Type: Hard Disk
           Capacity: 244198.3 MB = 238.4 GB (500118192 x 512)
... is now current device
** Unrecognized filesystem type **

JI'm just including the log this time so that you can see that I did use the proper images....

Once again, thanks.

@belfastraven thanks for testing, and pasting the full log here. Looks like your NVMe device is detected OK, I suspect the "Unrecognized filesystem type" problem is just because there is no BTRFS support in this build (it's not enabled in U-Boot by default).

I enabled BTRFS in the config and re-built U-Boot, I haven't tested this yet but you could try using the resulting build, attached: 
Edit: if you need the SPI binaries from this build, they can be downloaded from this build artifacts page.

I now have armbian booting from SPI flash u-boot,  its great to be booting from SPI running 5.4.8 kernel with current u-boot.  I have booted sdcard, emmc and pcie NVME.  I haven't tried with USB ports yet.  I had gone back to ext4. 

Have you thought of including an image like your flash image, that would just 0-out the flash in case of problems.  I believe ayufan has one https://github.com/ayufan-rock64/linux-u-boot/releases/download/2017.09-rockchip-ayufan-1065-g95f6152134/u-boot-erase-spi-rockpro64.img.xz  

thanks again.

@Bullet64,@sigmaris

@Bullet64,@sigmaris,Running the same self-built armbian image (5.4.8 kernel, ubuntu bionic) booting from SPI flash,
  I can boot sd, emmc, nvme,
USB3 sandisk extreme stick on esb2 ports,  but not on usb3 ports,
Samsung t1 portable SSD USb3 on USB3 ports,
and an old usb2 datatraveler stick on the usb3 ports.

  I don't understand why the usb3 stick won't boot from the USB3 port,  and have a feeling it might be a timing issue,  since the system will boot from slower devices on that port...I'm not good at hardware debugging,  but perhaps one of you will have an idea?   The way the "not booting" manifests itself is that I will get the "Starting kernel" message,  and then nothing...   which makes it hard to debug.
Sorry I had forgotten to turn the kernel log level up-- with log level at 7, I can now see that the one USB3 stick that wouldn't boot on the USB 3 port was not being recognized as a USB3 device.

usb 7-1: new full-speed USB device number 2 using xhci-hcd
[ 4.040775] usb 7-1: device descriptor read/64, error -71
[ 4.276738] usb 7-1: device descriptor read/64, error -71
[ 4.512670] usb 7-1: new full-speed USB device number 3 using xhci-hcd
[ 4.640722] usb 7-1: device descriptor read/64, error -71
[ 4.876677] usb 7-1: device descriptor read/64, error -71
[ 4.984926] usb usb7-port1: attempt power cycle
[ 5.636665] usb 7-1: new full-speed USB device number 4 using xhci-hcd
[ 5.637366] usb 7-1: Device not responding to setup address.
[ 5.844773] usb 7-1: Device not responding to setup address.
[ 6.052665] usb 7-1: device not accepting address 4, error -71
[ 6.180668] usb 7-1: new full-speed USB device number 5 using xhci-hcd
[ 6.181370] usb 7-1: Device not responding to setup address.
[ 6.388747] usb 7-1: Device not responding to setup address.
[ 6.596665] usb 7-1: device not accepting address 5, error -71
[ 6.597479] usb usb7-port1: unable to enumerate USB device
[ 10.364793] phy phy-ff770000.syscon:usb2-phy@e450.2: charger = USB_DCP_CHARGER

whereas a different USB3 stick was properly recognize and booted:

3.602830] hub 8-0:1.0: 1 port detected
[ 4.692711] usb 8-1: new SuperSpeed Gen 1 USB device number 2 using xhci-hcd
[ 4.713725] usb 8-1: New USB device found, idVendor=0781, idProduct=558a, bcdDevice= 1.00
[ 4.714490] usb 8-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 4.715152] usb 8-1: Product: Ultra
[ 4.715488] usb 8-1: Manufacturer: SanDisk
[ 4.715879] usb 8-1: SerialNumber: 4C530001241105113443
[ 4.718156] usb-storage 8-1:1.0: USB Mass Storage device detected
[ 4.719355] scsi host0: usb-storage 8-1:1.0
[ 4.725283] usbcore: registered new interface driver uas
[ 5.729461] scsi 0:0:0:0: Direct-Access SanDisk Ultra 1.00 PQ: 0 ANSI: 6
[ 5.731615] sd 0:0:0:0: [sda] 240254976 512-byte logical blocks: (123 GB/115 GiB)
[ 5.733312] sd 0:0:0:0: [sda] Write Protect is off
[ 5.734191] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 5.775435] sda: sda1
[ 5.778568] sd 0:0:0:0: [sda] Attached SCSI removable disk
[ 5.974332] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)


Probably there was something wrong with the stick tisellf.....

So all seems good here.

[sigmaris]

On the topic of USB boot, I've managed to get a USB-3 Samsung M3 Portable hard drive and done some testing.

With this mainline U-Boot build:
- USB 2.0 flash drive SanDisk Cruzer Blade

• plugged into USB-2 port: it's recognised by U-Boot and can be used
  
• plugged into USB-3 port: it's recognised by U-Boot and can be used
  
• plugged into USB-C port with adapter: it's not detected
  

- USB 3.0 HDD Samsung M3 Portable:

• plugged into USB-2 port: it's recognised by U-Boot and can be used
  
• plugged into USB-3 port: it's not detected
  
• plugged into USB-C port with adapter: it's not detected
  

With U-Boot 2017.09-rockchip-ayufan-1065-g95f6152134:
- USB 2.0 flash drive SanDisk Cruzer Blade

• plugged into USB-2 port: it's recognised by U-Boot
  
• plugged into USB-3 port: it's recognised by U-Boot
  
• plugged into USB-C port with adapter: it's not detected
  

- USB 3.0 HDD Samsung M3 Portable:

• plugged into USB-2 port: it's not detected
  
• plugged into USB-3 port: it's not detected
  
• plugged into USB-C port: it's not detected
  

I'd guess in mainline U-Boot there is support for USB 2.0, but not full USB 3.0 or USB Type-C support. This is backed up by this comment on the original submission of the USB support from Rockchip staff: https://lists.denx.de/pipermail/u-boot/2...64557.html

Only support usb2.0 currently for we have not enable the usb3.0 phy driver and PD(fusb302) driver.

So USB 3.0 devices will probably only work if they are forced to work in USB 2.0 mode, e.g. by being plugged into the USB 2.0 ports.

[sigmaris]

I have updated the OP with info about a new release, based on mainline U-Boot v2020.01, and now with PCIe SATA drive boot support as well as NVMe.

There are also patches on the U-Boot mailing list which get HDMI output working from U-Boot. I've tested this and it works as expected, but I've not included it in the current build because

• Using HDMI output from U-Boot seems to break HDMI output from mainline Linux until a Linux app runs that changes the video mode, which resets HDMI and gets it working again.
  
• to be properly useful to interact with U-Boot, it'd require a USB keyboard, but starting U-Boot's USB support (required before any keyboard is recognised) takes several seconds, and it doesn't seem worthwhile to slow down every boot with the "usb start" command when normally it's not required.
  

In any case, if anyone is interested, you can see the changes required for HDMI and USB keyboard support in this branch.

[Bullet64]

I have updated the OP with info about a new release, based on mainline U-Boot v2020.01, and now with PCIe SATA drive boot support as well as NVMe.

There are also patches on the U-Boot mailing list which get HDMI output working from U-Boot. I've tested this and it works as expected, but I've not included it in the current build because

• Using HDMI output from U-Boot seems to break HDMI output from mainline Linux until a Linux app runs that changes the video mode, which resets HDMI and gets it working again.
  
• to be properly useful to interact with U-Boot, it'd require a USB keyboard, but starting U-Boot's USB support (required before any keyboard is recognised) takes several seconds, and it doesn't seem worthwhile to slow down every boot with the "usb start" command when normally it's not required.
  

In any case, if anyone is interested, you can see the changes required for HDMI and USB keyboard support in this branch.

Will test this later. Thank you!

Edit: sata boot works as expected

[belfastraven]

I have updated the OP with info about a new release, based on mainline U-Boot v2020.01, and now with PCIe SATA drive boot support as well as NVMe.

There are also patches on the U-Boot mailing list which get HDMI output working from U-Boot. I've tested this and it works as expected, but I've not included it in the current build because

• Using HDMI output from U-Boot seems to break HDMI output from mainline Linux until a Linux app runs that changes the video mode, which resets HDMI and gets it working again.
  
• to be properly useful to interact with U-Boot, it'd require a USB keyboard, but starting U-Boot's USB support (required before any keyboard is recognised) takes several seconds, and it doesn't seem worthwhile to slow down every boot with the "usb start" command when normally it's not required.
  

In any case, if anyone is interested, you can see the changes required for HDMI and USB keyboard support in this branch.

Will test this later. Thank you!

Edit: sata boot works as expected

Everything that I tested before still works.  I am able to boot a samsung t1 on USB3 (not C) (that's an SSD,  but I thought it was SATA,  I'll need to check the specs),  but I did before.  Thanks for the "erase" image. 

People have started to build mainline u-boot for the Pinebook-pro,  I'd like to try with a more robust dts than is being used (something based more on the manjaro one).  People are going through all kins of trouble writing to spi_flash with rkpdeveloptools to write the current mainline,  but I'd love to be able to provide them with a bootable flashing image like yours.  Sometimes the connections with rkdevelptools are a bit problematic.
The one thing I don't understand (I'm a linux newbie) is how to write the flashable image to get the correct "padding" between  the idbloader.image  and u-boot.itb. Is it just a dd command?   thnx

Also, Is it possible to just (through the serial console)  stop u-boot, probe, load the spi image (from whatever you are running u-boot), and sf update?  I guess I don't understand why one would have to disable the other devices to do that.  OF course once the flash was written, you'd have to stop u-boot and reboot, I guess.  Just curious...

[sigmaris]

People have started to build mainline u-boot for the Pinebook-pro,  I'd like to try with a more robust dts than is being used (something based more on the manjaro one).  People are going through all kins of trouble writing to spi_flash with rkpdeveloptools to write the current mainline,  but I'd love to be able to provide them with a bootable flashing image like yours.  Sometimes the connections with rkdevelptools are a bit problematic.
The one thing I don't understand (I'm a linux newbie) is how to write the flashable image to get the correct "padding" between  the idbloader.image  and u-boot.itb. Is it just a dd command?   thnx

It would be cool to see a similar mainline U-Boot project for Pinebook Pro. I don't have a PBP, otherwise I would work on it, but if someone who has a PBP can take my work and adapt it that'd be great.

If you're asking about how to create spi_combined.img, it can be done using a dd command. On RK3399 the idbloader.img is always read first from a fixed location, SPI offset 0, by the boot ROM. Then, u-boot.itb is loaded by the code inside idbloader.img, from a configurable offset, set at U-Boot build time using CONFIG_SYS_SPI_U_BOOT_OFFS - you can see in that linked file it's set to 0x60000 for this build.
So, assuming idbloader.img is less than 0x60000 bytes long, we want to include padding after it so that u-boot.itb starts at 0x60000. This is done here in the build script, in particular this line:

Code:

dd if=/dev/zero of=$(img1name) conv=notrunc bs=1 count=1 seek=$padsize

pads the file $(img1name) to $padsize+1 (by seeking to offset $padsize and writing a single zero byte). $padsize is set to (0x60000 - 1) a few lines above. Then the u-boot.itb can just be concatenated on the end, using the cat command, to make spi_combined.img.

Then the flash script just loads that image to the address where a kernel would normally be loaded (${kernel_addr_r}) and then uses the "sf update" command to write it to SPI flash.

Also, Is it possible to just (through the serial console)  stop u-boot, probe, load the spi image (from whatever you are running u-boot), and sf update?  I guess I don't understand why one would have to disable the other devices to do that.  OF course once the flash was written, you'd have to stop u-boot and reboot, I guess.  Just curious...

Yes, it is possible to boot from any medium, stop autoboot, enable the SPI flash (if disabled) then run the commands in the flash script at the U-boot prompt to achieve the same effect as booting from a SD-card containing the flash_spi.img. Omitting the "run blink_work" commands, of course, since they just blink the white LED.
The only reason to need to disable SPI to do this is if somehow the code currently in your SPI won't boot up correctly as far as U-Boot proper, and you need to stop the Boot ROM trying to load it at all, to be able to do this.

[belfastraven]

People have started to build mainline u-boot for the Pinebook-pro,  I'd like to try with a more robust dts than is being used (something based more on the manjaro one).  People are going through all kins of trouble writing to spi_flash with rkpdeveloptools to write the current mainline,  but I'd love to be able to provide them with a bootable flashing image like yours.  Sometimes the connections with rkdevelptools are a bit problematic.
The one thing I don't understand (I'm a linux newbie) is how to write the flashable image to get the correct "padding" between  the idbloader.image  and u-boot.itb. Is it just a dd command?   thnx

It would be cool to see a similar mainline U-Boot project for Pinebook Pro. I don't have a PBP, otherwise I would work on it, but if someone who has a PBP can take my work and adapt it that'd be great.

If you're asking about how to create spi_combined.img, it can be done using a dd command. On RK3399 the idbloader.img is always read first from a fixed location, SPI offset 0, by the boot ROM. Then, u-boot.itb is loaded by the code inside idbloader.img, from a configurable offset, set at U-Boot build time using CONFIG_SYS_SPI_U_BOOT_OFFS - you can see in that linked file it's set to 0x60000 for this build.
So, assuming idbloader.img is less than 0x60000 bytes long, we want to include padding after it so that u-boot.itb starts at 0x60000. This is done here in the build script, in particular this line:

Code:

dd if=/dev/zero of=$(img1name) conv=notrunc bs=1 count=1 seek=$padsize

pads the file $(img1name) to $padsize+1 (by seeking to offset $padsize and writing a single zero byte). $padsize is set to (0x60000 - 1) a few lines above. Then the u-boot.itb can just be concatenated on the end, using the cat command, to make spi_combined.img.

Then the flash script just loads that image to the address where a kernel would normally be loaded (${kernel_addr_r}) and then uses the "sf update" command to write it to SPI flash.

Also, Is it possible to just (through the serial console)  stop u-boot, probe, load the spi image (from whatever you are running u-boot), and sf update?  I guess I don't understand why one would have to disable the other devices to do that.  OF course once the flash was written, you'd have to stop u-boot and reboot, I guess.  Just curious...

Yes, it is possible to boot from any medium, stop autoboot, enable the SPI flash (if disabled) then run the commands in the flash script at the U-boot prompt to achieve the same effect as booting from a SD-card containing the flash_spi.img. Omitting the "run blink_work" commands, of course, since they just blink the white LED.
The only reason to need to disable SPI to do this is if somehow the code currently in your SPI won't boot up correctly as far as U-Boot proper, and you need to stop the Boot ROM trying to load it at all, to be able to do this.

Thank you for your help/explanation.    I am going to try to implement this,  but I believe that Bluerise (Patrick Wildt) whose pcie driver you are now using , might have a PinebookPro, too, and if he tries,  it will probably happen faster :-)

[Thovthe]

Looking forward to seeing this come to the PBP

[hunderteins]

Wow! Thank you for your work. My RockPro64 boots directly from SPI into "scsi" without mmc. Well done!

I have two "problems" identified on v2020.01ci:

(1) reboot from linux will not longer work. The machine just stands. Kernel 4.4
(2) is this huge (16384 sectors) offset on mmc-u_boot.itb really necessary? mmc_idbloader.img is about 148kB. Do you wanted to go for 0x80000 (1024 sectors) and got 0x800000?

Keep on hacking,
hunderteins