Mainline U-Boot with SPI, NVMe and SATA boot support
#31
I can't believe I've only just stumbled on this thread. Amazing work getting this together. Seriously, congrats.

I've flashed this is an SD card as per the instructions, but I've run into issues. I have a Marvell 88SE9230 4-port SATA card, and my disk partition layouts look like this:

Disk 1 - 128GB Kingston SSD (GPT)
  • Partition 1 is by boot partition - 260MB FAT32 set as EFI in the partition table
  • Partition 2 is one half of my a RAID 1 array - md0 - which has LVM on top. My RootFS is an LVM volume in this.

Disk 2 - 128GB Kingston SSD (GPT)
  • Partition 1 is an empty 128MB partition
  • Partition 2 is the other half of my md0 RAID 1 array

Disk 3 - 3TB HDD (GPT)
  • Partition 1 is one half of another RAID 1 array - md1 - with LVM on top for data storage
Disk 4 - 3TB HDD (GPT)
  • Partition 1 is the half of another RAID 1 array - md1
Normally I boot this off an SD card. My boot partition resides on the SD card, and the initrd has the modules it needs to setup RAID and LVM to allow me to boot form the SSDs.

I can boot into your U-Boot no problem. If I run the bootcmd_scsi0 script, I get this output:
Code:
run bootcmd_scsi0                                                            
PCIE-0: Link up (Bus0)                                                          
scanning bus for devices...                                                    
controller reset failed (0x1)                                                  
                                                                               
Device 0: unknown device

This can be mitigated by putting a sleep of a few seconds between `pci enum` and `scsi reset`. However, `scsi reset` still outputs an error at the end:
Code:
                                                                               
Reset SCSI                                                                      
scanning bus for devices...                                                    
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
AHCI 0001.0200 32 slots 8 ports 6 Gbps 0xff impl SATA mode                      
flags: 64bit ncq pio                                                            
No Link on this port!                                                          
Can not start port 5                                                            
No Link on this port!                                                          
Can not start port 6                                                            
 Device 0: (0:0) Vendor: ATA Prod.: KINGSTON SA400S3 Rev: SBFK                
           Type: Hard Disk                                                    
           Capacity: 114473.4 MB = 111.7 GB (234441648 x 512)                  
 Device 1: (1:0) Vendor: ATA Prod.: KINGSTON SA400S3 Rev: SBFK                
           Type: Hard Disk                                                    
           Capacity: 114473.4 MB = 111.7 GB (234441648 x 512)                  
 Device 2: (2:0) Vendor: ATA Prod.: ST3000DM007-1WY1 Rev: 0001                
           Type: Hard Disk                                                    
           Capacity: 2861588.4 MB = 2794.5 GB (5860533168 x 512)              
 Device 3: (3:0) Vendor: ATA Prod.: WDC WD30EURX-63T Rev: 80.0                
           Type: Hard Disk                                                    
           Capacity: 2861588.4 MB = 2794.5 GB (5860533168 x 512)              
timeout exit!  

Nevertheless, running `run bootcmd_scsi0` at this point almost works:

Code:
=> run bootcmd_scsi0

Device 0: (0:0) Vendor: ATA Prod.: KINGSTON SA400S3 Rev: SBFK
           Type: Hard Disk
           Capacity: 114473.4 MB = 111.7 GB (234441648 x 512)
... is now current device
Scanning scsi 0:1...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
2089 bytes read in 15441 ms (0 Bytes/s)
select kernel
1:      kernel-4.4.202-1237-rockchip-ayufan-gfd4492386213
2:      kernel-4.4.202-1237-rockchip-ayufan-gfd4492386213-memtest
3:      kernel-4.4.132-1075-rockchip-ayufan-ga83beded8524
4:      kernel-4.4.132-1075-rockchip-ayufan-ga83beded8524-memtest
Enter choice: 1:        kernel-4.4.202-1237-rockchip-ayufan-gfd4492386213
Retrieving file: /boot/initrd.img-4.4.202-1237-rockchip-ayufan-gfd4492386213
Skipping kernel-4.4.202-1237-rockchip-ayufan-gfd4492386213 for failure retrieving initrd
2:      kernel-4.4.202-1237-rockchip-ayufan-gfd4492386213-memtest
Retrieving file: /boot/initrd.img-4.4.202-1237-rockchip-ayufan-gfd4492386213
Skipping kernel-4.4.202-1237-rockchip-ayufan-gfd4492386213-memtest for failure retrieving initrd
3:      kernel-4.4.132-1075-rockchip-ayufan-ga83beded8524
Retrieving file: /boot/initrd.img-4.4.132-1075-rockchip-ayufan-ga83beded8524
Skipping kernel-4.4.132-1075-rockchip-ayufan-ga83beded8524 for failure retrieving initrd
4:      kernel-4.4.132-1075-rockchip-ayufan-ga83beded8524-memtest
Retrieving file: /boot/initrd.img-4.4.132-1075-rockchip-ayufan-ga83beded8524
Skipping kernel-4.4.132-1075-rockchip-ayufan-ga83beded8524-memtest for failure retrieving initrd
SCRIPT FAILED: continuing...

As you can see, it's dead slow (2kB in 15 seconds) and never fully manages to fetch the initrd from the disk. Is there any troubleshooting I can do? I've worked a little on U-Boot source in the past, plus I have an incentive to get this working, so if there is anything I can do to help fix something like this I am more than happy to get involved.
  Reply
#32
@edrose There's been some patches posted recently on the U-Boot mailing list with a revised RK3399 PCIe driver, intended for U-Boot 2020.07, and some discussion of a need to delay probing PCIe cards (to allow the power supply to stabilise, or something like that). I'll try and make a build of the latest mainline code and the new PCIe driver patches this weekend to try out.
  Reply
#33
@edrose Here are a couple of test builds, based on mainline U-Boot including the mainline RK3399 PCIe driver, to try with your card:

Firstly, this build (based on this branch) adds 100ms delay before attempting link training - based on this patch for the Rockchip PCIe driver in Linux.

Secondly, this build (based on this branch) incorporates the 100ms pre-training delay, and also 2s (!) delay after training before reading PCIe config space, using this patch from the mailing list.

Note neither of these contain the patches to set up the SPI flash partitions in the device tree for Linux, but that shouldn't matter if you are booting it from MMC.

I'd be interested to hear if either of them helps with your issue.
  Reply
#34
(06-06-2020, 11:28 AM)sigmaris Wrote: @edrose Here are a couple of test builds, based on mainline U-Boot including the mainline RK3399 PCIe driver, to try with your card:

Firstly, this build (based on this branch) adds 100ms delay before attempting link training - based on this patch for the Rockchip PCIe driver in Linux.

Secondly, this build (based on this branch) incorporates the 100ms pre-training delay, and also 2s (!) delay after training before reading PCIe config space, using this patch from the mailing list.

Note neither of these contain the patches to set up the SPI flash partitions in the device tree for Linux, but that shouldn't matter if you are booting it from MMC.

I'd be interested to hear if either of them helps with your issue.
Thanks for getting back to me!

I tried both of those builds over the weekend. Unfortunately I had the same issues with both of them. Exactly the same behaviour.

I wonder whether I have a bad SATA card or something.
  Reply
#35
(05-28-2020, 03:54 PM)edrose Wrote: I can't believe I've only just stumbled on this thread. Amazing work getting this together. Seriously, congrats.

I've flashed this is an SD card as per the instructions, but I've run into issues. I have a Marvell 88SE9230 4-port SATA card, and my disk partition layouts look like this:

Disk 1 - 128GB Kingston SSD (GPT)
  • Partition 1 is by boot partition - 260MB FAT32 set as EFI in the partition table
  • Partition 2 is one half of my a RAID 1 array - md0 - which has LVM on top. My RootFS is an LVM volume in this.

Disk 2 - 128GB Kingston SSD (GPT)
  • Partition 1 is an empty 128MB partition
  • Partition 2 is the other half of my md0 RAID 1 array

Disk 3 - 3TB HDD (GPT)
  • Partition 1 is one half of another RAID 1 array - md1 - with LVM on top for data storage
Disk 4 - 3TB HDD (GPT)
  • Partition 1 is the half of another RAID 1 array - md1
Normally I boot this off an SD card. My boot partition resides on the SD card, and the initrd has the modules it needs to setup RAID and LVM to allow me to boot form the SSDs.

I can boot into your U-Boot no problem. If I run the bootcmd_scsi0 script, I get this output:
Code:
run bootcmd_scsi0                                                            
PCIE-0: Link up (Bus0)                                                          
scanning bus for devices...                                                    
controller reset failed (0x1)                                                  
                                                                               
Device 0: unknown device

This can be mitigated by putting a sleep of a few seconds between `pci enum` and `scsi reset`. However, `scsi reset` still outputs an error at the end:
Code:
                                                                               
Reset SCSI                                                                      
scanning bus for devices...                                                    
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
Target spinup took 0 ms.                                                        
AHCI 0001.0200 32 slots 8 ports 6 Gbps 0xff impl SATA mode                      
flags: 64bit ncq pio                                                            
No Link on this port!                                                          
Can not start port 5                                                            
No Link on this port!                                                          
Can not start port 6                                                            
 Device 0: (0:0) Vendor: ATA Prod.: KINGSTON SA400S3 Rev: SBFK                
           Type: Hard Disk                                                    
           Capacity: 114473.4 MB = 111.7 GB (234441648 x 512)                  
 Device 1: (1:0) Vendor: ATA Prod.: KINGSTON SA400S3 Rev: SBFK                
           Type: Hard Disk                                                    
           Capacity: 114473.4 MB = 111.7 GB (234441648 x 512)                  
 Device 2: (2:0) Vendor: ATA Prod.: ST3000DM007-1WY1 Rev: 0001                
           Type: Hard Disk                                                    
           Capacity: 2861588.4 MB = 2794.5 GB (5860533168 x 512)              
 Device 3: (3:0) Vendor: ATA Prod.: WDC WD30EURX-63T Rev: 80.0                
           Type: Hard Disk                                                    
           Capacity: 2861588.4 MB = 2794.5 GB (5860533168 x 512)              
timeout exit!  

Nevertheless, running `run bootcmd_scsi0` at this point almost works:

Code:
=> run bootcmd_scsi0

Device 0: (0:0) Vendor: ATA Prod.: KINGSTON SA400S3 Rev: SBFK
           Type: Hard Disk
           Capacity: 114473.4 MB = 111.7 GB (234441648 x 512)
... is now current device
Scanning scsi 0:1...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
2089 bytes read in 15441 ms (0 Bytes/s)
select kernel
1:      kernel-4.4.202-1237-rockchip-ayufan-gfd4492386213
2:      kernel-4.4.202-1237-rockchip-ayufan-gfd4492386213-memtest
3:      kernel-4.4.132-1075-rockchip-ayufan-ga83beded8524
4:      kernel-4.4.132-1075-rockchip-ayufan-ga83beded8524-memtest
Enter choice: 1:        kernel-4.4.202-1237-rockchip-ayufan-gfd4492386213
Retrieving file: /boot/initrd.img-4.4.202-1237-rockchip-ayufan-gfd4492386213
Skipping kernel-4.4.202-1237-rockchip-ayufan-gfd4492386213 for failure retrieving initrd
2:      kernel-4.4.202-1237-rockchip-ayufan-gfd4492386213-memtest
Retrieving file: /boot/initrd.img-4.4.202-1237-rockchip-ayufan-gfd4492386213
Skipping kernel-4.4.202-1237-rockchip-ayufan-gfd4492386213-memtest for failure retrieving initrd
3:      kernel-4.4.132-1075-rockchip-ayufan-ga83beded8524
Retrieving file: /boot/initrd.img-4.4.132-1075-rockchip-ayufan-ga83beded8524
Skipping kernel-4.4.132-1075-rockchip-ayufan-ga83beded8524 for failure retrieving initrd
4:      kernel-4.4.132-1075-rockchip-ayufan-ga83beded8524-memtest
Retrieving file: /boot/initrd.img-4.4.132-1075-rockchip-ayufan-ga83beded8524
Skipping kernel-4.4.132-1075-rockchip-ayufan-ga83beded8524-memtest for failure retrieving initrd
SCRIPT FAILED: continuing...

As you can see, it's dead slow (2kB in 15 seconds) and never fully manages to fetch the initrd from the disk. Is there any troubleshooting I can do? I've worked a little on U-Boot source in the past, plus I have an incentive to get this working, so if there is anything I can do to help fix something like this I am more than happy to get involved.





hi @edrose I am using the similar setup with same marvel pcie card. Did you manage to circumvent it and how does it work. For me I would like to boot from SSD instead of SD karte. I am running OMV and my kernal info :

Linux rockpro64 4.4.167-1213-rockchip-ayufan-g34ae07687fce #1 SMP Tue Jun 18 20:44:49 UTC 2019 aarch64 GNU/Linux

I have followed the artikel https://forum.frank-mankel.org/topic/208...vme-platte to boot from SSD.

But unfortunate it doesn't work.

and I am not sure how booting from u-boot-spi works. since I have to manually enable marvel Pci-e driver from kernal pci=nomsi and marvel-99.rules

If I keep SD card as bootler and boot from SSD is totally fine for me. But so far I could not able to achieve this.

Any idea.



specs:
OS: OMV5

2 -SSD: RAID 1 setup

2 partitions done using LVM (OS, Configs)

I would like to boot the OS from this OS partition which is in RAID 1 setup.



Thanks
  Reply
#36
hi guysI am using the similar setup with same marvel pcie card. Did you manage to circumvent it and how does it work. For me I would like to boot from SSD instead of SD karte. I am running OMV and my kernal info :

Linux rockpro64 4.4.167-1213-rockchip-ayufan-g34ae07687fce #1 SMP Tue Jun 18 20:44:49 UTC 2019 aarch64 GNU/Linux

I have followed the artikel https://forum.frank-mankel.org/topic/208...vme-platte to boot from SSD.

But unfortunate it doesn't work.

and I am not sure how booting from u-boot-spi works. since I have to manually enable marvel Pci-e driver from kernal pci=nomsi and marvel-99.rules

If I keep SD card as bootler and boot from SSD is totally fine for me. But so far I could not able to achieve this.

Any idea.



specs:
OS: OMV5

2 -SSD: RAID 1 setup

2 partitions done using LVM (OS, Configs)

I would like to boot the OS from this OS partition which is in RAID 1 setup.



Thanks
  Reply
#37
Hello, apologies for the noob question and also if this is already answered somewhere and I haven't managed to find it, but I have managed to build the U-boot loader and load it onto an SD card.  I can see that when running from the SD card on the rockPro64 that the NVMe drive is correctly identified and that the /boot/boot.scr is read and loaded and things seem to start off nicely.  Then there is a "Device nor0 not found!" message (not sure how significant this is) and then the boot just hangs at the "Starting kernel ..." stage, see below:

    U-Boot 2020.01-01987-g335875f1f5 (Jul 04 2020 - 16:09:43 +0100)

    Model: Pine64 RockPro64
    DRAM:  3.9 GiB
    PMIC:  RK808
    MMC:  dwmmc@fe320000: 1, sdhci@fe330000: 0
    Loading Environment from MMC... Card did not respond to voltage select!
    *** Warning - No block device, using default environment

    In:    serial@ff1a0000
    Out:  serial@ff1a0000
    Err:  serial@ff1a0000
    Model: Pine64 RockPro64
    Net:  eth0: ethernet@fe300000
    Hit any key to stop autoboot:  0
    Card did not respond to voltage select!
    switch to partitions #0, OK
    mmc1 is current device
    Scanning mmc 1:1...
    Scanning mmc 1:2...
    PCIE-0: Link up (Bus0)

    Device 0: Vendor: 0x2646 Rev: S5Z42105 Prod: 50026B7282510DDF   
                Type: Hard Disk
                Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
    ... is now current device
    Scanning nvme 0:1...
    Found U-Boot script /boot/boot.scr
    2940 bytes read in 3 ms (957 KiB/s)
    ## Executing script at 00500000
    Boot script loaded from nvme 0
    151 bytes read in 3 ms (48.8 KiB/s)
    14715101 bytes read in 34 ms (412.7 MiB/s)
    21914112 bytes read in 49 ms (426.5 MiB/s)
    72702 bytes read in 4 ms (17.3 MiB/s)
    2698 bytes read in 4 ms (658.2 KiB/s)
    Applying kernel provided DT fixup script (rockchip-fixup.scr)
    ## Executing script at 39000000
    ## Loading init Ramdisk from Legacy Image at 06000000 ...
      Image Name:  uInitrd
      Image Type:  AArch64 Linux RAMDisk Image (gzip compressed)
      Data Size:    14715037 Bytes = 14 MiB
      Load Address: 00000000
      Entry Point:  00000000
      Verifying Checksum ... OK
    ## Flattened Device Tree blob at 01f00000
      Booting using the fdt blob at 0x1f00000
      Loading Ramdisk to f50f4000, end f5efc89d ... OK
      Loading Device Tree to 00000000f5079000, end 00000000f50f3fff ... OK
    Device nor0 not found!
 
    Starting kernel ...


At this time other than the power LED there are no other LEDs alight on the rockPro64.  I guess that my real question is, is there a particular Linux OS image that works and also what is the best way of writing the image to the NVMe drive?

Thanks for your time.

Update 09/07/20:  Werhoo!!!  Having copied the prebuilt mmc_idbloader.img and mmc_u-boot.itb files from the link below:

              https://github.com/sigmaris/u-boot/relea...2020.01-ci

I can now boot from the NVMe drive, which by the way is a 250GB Kingston A2000.  So I guess that I have made a mistake with my copy of the build, which is not surprising given that I wasn't too sure how to use the azure-pipelines.yml script so I carried out my interpretation of it. Any tips on how to run this script would be greatly appreciated.  If I manage to get my copy of the build sorted then I shall look to try kicking things off from the SPI.
  Reply
#38
(07-07-2020, 02:33 PM)ChrisS Wrote: ...
Update 09/07/20:  Werhoo!!!  Having copied the prebuilt mmc_idbloader.img and mmc_u-boot.itb files from the link below:

              https://github.com/sigmaris/u-boot/relea...2020.01-ci

I can now boot from the NVMe drive, which by the way is a 250GB Kingston A2000.  So I guess that I have made a mistake with my copy of the build, which is not surprising given that I wasn't too sure how to use the azure-pipelines.yml script so I carried out my interpretation of it. Any tips on how to run this script would be greatly appreciated.  If I manage to get my copy of the build sorted then I shall look to try kicking things off from the SPI.

U-Boot mainline 2020.07 was released recently, and I've been working on an updated branch based on the mainline 2020.07 release: ci-2020.07-rockpro64-v3

Compared to 2020.01, more RK3399 drivers are in mainline now, including the PCI driver and NVMe support. The few patches I have added on top of 2020.07 are to enable serial console in TF-A, support booting from SATA AHCI, defining partitions of the SPI flash for Linux, and some scripts to erase or flash U-boot to the SPI.

The simplified gist of how the azure-pipelines.yml tells Azure how to build this branch would be
Code:
# clone arm-trusted-firmware from ARM
git clone https://github.com/ARM-software/arm-trusted-firmware.git
cd arm-trusted-firmware

# Clean and Build arm-trusted-firmware bl31.elf
make realclean
make -j$(getconf _NPROCESSORS_ONLN) CROSS_COMPILE=aarch64-linux-gnu- PLAT=rk3399 bl31

# export environment variable BL31 containing the path to the built bl31.elf
export BL31=$(realpath build/rk3399/release/bl31/bl31.elf)

# change dir out of arm-trusted-firmware directory
cd ..

# clone my u-boot branch
git clone --single-branch --branch ci-2020.07-rockpro64-v3 https://github.com/sigmaris/u-boot.git

# change dir into u-boot
cd u-boot

# Clean and Configure U-Boot with rockpro64 default config
make mrproper
make rockpro64-rk3399_defconfig

# Build U-Boot
make -j$(getconf _NPROCESSORS_ONLN) CROSS_COMPILE=aarch64-linux-gnu-

# Make idbloader.img for MMC/SD
tools/mkimage -n rk3399 -T rksd -d tpl/u-boot-tpl.bin:spl/u-boot-spl.bin mmc_idbloader.img

# Make idbloader.img for SPI
tools/mkimage -n rk3399 -T rkspi -d tpl/u-boot-tpl.bin:spl/u-boot-spl.bin spi_idbloader.img

At the end of this, you'll have three useful files: mmc_idbloader.img, spi_idbloader.img and u-boot.itb. These are to be written to MMC/SD or SPI flash as described in the first post in this thread (in 2020.07 there is no difference between the SPI and MMC u-boot.itb - the built u-boot.itb can be used in either).

One caveat about mainline U-Boot is that it requires the Linux kernel image to be uncompressed. Default builds of upstream Linux build a gzipped kernel, and you must gunzip it or patch the kernel to not build a gzipped kernel, like ayufan's kernel does here.
  Reply
#39
Do you know if the USB3/USB-C boot process works properly now?
Sorry for any mistakes. English is not my native language

1. Quartz64 Model B, 4GB RAM

2. Quartz64 Model A, 4GB RAM

3. RockPro64 v2.1

https://linux-nerds.org/
  Reply
#40
(07-12-2020, 05:06 AM)Bullet64 Wrote: Do you know if the USB3/USB-C boot process works properly now?

I tried enabling config options from this patch but it didn't seem to change anything - my USB3 external disk is only recognised if plugged in to the USB2 ports.
  Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to use ROCKPro64 PCI-e X4 to M.2/NGFF NVMe SSD Interface Card jykeith123 1 261 11-26-2024, 11:08 AM
Last Post: wdt
  enble boot after power loss/restore dkebler 18 12,765 12-04-2023, 12:14 PM
Last Post: ok38
Bug Broken boot: What am I missing? mkosarek 1 1,094 09-08-2023, 08:14 AM
Last Post: wdt
  Unable to boot Armbian on new RockPro64 mooseball 5 5,398 07-14-2023, 08:59 AM
Last Post: rockjonn
  no boot white led flashing moserwi 7 5,757 05-18-2023, 10:46 AM
Last Post: wdt
  u-boot locked on pine64pro ljones 1 1,944 09-06-2022, 10:32 AM
Last Post: ljones
  Cannot get my board to boot deutschlmao 11 10,403 09-05-2022, 04:23 PM
Last Post: ljones
  U-BOOT Tutorial hazz 0 1,393 07-19-2022, 10:48 PM
Last Post: hazz
  Installation Debian on emmc: which U-Boot and where? vongillus 3 3,565 07-02-2022, 09:24 AM
Last Post: dkebler
  ROCKPRO64 PCI SSD SD-boot Install pspgarret 0 1,415 06-09-2022, 10:56 AM
Last Post: pspgarret

Forum Jump:


Users browsing this thread: 46 Guest(s)