Xen hypervisor on Rock64

[tumbleweed]

Hello,

I'm trying to install the Xen hypervisor on rock64, starting with the info at Xen ARM with Virtualization.
Here are the steps I followed so far:

1. Installed a minimal debian stretch image on the rock64 mmc (thanks @ayufan !)
   
2. Built Xen on rock64 (for XEN_TARGET_ARCH=arm64)
   
3. Used mkimage to prep xen for uboot: mkimage -A arm -C none -T kernel -a 0x00800800 -e 0x00800800 -d dist/install/boot/xen xen-uImage
   
4. Modified //boot/efi/extlinux/extlinux.conf  to add a custom configuration that loads the xen kernel (xen-uImage) using the same dtb that came with the stretch image (see config code below)
   
5. Reboot with serial console attached, select my custom extlinux configuration
   
6. rock64 hangs (serial output stops) at "Starting kernel ..."
   

As far as I know, nobody has written up a guide to doing this, on any RK3328 board, and it's my first time trying.
Anyone have suggestions on what I might try next to get past this hanging problem? 

Code:

label kernel-xen
   kernel /xen-uImage
   initrd /initrd.img
   fdt /dtb
   append earlyprintk=uart8250-32bit,0xff130000 rw root=LABEL=linux-root rootwait rootfstype=ext4 panic=10 init=/sbin/init coherent_pool=1M ethaddr=${ethaddr} eth1addr=${eth1addr} serial=${serial#} cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1

[evilbunny]

Anyone have suggestions on what I might try next to get past this hanging problem? 

I haven't tried xen on arm, but there are builds for docker and kubernetes, are these an option for you?

[tumbleweed]

Anyone have suggestions on what I might try next to get past this hanging problem? 

I haven't tried xen on arm, but there are builds for docker and kubernetes, are these an option for you?

Thanks for the reply. 
The goal is to run a unikernel under the xen hypervisor. 
Similar to running containers, but not the same. 
Is there another forum you’d suggest for posting this question?

[sstabellini]

Hi Tumbleweed,I haven\'t tried Xen on Rock64, but I did run Xen on Pine64 successfully in the past.The mkimage step below is only required if you boot Xen from U-Boot. If you boot Xen from extlinux or another bootloader, it will actually cause troubles. Instead, to simplify the boot sequence, I would boot Xen directly from EFI, skipping extlinux completely. To do that, you need to copy xen.efi to the FAT partition, and write a xen.cfg config file. You also need to copy the Dom0 kernel, initrd and device tree to the fat partition. For instance, see this guide for booting Xen on ARM on QEMU:https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/qemu-system-aarch64There are a few QEMU specific steps in there, but it also contains the steps to boot Xen directly from UEFI including an example xen.cfg.It might also be possible to boot Xen from extlinux but I don't have experience with that. I recommend booting from UEFI first. Cheers,Stefano

Hello,

I'm trying to install the Xen hypervisor on rock64, starting with the info at Xen ARM with Virtualization.
Here are the steps I followed so far:

1. Installed a minimal debian stretch image on the rock64 mmc (thanks @ayufan !)
   
2. Built Xen on rock64 (for XEN_TARGET_ARCH=arm64)
   
3. Used mkimage to prep xen for uboot: mkimage -A arm -C none -T kernel -a 0x00800800 -e 0x00800800 -d dist/install/boot/xen xen-uImage
   
4. Modified //boot/efi/extlinux/extlinux.conf  to add a custom configuration that loads the xen kernel (xen-uImage) using the same dtb that came with the stretch image (see config code below)
   
5. Reboot with serial console attached, select my custom extlinux configuration
   
6. rock64 hangs (serial output stops) at "Starting kernel ..."
   

As far as I know, nobody has written up a guide to doing this, on any RK3328 board, and it's my first time trying.
Anyone have suggestions on what I might try next to get past this hanging problem? 

Code:

label kernel-xen
   kernel /xen-uImage
   initrd /initrd.img
   fdt /dtb
   append earlyprintk=uart8250-32bit,0xff130000 rw root=LABEL=linux-root rootwait rootfstype=ext4 panic=10 init=/sbin/init coherent_pool=1M ethaddr=${ethaddr} eth1addr=${eth1addr} serial=${serial#} cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1

[tumbleweed]

Hi sstabellini, thanks for the suggestions.  
I tried to setup direct UEFI boot without success.  (Basically I started with the QEMU instructions and created  /boot/efi/EFI/xen/ with xen.efi , xen.cfg and so forth. )

It appears that EFI booting is skipped altogether, and for example:

Code:

rock64@rock64:~$ efibootmgr -w -L Xen -l "\EFI\Xen\xen.efi" -c
EFI variables are not supported on this system.

Looking at the Rockchip boot options diagram, it appears that either u-boot.bin or UEFI.FD is run as "Loader 2" , only for "Boot Flow 1".  Looking at the serial console output below, it seems to me the Rock64 is following "Boot Flow 2", which uses u-boot, since we see "BL31" running.

The serial console output at reboot, if I read it correctly, shows the system directly booting into u-boot.  
How can I ensure the Rock64 chooses UEFI boot?  

Code:

=> reset
DDR version 1.08 20170628
In
SRX
LPDDR3
333MHz
Bus Width=32 Col=11 Bank=8 Row=15/15 CS=2 Die Bus-Width=32 Size=4096MB
ddrconfig:7
OUT

U-Boot SPL 2017.09-ga0a2b48 (Apr 29 2018 - 20:24:03)
setup_ddr_param  1
booted from eMMC
Trying to boot from MMC1
NOTICE:  BL31: v1.3(debug):9d3f591
NOTICE:  BL31: Built : 14:39:02, Jan 17 2018
NOTICE:  BL31:Rockchip release version: v1.3
INFO:    ARM GICv2 driver initialized
INFO:    Using opteed sec cpu_context!
INFO:    boot cpu mask: 1
INFO:    plat_rockchip_pmu_init: pd status 0xe
INFO:    BL31: Initializing runtime services
WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
ERROR:   Error initializing runtime service opteed_fast
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x200000
INFO:    SPSR = 0x3c9


U-Boot 2017.09-ga0a2b48 (Apr 29 2018 - 20:24:18 +0000), Build: jenkins-linux-build-rock-64-213

Model: Pine64 Rock64
DRAM:  4 GiB
MMC:   rksdmmc@ff520000: 0, rksdmmc@ff500000: 1
Card did not respond to voltage select!
mmc_init: -95, time 10
*** Warning - No block device, using default environment

In:    serial@ff130000
Out:   serial@ff130000
Err:   serial@ff130000
Model: Pine64 Rock64
misc_init_r
cpuid=55524b50303930343000000000010715
serial=9a6b577037c4808
normal boot
Net:   eth0: ethernet@ff540000
Hit any key to stop autoboot:  0

sstabelliniHi Tumbleweed,I haven\'t tried Xen on Rock64, but I did run Xen on Pine64 successfully in the past.The mkimage step below is only required if you boot Xen from U-Boot. If you boot Xen from extlinux or another bootloader, it will actually cause troubles. Instead, to simplify the boot sequence, I would boot Xen directly from EFI, skipping extlinux completely. To do that, you need to copy xen.efi to the FAT partition, and write a xen.cfg config file. You also need to copy the Dom0 kernel, initrd and device tree to the fat partition. For instance, see this guide for booting Xen on ARM on QEMU: https://wiki.xenproject.org/wiki/Xen_ARM...em-aarch64 There are a few QEMU specific steps in there, but it also contains the steps to boot Xen directly from UEFI including an example xen.cfg.It might also be possible to boot Xen from extlinux but I don't have experience with that. I recommend booting from UEFI first. Cheers,Stefano

[tumbleweed]

I was able to get a bit further with loading with the EFI approach.

First I flashed u-boot onto SPI flash.

Then I loaded the xen.efi onto a micro sd card (as bootaa64.efi), stopped the uboot autoboot, 
and attempted to manually boot using efi.  This yields the errors shown below -- Any suggestions?

Code:

Hit any key to stop autoboot:  0
=> set devtype mmc
=> set devnum 1
=> set distro_bootpart 6
=> run boot_efi_binary
reading efi/boot/bootaa64.efi
885072 bytes read in 128 ms (6.6 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
## Starting EFI application at 02000000 ...
Card did not respond to voltage select!
mmc_init: -95, time 10
Scanning disk rksdmmc@ff520000.blk...
Scanning disk rksdmmc@ff500000.blk...
Found 2 disks
Xen 4.11-rc (c/s Tue May 1 09:03:13 2018 +0100 git:0306a1311d) EFI loader
Couldn't obtain the File System Protocol Interface: ErrCode: 0x8000000000000003
## Application terminated, r = 0

For reference, here's what's on the sd card:

Code:

=> ls mmc 1:6
19644424   image
           efi/
           extlinux/
 2920159   initrd.img
   56922   dtb
     134   xen.cfg

=> ls mmc 1:6 efi/boot
            ./
            ../
   885072   bootaa64.efi 

The existing Image and dtb boot OK (using /extlinux/extlinux.conf )  but xen does not boot.