Xen hypervisor on Rock64

[tumbleweed]

Hello,

I'm trying to install the Xen hypervisor on rock64, starting with the info at Xen ARM with Virtualization.
Here are the steps I followed so far:

1. Installed a minimal debian stretch image on the rock64 mmc (thanks @ayufan !)
   
2. Built Xen on rock64 (for XEN_TARGET_ARCH=arm64)
   
3. Used mkimage to prep xen for uboot: mkimage -A arm -C none -T kernel -a 0x00800800 -e 0x00800800 -d dist/install/boot/xen xen-uImage
   
4. Modified //boot/efi/extlinux/extlinux.conf  to add a custom configuration that loads the xen kernel (xen-uImage) using the same dtb that came with the stretch image (see config code below)
   
5. Reboot with serial console attached, select my custom extlinux configuration
   
6. rock64 hangs (serial output stops) at "Starting kernel ..."
   

As far as I know, nobody has written up a guide to doing this, on any RK3328 board, and it's my first time trying.
Anyone have suggestions on what I might try next to get past this hanging problem? 

Code:

label kernel-xen
   kernel /xen-uImage
   initrd /initrd.img
   fdt /dtb
   append earlyprintk=uart8250-32bit,0xff130000 rw root=LABEL=linux-root rootwait rootfstype=ext4 panic=10 init=/sbin/init coherent_pool=1M ethaddr=${ethaddr} eth1addr=${eth1addr} serial=${serial#} cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1

[evilbunny]

Anyone have suggestions on what I might try next to get past this hanging problem? 

I haven't tried xen on arm, but there are builds for docker and kubernetes, are these an option for you?

[tumbleweed]

Anyone have suggestions on what I might try next to get past this hanging problem? 

I haven't tried xen on arm, but there are builds for docker and kubernetes, are these an option for you?

Thanks for the reply. 
The goal is to run a unikernel under the xen hypervisor. 
Similar to running containers, but not the same. 
Is there another forum you’d suggest for posting this question?

[sstabellini]

Hi Tumbleweed,I haven\'t tried Xen on Rock64, but I did run Xen on Pine64 successfully in the past.The mkimage step below is only required if you boot Xen from U-Boot. If you boot Xen from extlinux or another bootloader, it will actually cause troubles. Instead, to simplify the boot sequence, I would boot Xen directly from EFI, skipping extlinux completely. To do that, you need to copy xen.efi to the FAT partition, and write a xen.cfg config file. You also need to copy the Dom0 kernel, initrd and device tree to the fat partition. For instance, see this guide for booting Xen on ARM on QEMU:https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/qemu-system-aarch64There are a few QEMU specific steps in there, but it also contains the steps to boot Xen directly from UEFI including an example xen.cfg.It might also be possible to boot Xen from extlinux but I don't have experience with that. I recommend booting from UEFI first. Cheers,Stefano

Hello,

I'm trying to install the Xen hypervisor on rock64, starting with the info at Xen ARM with Virtualization.
Here are the steps I followed so far:

1. Installed a minimal debian stretch image on the rock64 mmc (thanks @ayufan !)
   
2. Built Xen on rock64 (for XEN_TARGET_ARCH=arm64)
   
3. Used mkimage to prep xen for uboot: mkimage -A arm -C none -T kernel -a 0x00800800 -e 0x00800800 -d dist/install/boot/xen xen-uImage
   
4. Modified //boot/efi/extlinux/extlinux.conf  to add a custom configuration that loads the xen kernel (xen-uImage) using the same dtb that came with the stretch image (see config code below)
   
5. Reboot with serial console attached, select my custom extlinux configuration
   
6. rock64 hangs (serial output stops) at "Starting kernel ..."
   

As far as I know, nobody has written up a guide to doing this, on any RK3328 board, and it's my first time trying.
Anyone have suggestions on what I might try next to get past this hanging problem? 

Code:

label kernel-xen
   kernel /xen-uImage
   initrd /initrd.img
   fdt /dtb
   append earlyprintk=uart8250-32bit,0xff130000 rw root=LABEL=linux-root rootwait rootfstype=ext4 panic=10 init=/sbin/init coherent_pool=1M ethaddr=${ethaddr} eth1addr=${eth1addr} serial=${serial#} cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1

[tumbleweed]

Hi sstabellini, thanks for the suggestions.  
I tried to setup direct UEFI boot without success.  (Basically I started with the QEMU instructions and created  /boot/efi/EFI/xen/ with xen.efi , xen.cfg and so forth. )

It appears that EFI booting is skipped altogether, and for example:

Code:

rock64@rock64:~$ efibootmgr -w -L Xen -l "\EFI\Xen\xen.efi" -c
EFI variables are not supported on this system.

Looking at the Rockchip boot options diagram, it appears that either u-boot.bin or UEFI.FD is run as "Loader 2" , only for "Boot Flow 1".  Looking at the serial console output below, it seems to me the Rock64 is following "Boot Flow 2", which uses u-boot, since we see "BL31" running.

The serial console output at reboot, if I read it correctly, shows the system directly booting into u-boot.  
How can I ensure the Rock64 chooses UEFI boot?  

Code:

=> reset
DDR version 1.08 20170628
In
SRX
LPDDR3
333MHz
Bus Width=32 Col=11 Bank=8 Row=15/15 CS=2 Die Bus-Width=32 Size=4096MB
ddrconfig:7
OUT

U-Boot SPL 2017.09-ga0a2b48 (Apr 29 2018 - 20:24:03)
setup_ddr_param  1
booted from eMMC
Trying to boot from MMC1
NOTICE:  BL31: v1.3(debug):9d3f591
NOTICE:  BL31: Built : 14:39:02, Jan 17 2018
NOTICE:  BL31:Rockchip release version: v1.3
INFO:    ARM GICv2 driver initialized
INFO:    Using opteed sec cpu_context!
INFO:    boot cpu mask: 1
INFO:    plat_rockchip_pmu_init: pd status 0xe
INFO:    BL31: Initializing runtime services
WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
ERROR:   Error initializing runtime service opteed_fast
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x200000
INFO:    SPSR = 0x3c9


U-Boot 2017.09-ga0a2b48 (Apr 29 2018 - 20:24:18 +0000), Build: jenkins-linux-build-rock-64-213

Model: Pine64 Rock64
DRAM:  4 GiB
MMC:   rksdmmc@ff520000: 0, rksdmmc@ff500000: 1
Card did not respond to voltage select!
mmc_init: -95, time 10
*** Warning - No block device, using default environment

In:    serial@ff130000
Out:   serial@ff130000
Err:   serial@ff130000
Model: Pine64 Rock64
misc_init_r
cpuid=55524b50303930343000000000010715
serial=9a6b577037c4808
normal boot
Net:   eth0: ethernet@ff540000
Hit any key to stop autoboot:  0

sstabelliniHi Tumbleweed,I haven\'t tried Xen on Rock64, but I did run Xen on Pine64 successfully in the past.The mkimage step below is only required if you boot Xen from U-Boot. If you boot Xen from extlinux or another bootloader, it will actually cause troubles. Instead, to simplify the boot sequence, I would boot Xen directly from EFI, skipping extlinux completely. To do that, you need to copy xen.efi to the FAT partition, and write a xen.cfg config file. You also need to copy the Dom0 kernel, initrd and device tree to the fat partition. For instance, see this guide for booting Xen on ARM on QEMU: https://wiki.xenproject.org/wiki/Xen_ARM...em-aarch64 There are a few QEMU specific steps in there, but it also contains the steps to boot Xen directly from UEFI including an example xen.cfg.It might also be possible to boot Xen from extlinux but I don't have experience with that. I recommend booting from UEFI first. Cheers,Stefano

[tumbleweed]

I was able to get a bit further with loading with the EFI approach.

First I flashed u-boot onto SPI flash.

Then I loaded the xen.efi onto a micro sd card (as bootaa64.efi), stopped the uboot autoboot, 
and attempted to manually boot using efi.  This yields the errors shown below -- Any suggestions?

Code:

Hit any key to stop autoboot:  0
=> set devtype mmc
=> set devnum 1
=> set distro_bootpart 6
=> run boot_efi_binary
reading efi/boot/bootaa64.efi
885072 bytes read in 128 ms (6.6 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
## Starting EFI application at 02000000 ...
Card did not respond to voltage select!
mmc_init: -95, time 10
Scanning disk rksdmmc@ff520000.blk...
Scanning disk rksdmmc@ff500000.blk...
Found 2 disks
Xen 4.11-rc (c/s Tue May 1 09:03:13 2018 +0100 git:0306a1311d) EFI loader
Couldn't obtain the File System Protocol Interface: ErrCode: 0x8000000000000003
## Application terminated, r = 0

For reference, here's what's on the sd card:

Code:

=> ls mmc 1:6
19644424   image
           efi/
           extlinux/
 2920159   initrd.img
   56922   dtb
     134   xen.cfg

=> ls mmc 1:6 efi/boot
            ./
            ../
   885072   bootaa64.efi 

The existing Image and dtb boot OK (using /extlinux/extlinux.conf )  but xen does not boot.

[darkfader]

Hey,

I know this is an old post but I think the problem is common enough and hard enough to figure.

So when you see this scenario - Xen kernel starting and then just "Starting kernel" with regards to Linux, or not even see the Xen kernel starting.

I went through that last week with a Cubietruck.

It almost always means no more than your output is going to a different or misconfigured or non-existent console.
It does almost never mean there was a failure to execute your kernel.

In my case, I had to _remove_ the whole set of definitions like dtuart, console=dtuart, earlycons et ceterea.
EVERYTHING. just no special setting for Xen, load the correct DTB of course, and for Linux kernel, plain and simple console=hvc0.

From that moment on, I had Xen boot output on serial, then Linux kernel boot output on serial, then Linux console on serial.

There was other issues, mostly Xen vs Kernel vs. xl toolstack mismatches
A non-Xen enabled kernel (alpine-lts iirc) and a few more stupid mishaps.

But the main and most common issue was simply that I could not find any indication it was booting when in fact it was booting (and then failed of course), and that the serial console setup didn't work in the "oh lets do it super perfectly addressing the device from dt" way or any other, but worked perfectly in the default which i had never tried since all instructions very specific about giving a device tree address. Just try not doing that and see how it goes.

There's also reports that some DTB's have pins inverted and stuff like that, I tried reversing tx+rx cables, nope.
The failing setup is more complex than that, but there's no reason (unless you do it commercially) to look into that when you can just not specify it and be done with it.

Best,
Flo