Xen hypervisor on Rock64 - Printable Version

Xen hypervisor on Rock64 - Printable Version

+- PINE64 (https://forum.pine64.org)
+-- Forum: ROCK64 (https://forum.pine64.org/forumdisplay.php?fid=85)
+--- Forum: Linux on Rock64 (https://forum.pine64.org/forumdisplay.php?fid=88)
+--- Thread: Xen hypervisor on Rock64 (/showthread.php?tid=6059)

Xen hypervisor on Rock64 - tumbleweed - 05-13-2018

Hello,

I'm trying to install the Xen hypervisor on rock64, starting with the info at Xen ARM with Virtualization.
Here are the steps I followed so far:

Installed a minimal debian stretch image on the rock64 mmc (thanks @ayufan !)
Built Xen on rock64 (for XEN_TARGET_ARCH=arm64)
Used mkimage to prep xen for uboot: mkimage -A arm -C none -T kernel -a 0x00800800 -e 0x00800800 -d dist/install/boot/xen xen-uImage
Modified //boot/efi/extlinux/extlinux.conf to add a custom configuration that loads the xen kernel (xen-uImage) using the same dtb that came with the stretch image (see config code below)
Reboot with serial console attached, select my custom extlinux configuration
rock64 hangs (serial output stops) at "Starting kernel ..."

As far as I know, nobody has written up a guide to doing this, on any RK3328 board, and it's my first time trying.
Anyone have suggestions on what I might try next to get past this hanging problem?

Code:
label kernel-xen

    kernel /xen-uImage

    initrd /initrd.img

    fdt /dtb

    append earlyprintk=uart8250-32bit,0xff130000 rw root=LABEL=linux-root rootwait rootfstype=ext4 panic=10 init=/sbin/init coherent_pool=1M ethaddr=${ethaddr} eth1addr=${eth1addr} serial=${serial#} cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1

RE: Xen hypervisor on Rock64 - evilbunny - 05-13-2018

(05-13-2018, 09:03 PM)tumbleweed Wrote: Anyone have suggestions on what I might try next to get past this hanging problem?

I haven't tried xen on arm, but there are builds for docker and kubernetes, are these an option for you?

RE: Xen hypervisor on Rock64 - tumbleweed - 05-15-2018

(05-13-2018, 11:40 PM)evilbunny Wrote:
(05-13-2018, 09:03 PM)tumbleweed Wrote: Anyone have suggestions on what I might try next to get past this hanging problem?
I haven't tried xen on arm, but there are builds for docker and kubernetes, are these an option for you?

Thanks for the reply.
The goal is to run a unikernel under the xen hypervisor.
Similar to running containers, but not the same.
Is there another forum you’d suggest for posting this question?

RE: Xen hypervisor on Rock64 - sstabellini - 05-17-2018

Hi Tumbleweed,I haven\'t tried Xen on Rock64, but I did run Xen on Pine64 successfully in the past.The mkimage step below is only required if you boot Xen from U-Boot. If you boot Xen from extlinux or another bootloader, it will actually cause troubles. Instead, to simplify the boot sequence, I would boot Xen directly from EFI, skipping extlinux completely. To do that, you need to copy xen.efi to the FAT partition, and write a xen.cfg config file. You also need to copy the Dom0 kernel, initrd and device tree to the fat partition. For instance, see this guide for booting Xen on ARM on QEMU:https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/qemu-system-aarch64There are a few QEMU specific steps in there, but it also contains the steps to boot Xen directly from UEFI including an example xen.cfg.It might also be possible to boot Xen from extlinux but I don't have experience with that. I recommend booting from UEFI first. Cheers,Stefano

(05-13-2018, 09:03 PM)tumbleweed Wrote: Hello,

I'm trying to install the Xen hypervisor on rock64, starting with the info at Xen ARM with Virtualization.
Here are the steps I followed so far:

Installed a minimal debian stretch image on the rock64 mmc (thanks @ayufan !)

Built Xen on rock64 (for XEN_TARGET_ARCH=arm64)

Used mkimage to prep xen for uboot: mkimage -A arm -C none -T kernel -a 0x00800800 -e 0x00800800 -d dist/install/boot/xen xen-uImage

Modified //boot/efi/extlinux/extlinux.conf to add a custom configuration that loads the xen kernel (xen-uImage) using the same dtb that came with the stretch image (see config code below)

Reboot with serial console attached, select my custom extlinux configuration

rock64 hangs (serial output stops) at "Starting kernel ..."

As far as I know, nobody has written up a guide to doing this, on any RK3328 board, and it's my first time trying.
Anyone have suggestions on what I might try next to get past this hanging problem?

Code:
label kernel-xen kernel /xen-uImage initrd /initrd.img fdt /dtb append earlyprintk=uart8250-32bit,0xff130000 rw root=LABEL=linux-root rootwait rootfstype=ext4 panic=10 init=/sbin/init coherent_pool=1M ethaddr=${ethaddr} eth1addr=${eth1addr} serial=${serial#} cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1

RE: Xen hypervisor on Rock64 - tumbleweed - 05-19-2018

Hi sstabellini, thanks for the suggestions.
I tried to setup direct UEFI boot without success. (Basically I started with the QEMU instructions and created /boot/efi/EFI/xen/ with xen.efi , xen.cfg and so forth. )

It appears that EFI booting is skipped altogether, and for example:

Code:
rock64@rock64:~$ efibootmgr -w -L Xen -l "\EFI\Xen\xen.efi" -c

EFI variables are not supported on this system.

Looking at the Rockchip boot options diagram, it appears that either u-boot.bin or UEFI.FD is run as "Loader 2" , only for "Boot Flow 1". Looking at the serial console output below, it seems to me the Rock64 is following "Boot Flow 2", which uses u-boot, since we see "BL31" running.

The serial console output at reboot, if I read it correctly, shows the system directly booting into u-boot.
How can I ensure the Rock64 chooses UEFI boot?

Code:
=> reset

DDR version 1.08 20170628

In

SRX

LPDDR3

333MHz

Bus Width=32 Col=11 Bank=8 Row=15/15 CS=2 Die Bus-Width=32 Size=4096MB

ddrconfig:7

OUT

U-Boot SPL 2017.09-ga0a2b48 (Apr 29 2018 - 20:24:03)

setup_ddr_param  1

booted from eMMC

Trying to boot from MMC1

NOTICE:  BL31: v1.3(debug):9d3f591

NOTICE:  BL31: Built : 14:39:02, Jan 17 2018

NOTICE:  BL31:Rockchip release version: v1.3

INFO:    ARM GICv2 driver initialized

INFO:    Using opteed sec cpu_context!

INFO:    boot cpu mask: 1

INFO:    plat_rockchip_pmu_init: pd status 0xe

INFO:    BL31: Initializing runtime services

WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK

ERROR:   Error initializing runtime service opteed_fast

INFO:    BL31: Preparing for EL3 exit to normal world

INFO:    Entry point address = 0x200000

INFO:    SPSR = 0x3c9

U-Boot 2017.09-ga0a2b48 (Apr 29 2018 - 20:24:18 +0000), Build: jenkins-linux-build-rock-64-213

Model: Pine64 Rock64

DRAM:  4 GiB

MMC:   rksdmmc@ff520000: 0, rksdmmc@ff500000: 1

Card did not respond to voltage select!

mmc_init: -95, time 10

*** Warning - No block device, using default environment

In:    serial@ff130000

Out:   serial@ff130000

Err:   serial@ff130000

Model: Pine64 Rock64

misc_init_r

cpuid=55524b50303930343000000000010715

serial=9a6b577037c4808

normal boot

Net:   eth0: ethernet@ff540000

Hit any key to stop autoboot:  0

Quote:sstabelliniHi Tumbleweed,I haven\'t tried Xen on Rock64, but I did run Xen on Pine64 successfully in the past.The mkimage step below is only required if you boot Xen from U-Boot. If you boot Xen from extlinux or another bootloader, it will actually cause troubles. Instead, to simplify the boot sequence, I would boot Xen directly from EFI, skipping extlinux completely. To do that, you need to copy xen.efi to the FAT partition, and write a xen.cfg config file. You also need to copy the Dom0 kernel, initrd and device tree to the fat partition. For instance, see this guide for booting Xen on ARM on QEMU: https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/qemu-system-aarch64 There are a few QEMU specific steps in there, but it also contains the steps to boot Xen directly from UEFI including an example xen.cfg.It might also be possible to boot Xen from extlinux but I don't have experience with that. I recommend booting from UEFI first. Cheers,Stefano

RE: Xen hypervisor on Rock64 - tumbleweed - 05-23-2018

I was able to get a bit further with loading with the EFI approach.

First I flashed u-boot onto SPI flash.

Then I loaded the xen.efi onto a micro sd card (as bootaa64.efi), stopped the uboot autoboot,
and attempted to manually boot using efi. This yields the errors shown below -- Any suggestions?

Code:
Hit any key to stop autoboot:  0 

=> set devtype mmc

=> set devnum 1

=> set distro_bootpart 6

=> run boot_efi_binary

reading efi/boot/bootaa64.efi

885072 bytes read in 128 ms (6.6 MiB/s)

libfdt fdt_check_header(): FDT_ERR_BADMAGIC

## Starting EFI application at 02000000 ...

Card did not respond to voltage select!

mmc_init: -95, time 10

Scanning disk rksdmmc@ff520000.blk...

Scanning disk rksdmmc@ff500000.blk...

Found 2 disks

Xen 4.11-rc (c/s Tue May 1 09:03:13 2018 +0100 git:0306a1311d) EFI loader

Couldn't obtain the File System Protocol Interface: ErrCode: 0x8000000000000003

## Application terminated, r = 0

For reference, here's what's on the sd card:

Code:
=> ls mmc 1:6

 19644424   image 

            efi/

            extlinux/

  2920159   initrd.img 

    56922   dtb 

      134   xen.cfg

=> ls mmc 1:6 efi/boot

            ./

            ../

   885072   bootaa64.efi

The existing Image and dtb boot OK (using /extlinux/extlinux.conf ) but xen does not boot.

RE: Xen hypervisor on Rock64 - darkfader - 11-03-2025

Hey,

I know this is an old post but I think the problem is common enough and hard enough to figure.

So when you see this scenario - Xen kernel starting and then just "Starting kernel" with regards to Linux, or not even see the Xen kernel starting.

I went through that last week with a Cubietruck.

It almost always means no more than your output is going to a different or misconfigured or non-existent console.
It does almost never mean there was a failure to execute your kernel.

In my case, I had to _remove_ the whole set of definitions like dtuart, console=dtuart, earlycons et ceterea.
EVERYTHING. just no special setting for Xen, load the correct DTB of course, and for Linux kernel, plain and simple console=hvc0.

From that moment on, I had Xen boot output on serial, then Linux kernel boot output on serial, then Linux console on serial.

There was other issues, mostly Xen vs Kernel vs. xl toolstack mismatches
A non-Xen enabled kernel (alpine-lts iirc) and a few more stupid mishaps.

But the main and most common issue was simply that I could not find any indication it was booting when in fact it was booting (and then failed of course), and that the serial console setup didn't work in the "oh lets do it super perfectly addressing the device from dt" way or any other, but worked perfectly in the default which i had never tried since all instructions very specific about giving a device tree address. Just try not doing that and see how it goes.

There's also reports that some DTB's have pins inverted and stuff like that, I tried reversing tx+rx cables, nope.
The failing setup is more complex than that, but there's no reason (unless you do it commercially) to look into that when you can just not specify it and be done with it.

Best,
Flo