Step by step guide PXE diskless configuration.

Step by step guide PXE diskless configuration. - Printable Version

+- PINE64 (https://forum.pine64.org)
+-- Forum: ROCK64 (https://forum.pine64.org/forumdisplay.php?fid=85)
+--- Forum: Rock64 Tutorials (https://forum.pine64.org/forumdisplay.php?fid=90)
+--- Thread: Step by step guide PXE diskless configuration. (/showthread.php?tid=6814)

Pages: 1 2

Step by step guide PXE diskless configuration. - burglar_ot - 11-15-2018

I haven't find a guide to setup a proper PXE boot for Rock64 (and Pine64) cards, so I will write down mine here hoping that it is useful.

In this guide I assume that the user is familiar with Linux, another assumption is about the distributions:
- on the Rock64 card we are going to install the Debian minimal;
- on the server that will provide the operating system I use Debian.
I think this guide can work for other distributions too but I haven't tried other combinations.
For the rest of the guide all the commands will run on the server Debian.

About the hardware I tried on a Rock64pro, I have also a Pine64 and a Rock64 but are in production and I needed a spare board to do the tests. Now that it works I will try also for the others, but for this guide I assume the Rock64pro.

Step 1: Flash the SPI
This is actually the only step that requires an SD card.
The SPI is a little memory (128MB) that can be flashed to add instruction at the boot time. Ayufan has a nice SPI image that adds the possibility to boot from network (PXE), and we are going to use it.

So go to this link and download the image file that is useful for your system. I used the u-boot-flash-spi-rockpro64.img.xz file.
Uncompress it with

Code:
unxz u-boot-flash-spi-rockpro64.img.xz

and flash it on the SD card with dd

Code:
dd bs=1M if=u-boot-flash-spi-rockpro64.img of=/dev/sd<SOMETHING>

Put the card into the Rock64 board and boot it. I had no monitor connected so I do not know what it writes on a screen, but it is enough that you wait some time (one minute is probably enough) and you will see the boot (white) led flashing once per second. It means that the SPI is flashed. You can now turn off the board and remove the SD card.
Step 2: find the ethernet address of the board.
I assume that you have already a DHCP server installed in your network that assign the addresses to every device connected.
If this DHCP server is the router of your internet connection you will have to change it later (unless is a very advanced router and you can specify advanced features). In any case to identify the mac address is sufficient that something assign an IP address to your board.
So turn on your Rock64. It will now spin up and try to boot from SD, eMMC, USB and PXE, and it will fail in all of these tentatives because it has no cards installed and the network boot is not yet configured. Nevertheless, when it tries the PXE the board will get an IP and it will announce its existence to the local network. This means that after few seconds that the ethernet led flashes you can write on your Debian server

Code:
arp -a

and you will get the list of connected devices on the network with their IP address and hardware address.
Write down the ethernet address; the one that I will use in my example is 56:31:cd:bb:92:7b.
Step 3: the DHCP server.
We need a DHCP server capable of some advanced features if you have the DHCP on your router you have to disable it and from this moment on you will use the DHCP on your Debian server.
Install it with

Code:
sudo apt-get install isc-dhcp-server

Once installed edit the /etc/dhcp/dhcpd.conf in this way

Code:
ddns-update-style none;

option domain-name "whatever.com";

option domain-name-servers 192.168.0.1;

default-lease-time 6000;

max-lease-time 72000;

log-facility local7;

subnet 192.168.0.0 netmask 255.255.255.0 {

   range 192.168.0.2 192.168.0.99;

   option subnet-mask 255.255.255.0;

   option broadcast-address 192.168.0.255;

   option routers 192.168.0.1;

   option domain-name-servers 192.168.0.1;

}

host rock64pro {

  hardware ethernet 56:31:cd:bb:92:7b;

  fixed-address 192.168.0.100;

  next-server 192.168.0.1;

}

authoritative;

I don't want to go into the details but basically this configuration allocates random addresses between 192.168.0.2 and 192.168.0.99 for the normal devices (like computers or telephones) and assign the 192.168.0.100 to the rock64pro identifying it with the ethernet address that we find in the previous step. Moreover, and this is the advanced feature, the server will tell to the rock64pro that after it takes the address it has to communicate with 192.168.0.1 for "further instructions" (the next-server directive).
Here I am assuming that 192.168.0.1 is the IP address of the Debian server.
Once this file is in place restart the DHCP server with

Code:
sudo systemctl restart isc-dhcp-server

You can test if this configuration works turning on the Rock64pro board, waiting couple of seconds and try again the arp -a. If everything is fine you should see that now the Rock has the address 192.168.0.100.

Step 4: TFTP server
We have now to instruct the Rock64pro for the next step. The way to take information is through a special version of an ftp server called tftp (t is for trivial). So we need a tftp server running on our Debian and to install it we do

Code:
sudo apt-get install tftp-hpa

once installed edit the file /etc/defaults/tftpd-hpa as

Code:
TFTP_USERNAME="tftp"

TFTP_DIRECTORY="/storage/pxe-boot/tftp"

TFTP_ADDRESS="192.168.0.1:69"

TFTP_OPTIONS="--secure"

here I am assuming that we will store the data to boot the Rock64pro into a directory /storage/pxe-boot and in this directory we will put all the files for the boot and for the operating system.
So we need to do these steps

Code:
sudo mkdir /storage

sudo mkdir /storage/rock64pro

sudo mkdir /storage/tftp

sudo mkdir /storage/tftp/pxelinux.cfg

In rock64pro we will put the operating system (the content of the boot image) while in the tftp we will put all the files to the initial boot procedure (linux kernel and its configurations).
Now we can restart the tftp server

Code:
sudo systemctl restart tftpd-hpa

Step 5: NFS server
The kernel and the boot execution is provided through the tftp server, but the operating system will live entirely on a folder that will be served to the rock64pro through NFS, so we need a working NFS server.
To install it

Code:
sudo apt-get install nfs-kernel-server rpcbind

the edit the file /etc/exports as

Code:
/storage/pxe-boot/rock64pro     *(rw,sync,no_root_squash)

and the file /etc/hosts.allow as

Code:
portmap: 192.168.0.

The first file says to NFS server that has to export the directory /storage/pxe-boot/rock64pro to everyone with specific flags (read/write, sync, and no limitation for root). The second file allows the IPs of the subnet 192.168.0.x to connect to the NFS server. You may want to limit the NFS server only to the IP of the rock64pro substituting 192.168.0.100 to the * of the exports file and writing the full ip into the hosts.allow.
Restart the NFS server with

Code:
sudo systemctl restart nfs-kernel-server

Step 6: setup the filesystem
Download the Debian image for your Rock card from this link (you can use wget). I used the stretch-minimal-rockpro64-0.7.11-1075-arm64.img.xz. Uncompress it with

Code:
unxz stretch-minimal-rockpro64-0.7.11-1075-arm64.img.xz

Once downloaded you have to mount the linux partition contained in the img file. The file contains also another partition that is to boot and we are not interested in it, so we have to find the right position of the linux. To do it we use fdisk as

Code:
sudo fdisk -l stretch-minimal-rockpro64-0.7.11-1075-arm64.img

the result is something like

Code:
Disk stretch-minimal-rockpro64-0.7.11-1075-arm64.img: 2 GiB, 2144337920 bytes, 4188160 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: gpt

Disk identifier: FB84DCAE-4E0E-486F-890C-A979BEECCFF9

Device                                            Start     End Sectors  Size Type

stretch-minimal-rockpro64-0.7.11-1075-arm64.img1     64    8063    8000  3.9M Linux filesystem

stretch-minimal-rockpro64-0.7.11-1075-arm64.img2   8064    8191     128   64K Linux filesystem

stretch-minimal-rockpro64-0.7.11-1075-arm64.img3   8192   16383    8192    4M Linux filesystem

stretch-minimal-rockpro64-0.7.11-1075-arm64.img4  16384   24575    8192    4M Linux filesystem

stretch-minimal-rockpro64-0.7.11-1075-arm64.img5  24576   32767    8192    4M Linux filesystem

stretch-minimal-rockpro64-0.7.11-1075-arm64.img6  32768  262143  229376  112M Microsoft basic data

stretch-minimal-rockpro64-0.7.11-1075-arm64.img7 262144 4186111 3923968  1.9G Linux filesystem

Here you have to identify two things: the first one is the size of one unit sector (512 bytes) and the second information is the begin of the filesystem that we are looking for. In our case it is the last one and we know because is the one of 1.9GB of size. Then this filesystem starts at the position 262144. Now we multiply these two numbers: 512x262144=134217728. This is the starting position of the linux partition. We can use it to mount the partition as

Code:
sudo mkdir /storage/tmpimage

sudo mount -o loop,offset=134217728 stretch-minimal-rockpro64-0.7.11-1075-arm64.img /storage/tmpimage

now we copy the content of the filesystem into our nfs server. I use rsync to do it because I trust the way it preserves permissions and links, but probably a simple cp -a can do the job.

Code:
sudo rsync -val /storage/tmpimage/* /storage/pxe-boot/rock64pro/

After the sync the image is no longer needed

Code:
sudo umount /storage/tmpimage

sudo rmdir /storage/tmpimage

sudo rm stretch-minimal-rockpro64-0.7.11-1075-arm64.img

Step 7: Finalize the boot
We have now the system ready, we need only to put in the tftp directory the instructions to send the kernel to the board for the boot.
To do this we first copy the boot folder of the rock64pro filesystem to the tftp directory. We need to do it because we need to provide the kernel via tftp before mounting the NFS root.

Code:
sudo rsync -val /storage/pxe-boot/rock64pro/boot /storage/pxe-boot/tftp/

finally we need to provide to the board the extlinux.conf boot file. When the Rock boots it searches for a boot file with various names, we will use the pxelinux.cfg/default-arm-rockchip. So we first copy the extlinux.conf as

Code:
sudo cp /storage/pxe-boot/tftp/boot/extlinux/extlinux.conf /storage/pxe-boot/tftp/pxelinux.cfg/default-arm-rockchip

then we need to edit this file because by default it will try to load the kernel from the storage device (SD Card) but we want to mount the root directory from the nfs server. So edit it, remove the second block and modify the line of root as the following

Code:
timeout 10

menu title select kernel

label kernel-4.4.154-1124-rockchip-ayufan-ged3ce4d15ec1

    kernel /boot/vmlinuz-4.4.154-1124-rockchip-ayufan-ged3ce4d15ec1

    initrd /boot/initrd.img-4.4.154-1124-rockchip-ayufan-ged3ce4d15ec1

    devicetreedir /boot/dtbs/4.4.154-1124-rockchip-ayufan-ged3ce4d15ec1

    append rw panic=10 init=/sbin/init coherent_pool=1M ethaddr=${ethaddr} eth1addr=${eth1addr} serial=${serial#} cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1 root=/dev/nfs nfsroot=${serverip}:/storage/pxe-boot/rock64pro rootwait rootfstype=ext4

the only part to modify is the root= and the nfsroot= the rest of the file is the same as the extlinux.conf.
Finally, the SD card mounts the boot directory from a vfat partition while we are using our tftp boot, so we need to fix this.
The way is to edit the file /storage/pxe-boot/rock64pro/etc/fstab and remove the line about the boot (that should be the only one, so the final file will be empty).
Known Issues.
The reboot does't work always. It seems that the tftp server stucks and doesn't work in the next boot. I will try another tftp server.
The workaround is to force kill the tftpd and restart it.

If in the update process some component of the boot is upgraded, it is necessary to update the tftp boot folder and the pxelinux.cfg/default-arm-rockchip.

RE: Step by step guide PXE diskless configuration. - burglar_ot - 11-18-2018

It seems that tftp-hpa is very buggy and hangs randomly. I uninstalled it and installed the regular tftpd daemon that works fine.
The configuration in in /etc/inetd.conf.
The advantage of tftpd is that the boot directory can be a symbolic link to the boot folder under the linux distribution, so when an update is done the boot is automatically updated (but not the boot file that has to be changed manually).
The only difference with tftpd-hpa is that the path is now relative, so instead of kernel /boot/vmlinuz-4.4.154-1124-rockchip-ayufan-ged3ce4d15ec1 you have to use kernel boot/vmlinuz-4.4.154-1124-rockchip-ayufan-ged3ce4d15ec1 without the slash in front of boot. The same for the other boot parameters. The rest is unchanged.

RE: Step by step guide PXE diskless configuration. - tllim - 11-19-2018

Appreciated and thanks on the write-up, I just pin this thread so that forum user can notice.

RE: Step by step guide PXE diskless configuration. - burglar_ot - 11-25-2018

Cool, thanks. I add that every TFTP has problems, or it is a general issue with TFTP servers. After couple of boots the cards go in timeout during the boot time. I tried to debug in any possible way I know but the issue persists with any TFTP server and the only option is to restart the daemon.
I am thinking to recompile one TFTP server and run in a debugger to try to understand why it hangs, but I do not have a lot of free time right now. If someone here has a workaround or a fix it would be nice.

Another thing: I wasn't able to network boot any xELEC distribution (I tried Open, Core, Libre). The kernel boots but the NFS server is not mounted as root. If someone solved this issue I can extend the guide.

RE: Step by step guide PXE diskless configuration. - tllim - 11-30-2018

(11-25-2018, 01:48 PM)burglar_ot Wrote: Cool, thanks. I add that every TFTP has problems, or it is a general issue with TFTP servers. After couple of boots the cards go in timeout during the boot time. I tried to debug in any possible way I know but the issue persists with any TFTP server and the only option is to restart the daemon.
I am thinking to recompile one TFTP server and run in a debugger to try to understand why it hangs, but I do not have a lot of free time right now. If someone here has a workaround or a fix it would be nice.

Another thing: I wasn't able to network boot any xELEC distribution (I tried Open, Core, Libre). The kernel boots but the NFS server is not mounted as root. If someone solved this issue I can extend the guide.

If you need the spare PINE A64-LTS and ROCK64 board, please PM me your address with phone number (for shipping purpose), I will ship out to you.

RE: Step by step guide PXE diskless configuration. - burglar_ot - 12-21-2018

Finally, after many tentatives, I find a tftpd server that works perfectly without hanging on reboots or multiple cards booting at the same time.
Unfortunately is a pure python server and requires python 3.5. To install it it is possible to use

pip3 install py3tftp

and it has to be executed in the directory that has to be served. In the case of our guide is under /storage/pxe-boot/tftp.

RE: Step by step guide PXE diskless configuration. - burglar_ot - 01-08-2019

Now that I have a usb2serial to see the console I am able to boot other linux versions (I did the first totally blind and I have to say that I am very lucky). To boot the armbian is exactly the same procedure but the configuration file for the initial tftp has to point to the correct kernel. The relevant part of mine reads:

Code:
label kernel-4.4.166-rockchip64

   kernel boot-rock64pro/vmlinuz-4.4.166-rockchip64

   initrd boot-rock64pro/initrd.img-4.4.166-rockchip64

   devicetreedir boot-rock64pro/dtb-4.4.166-rockchip64

the rest is the same.

RE: Step by step guide PXE diskless configuration. - knyaz2020 - 04-02-2019

Seems it's working only on Rock64Pro... Tried on Rock64 - doesn't work, have issue with "UDP wrong checksum" Sad

Code:
TFTP from server 192.168.120.5; our IP address is 192.168.120.101

Filename 'pxelinux.cfg/01-22-62-24-29-72-b4'.

Load address: 0x600000

Loading: #

        72.3 KiB/s

done

Bytes transferred = 1049 (419 hex)

Config file found

1:      rock64

missing environment variable: bootfile

Retrieving file: boot/initrd.img-4.4.167-1161-rockchip-ayufan-g6f1664023387

Speed: 1000, full duplex

Using ethernet@ff540000 device

TFTP from server 192.168.120.5; our IP address is 192.168.120.101

Filename 'boot/initrd.img-4.4.167-1161-rockchip-ayufan-g6f1664023387'.

Load address: 0x4000000

Loading:  UDP wrong checksum 00000800 000022da

T  UDP wrong checksum 00001008 00003091

UDP wrong checksum 00001008 000022da

T  UDP wrong checksum 00001008 000022da

UDP wrong checksum 00001008 00003091

T  UDP wrong checksum 00000800 00003091

UDP wrong checksum 00000810 000022da

# UDP wrong checksum 00000008 00004c20

UDP wrong checksum 00000800 00003769

UDP wrong checksum 00000800 00003091

UDP wrong checksum 00000800 000022da

UDP wrong checksum 00000800 00008016

Maybe someone saw this issue and know how to solve?

P.S. Also created issue (full console output here): https://github.com/ayufan-rock64/linux-build/issues/341

RE: Step by step guide PXE diskless configuration. - burglar_ot - 04-03-2019

(04-02-2019, 11:22 AM)knyaz2020 Wrote: Seems it's working only on Rock64Pro... Tried on Rock64 - doesn't work, have issue with "UDP wrong checksum" :

Code:
TFTP from server 192.168.120.5; our IP address is 192.168.120.101 Filename 'pxelinux.cfg/01-22-62-24-29-72-b4'. Load address: 0x600000 Loading: # 72.3 KiB/s done Bytes transferred = 1049 (419 hex) Config file found 1: rock64 missing environment variable: bootfile Retrieving file: boot/initrd.img-4.4.167-1161-rockchip-ayufan-g6f1664023387 Speed: 1000, full duplex Using ethernet@ff540000 device TFTP from server 192.168.120.5; our IP address is 192.168.120.101 Filename 'boot/initrd.img-4.4.167-1161-rockchip-ayufan-g6f1664023387'. Load address: 0x4000000 Loading: UDP wrong checksum 00000800 000022da T UDP wrong checksum 00001008 00003091 UDP wrong checksum 00001008 000022da T UDP wrong checksum 00001008 000022da UDP wrong checksum 00001008 00003091 T UDP wrong checksum 00000800 00003091 UDP wrong checksum 00000810 000022da # UDP wrong checksum 00000008 00004c20 UDP wrong checksum 00000800 00003769 UDP wrong checksum 00000800 00003091 UDP wrong checksum 00000800 000022da UDP wrong checksum 00000800 00008016

Maybe someone saw this issue and know how to solve?

P.S. Also created issue (full console output here): https://github.com/ayufan-rock64/linux-build/issues/341

I use the same procedure on Rock64 and Pine64 LTS without any issue, so it seems strange. The UDP checksum is very suspicious for me and it seems related to the TFTP Server. I have to say that I had tons of problems with all the TFTP Server I tried except the one written in Python. What are you using as TFTP?

RE: Step by step guide PXE diskless configuration. - knyaz2020 - 04-03-2019

(04-03-2019, 08:36 AM)burglar_ot Wrote: I use the same procedure on Rock64 and Pine64 LTS without any issue, so it seems strange. The UDP checksum is very suspicious for me and it seems related to the TFTP Server. I have to say that I had tons of problems with all the TFTP Server I tried except the one written in Python. What are you using as TFTP?

I tried tftpd-hpa and ftpd (via xinetd) with different versions and on different boards (Raspberry Pi 3 B+ and from Rock64 board). In all cases have the same issue "UDP wrong checksum".
Could you give me URL on tftp server which written on Python?
Thank you.