The challenge before me is to bring up a mainline Debian install on the RockPro64. It will be a headless server with at least one SATA disk. I know there will be snags, but I'm hoping to work through them and maybe turn to you folks for help. I'll document what I discover along the way so others can use this as a guide.
Problem 1: (deferred) NAS case defects.
The first unit I received was warped or otherwise misshapen. (Maybe damaged in shipping?) The board's screw holes did not quite align with the case's standoffs, and the ports did not quite align with the case holes, so I could not mount the board.
The second unit I received has other problems, but at least the mounting holes are properly aligned, so I can move forward with the project. I will revisit the remaining case defects later.
Problem 2: (solved) Debian installer image.
The netboot installer daily build has been failing for the past week, so the link to the latest installer is (at the time of this writing) broken.
I browsed for earlier builds and used the last one that succeeded. Following the readme in that directory, I decompressed and concatenated the rockpro64 part with the generic part to form a complete SD card image, and wrote it to a microsd card using dd. This allowed me to proceed, but only briefly, as a related problem came up later. (See problem 4.)
Solution: It turns out the installer daily builds stop working whenever the Debian archive's kernel is upgraded, because the kernel module ABI no longer matches the version used by the installer. Submit a report to the debian-installer maintainers and/or contact them on irc to see if someone has time to update the installer, and then wait for a new daily build.
Problem 3: (workaround exists) Serial console.
Connecting my USB-to-TTL UART/serial adapter to pins 6, 8, and 10 of the rockpro64 (described here) established communication, but the serial line settings were wrong. At 115200 bps, booting yielded a little garbage, then several lines of readable text, then a lot more garbage. At 1500000 bps, only garbage was received for the entire boot sequence.
I found this report that the boot firmware runs the console at 115200 while the Debian installer runs at 1500000, which was helpful information, but it didn't explain why 1500000 bps didn't work at all.
It turns out the 1.5 Mbps speed is too fast for the CP2102 chip that is used on many USB-to-serial adapters, including mine. (My adapter was advertised as supporting that speed, but I suppose that was false advertising.)
Workaround: Reconfigure the installer. I mounted the SD card on my computer, opened the extlinux/extlinux.conf file in a text editor, and added this text on its own line at the end of the default boot stanza (the section that starts with "label l0"):
I was then able to open my serial port at 115200 bps, power up the rockpro64, see the text from the entire boot sequence (except for a little garbage before and after the initial bootloader messages), and operate the Debian installer.
I will order a new adapter, and I expect it will clear up most of the serial port garbage, but this won't change the fact that the 1500000 bps setting used by the bootloader and installer is too fast for about half the adapters on the market, meaning that the debian installer image will by default not work for many people. Either the speed should be reduced to 115200, or the instructions should be amended to explain the problem and the workaround. I may end up submitting a bug report to Debian about this.
Problem 4: (solved) No kernel modules found.
After configuring the language and network settings, and selecting a Debian mirror, the installer failed with a "No kernel modules found" message. The same error appeared with every mirror I chose. It also appeared with every available installer image.
Just like the daily build problem that I mentioned earlier, this happens whenever the Debian archive's kernel is upgraded, because the module ABI no longer matches the version used by the installer. When it happens, all of the installer daily builds are broken, so it is pointless to try the others.
Solution: Report the problem and wait for the maintainers to update the installer to use the new kernel.
Problem 5: (workaround exists) The installer does not install a bootloader.
The installer finished setting up the /boot and / (root) filesystems, but it did not even try to install a bootloader of any kind. That's a critical problem for an operating system installer. I suppose that part of supporting the RockPro64 simply isn't done yet.
The debian u-boot-rockchip package looks like it is meant to solve this issue, but I haven't tried it yet because it doesn't yet recognize the RockPro64 v2.1 board revision. I'll probably revisit it later.
Workaround: Until I learn how to build and install the required u-boot components on a fresh SD card, I can boot from the card that contains the debian installer image. Here's what I did to make it work:
With those modifications to the installer SD card, the RockPro64 will boot from it, launch the kernel and initrd from the new partition, and bring up the installed OS. I believe this works because the installer SD card image already contains the u-boot components needed for boot, in the correct disk locations preceding the first partition.
It has been suggested that a custom script in /etc/kernel/postinst.d/ could copy the latest .dtb file to the /boot directory (which lives on SD card partition 2) whenever a new kernel is installed. I like this idea, but have not tried it yet.
Problem 6: (workaround exists) Debian stable/buster does not support the RockPro64.
I want to take advantage of Debian's stable release, but only the unstable suite supports this board for now, so the installer uses packages from unstable and configures the installed system to do the same.
Workaround: Preseed the installer to use the current stable suite (named buster) instead of its defaults, and then use apt pinning to configure the installed system to use the unstable suite for kernel packages.
With those changes in place, the installed system is entirely Debian stable, except for the kernel. OS updates will work just like a vanilla Debian stable system.
Problem 7: (solved) Network services try to start before the network is up, and fail.
Any network daemons configured to listen only on the eth0 interface may sometimes fail to start, because systemd attempts to launch them without waiting for eth0 to initialize. (The problem may not show up with default network daemon configurations, since those often listen on all interfaces and therefore do not require one to be ready before the daemon starts.)
This happens because Debian's installer sets up eth0 as a hotplug interface, leaving it disconnected from systemd network targets and dependency chains.
Solution: Edit /etc/network/interfaces and add this line just above the "allow-hotplug eth0" line:
I may add more here when I continue the project.
Problem 1: (deferred) NAS case defects.
The first unit I received was warped or otherwise misshapen. (Maybe damaged in shipping?) The board's screw holes did not quite align with the case's standoffs, and the ports did not quite align with the case holes, so I could not mount the board.
The second unit I received has other problems, but at least the mounting holes are properly aligned, so I can move forward with the project. I will revisit the remaining case defects later.
Problem 2: (solved) Debian installer image.
The netboot installer daily build has been failing for the past week, so the link to the latest installer is (at the time of this writing) broken.
I browsed for earlier builds and used the last one that succeeded. Following the readme in that directory, I decompressed and concatenated the rockpro64 part with the generic part to form a complete SD card image, and wrote it to a microsd card using dd. This allowed me to proceed, but only briefly, as a related problem came up later. (See problem 4.)
Solution: It turns out the installer daily builds stop working whenever the Debian archive's kernel is upgraded, because the kernel module ABI no longer matches the version used by the installer. Submit a report to the debian-installer maintainers and/or contact them on irc to see if someone has time to update the installer, and then wait for a new daily build.
Problem 3: (workaround exists) Serial console.
Connecting my USB-to-TTL UART/serial adapter to pins 6, 8, and 10 of the rockpro64 (described here) established communication, but the serial line settings were wrong. At 115200 bps, booting yielded a little garbage, then several lines of readable text, then a lot more garbage. At 1500000 bps, only garbage was received for the entire boot sequence.
I found this report that the boot firmware runs the console at 115200 while the Debian installer runs at 1500000, which was helpful information, but it didn't explain why 1500000 bps didn't work at all.
It turns out the 1.5 Mbps speed is too fast for the CP2102 chip that is used on many USB-to-serial adapters, including mine. (My adapter was advertised as supporting that speed, but I suppose that was false advertising.)
Workaround: Reconfigure the installer. I mounted the SD card on my computer, opened the extlinux/extlinux.conf file in a text editor, and added this text on its own line at the end of the default boot stanza (the section that starts with "label l0"):
Code:
append console=ttyS2,115200n8
I will order a new adapter, and I expect it will clear up most of the serial port garbage, but this won't change the fact that the 1500000 bps setting used by the bootloader and installer is too fast for about half the adapters on the market, meaning that the debian installer image will by default not work for many people. Either the speed should be reduced to 115200, or the instructions should be amended to explain the problem and the workaround. I may end up submitting a bug report to Debian about this.
Problem 4: (solved) No kernel modules found.
After configuring the language and network settings, and selecting a Debian mirror, the installer failed with a "No kernel modules found" message. The same error appeared with every mirror I chose. It also appeared with every available installer image.
Just like the daily build problem that I mentioned earlier, this happens whenever the Debian archive's kernel is upgraded, because the module ABI no longer matches the version used by the installer. When it happens, all of the installer daily builds are broken, so it is pointless to try the others.
Solution: Report the problem and wait for the maintainers to update the installer to use the new kernel.
Problem 5: (workaround exists) The installer does not install a bootloader.
The installer finished setting up the /boot and / (root) filesystems, but it did not even try to install a bootloader of any kind. That's a critical problem for an operating system installer. I suppose that part of supporting the RockPro64 simply isn't done yet.
The debian u-boot-rockchip package looks like it is meant to solve this issue, but I haven't tried it yet because it doesn't yet recognize the RockPro64 v2.1 board revision. I'll probably revisit it later.
Workaround: Until I learn how to build and install the required u-boot components on a fresh SD card, I can boot from the card that contains the debian installer image. Here's what I did to make it work:
- Tell the installer to create the /boot partition on the same SD card that contains the installer image. This becomes partition 2 on that card.
- Set the bootable flag on partition 2.
- After installing, create the boot/dtbs/rockchip directory on partition 2.
- Copy boot/dtbs/rockchip/rk3399-rockpro64-v2.dtb from partition 1 (the installer partition) to the same directory on partition 2.
- Create the /extlinux directory on partition 2.
- Create an extlinux/extlinux.conf file in that directory on parition 2, with this content: (The value after root= must match what the installer says in its "You will need to boot manually" message.)
Code:
menu title Debian
prompt 0
timeout 50
label l0
menu label Debian
linux /vmlinuz
append root=<your-root-device>
initrd /initrd.img
fdt /dtbs/rockchip/rk3399-rockpro64-v2.dtb
With those modifications to the installer SD card, the RockPro64 will boot from it, launch the kernel and initrd from the new partition, and bring up the installed OS. I believe this works because the installer SD card image already contains the u-boot components needed for boot, in the correct disk locations preceding the first partition.
It has been suggested that a custom script in /etc/kernel/postinst.d/ could copy the latest .dtb file to the /boot directory (which lives on SD card partition 2) whenever a new kernel is installed. I like this idea, but have not tried it yet.
Problem 6: (workaround exists) Debian stable/buster does not support the RockPro64.
I want to take advantage of Debian's stable release, but only the unstable suite supports this board for now, so the installer uses packages from unstable and configures the installed system to do the same.
Workaround: Preseed the installer to use the current stable suite (named buster) instead of its defaults, and then use apt pinning to configure the installed system to use the unstable suite for kernel packages.
- Mount the installer SD card on a computer, and create a file called preseed.cfg in the root directory, containing this text:
Code:
#_preseed_V1
d-i mirror/suite string buster
- Tell the installer about the preseed configuration by appending preseed/file=/media/preseed.cfg to the kernel command line. Together with the argument we added earlier for the serial console speed, the append line should look like this:
Code:
append preseed/file=/media/preseed.cfg console=ttyS2,115200n8
- Boot from the installer SD card and proceed through the installation process, but stop at the "Finish the installation" screen. Do not press the <Continue> button yet.
- Switch away from the installer's virtual screen to a terminal screen, by pressing Control+A and then 2.
- Start a shell in the newly installed filesystem (which is already mounted at /target) by running these commands:
Code:
mount -t proc none /target/proc
mount --rbind /sys /target/sys
chroot /target bash
- Add these lines to the end of /etc/apt/sources.list:
Code:
# A few components from unstable are needed to boot the RockPro64
deb http://deb.debian.org/debian/ unstable main
deb-src http://deb.debian.org/debian/ unstable main
- Create a file called /etc/apt/preferences.d/unstable-rockpro64.pref containing this text:
Code:
Explanation: Avoid most packages from unstable
Package: *
Pin: release a=unstable
Pin-Priority: 1
Explanation: RockPro64 kernel packages from unstable
Package: linux-image-* linux-headers-*
Pin: release a=unstable
Pin-Priority: 500
- Upgrade the newly installed system's kernel packages to the latest available from the unstable suite, using these commands:
Code:
apt update
apt upgrade
- Exit the installed filesystem environment:
Code:
exit
- Return to the installer's virtual screen, by pressing Control+A and then 1.
- Finally, press the <Continue> button on the "Finish the installation" screen.
With those changes in place, the installed system is entirely Debian stable, except for the kernel. OS updates will work just like a vanilla Debian stable system.
Problem 7: (solved) Network services try to start before the network is up, and fail.
Any network daemons configured to listen only on the eth0 interface may sometimes fail to start, because systemd attempts to launch them without waiting for eth0 to initialize. (The problem may not show up with default network daemon configurations, since those often listen on all interfaces and therefore do not require one to be ready before the daemon starts.)
This happens because Debian's installer sets up eth0 as a hotplug interface, leaving it disconnected from systemd network targets and dependency chains.
Solution: Edit /etc/network/interfaces and add this line just above the "allow-hotplug eth0" line:
Code:
auto eth0
I may add more here when I continue the project.