k3os

[merquerio]

Hi there, first post here, so any recommendations are more than welcomed.

0. Context

I wanted a home server as minimalist as possible, to lower the maintenance burden, so I bought the RockPro64 and it arrived January 20th.

We use Kubernetes at work so I wanted something similar that is easy to operate and maintain. I found k3os a Linux distro that it's single purpose is to run Kubernetes. This will allow me to focus on deploying the Helm charts we need at home and keeping the data safe (redundancy, backups ...).

I open the thread to share my experience as I go through the process of getting everything up and running.

1. Installing k3os

As far as I understand the RockPro64 cannot be run using the upstream kernel (please correct me if I'm wrong.). So I decided to look for a distro that supported Docker out of the box (just in case). Thanks to @ayufan, there is a Ubuntu (Bionic) image with Docker support available here. So I decided to test it, burning it into the eMMC (the microsd card boots too).

Code:

# Extract the image
unxz bionic-containers-rockpro64-0.9.16-1163-arm64.img.xz

# Burn the image to the eMMC
dd if=bionic-containers-rockpro64-0.9.16-1163-arm64.img of=/dev/sdc bs=10M

I don't have any external monitors, so SSH access was key for me. This image starts a SSH server that I found easily on my network (eth access).

Code:

# pass: rock64
ssh rock64@192.168.1.20

Once inside I tested that Docker was running correctly, that the PCIe Dual SATA-II Interface Card was correctly detected and that the 2x HDD 1TB drives were there. In fact I found out that one of the drives is dead (but replacement is on it's way). I prepared the drive, but I will go in more details as soon as I receive the replacement.

Getting k3os to work was surprisingly easy. They provide a takeover installation that works like a charm. First you need to create the configuration file `config.yaml` that will set all the necessary stuff for the host OS. In fact, all the host OS should be done here. The details about this file are available here. In my case, for now, mounting the HDD was enough.

Code:

ssh_authorized_keys:
 - "ssh-rsa <the content of your id_rsa.pub"
hostname: nucloud
write_files:
 - content: |-
     /dev/sda1      /var/k8s-storage      ext4    auto      0 0
   owner: root
   path: /etc/fstab
   permissions: '0644'
k3os:
 labels:
   whatever: you-want
 dns_nameservers:
   - 192.168.1.1 # update this to your local or public DNS server if needed
 ntp_servers:
   - hora.rediris.es
 password: <pass>
 token: <token>

Then we download the rootfs and place in the root of our fresh install:

Code:

curl -sfL https://github.com/rancher/k3os/releases/download/v0.8.0/k3os-rootfs-arm.tar.gz | tar zxvf - --strip-components=1 -C /
cp myconfig.yaml /k3os/system/config.yaml
sync
reboot -f

And now our system will boot into k3os. This method places k3OS on disk and also overwrites `/sbin/init`. On next reboot the bootloader and kernel should be loaded, but then when user space is to be initialized k3OS should take over.


2. Access the Kubernetes node

Now that we have our system up and running, further access to the RockPro64 is not mandatory. k3os install k3s, a kubernetes lightweight distribution, and from now on we can operate the single node "cluster" remotely. To gain access from your laptop/desktop using kubectl, you just need to copy the access credentials and add the to your `~/.kube/config`:

Code:

scp rancher@192.168.1.20:/etc/rancher/k3s/k3s.yaml .

# Edit your ~/.kube/config to add the content of k3s.yaml or extend your $KUBECONFIG:

mv k3s.yaml ~/.kube
echo "export KUBECONFIG=$KUBECONFIG:~/.config/kube/k3s.yaml" >> .bashrc

ø. (WIP)

- Launch all the Helm charts
- Encrypt local data.
- RAID (software?) vs LVM vs ZFS, suggestions are more than welcomed
- Backup data locally and to a remote node (encrypted)
- Provide DDNS access to the kubernetes ingress

[plod]

How did you get on with this, I just bought my rockpro64 for my home k3s cluster so intend to do exactly the same