NAS Optimizations for RockPro64

[MNtinkerer]

I am curious if anyone who has deployed the RockPro64 as a NAS have any optimizations that has improved system performance. Especially if they are using a PCIe to SATA adapter. I would like to find/develop a script to reduce the burden of the initial portion of a data sync or even just listing files. Whether as a SAMBA server, local file listing, or during the initial portion of a data backup, the CPU seems to run at max until the file attributes list is populated. Although other optimizations that are useful to you would be of interest too, this doesn't have to be specific to one OS, or one particular filesystem, back up scheme, or system admin maintenance task. Basically what is useful to you that might benefit someone else.

I have 2 RockPro64 deployed as NAS, one hosting Debian 10 and the other FreeBSD 13.0. I have configured it many different ways but both are using 3rd party PCIe to SATA adapters each hosting 3 HDDs. One is a kludge of 1TB disks both 3.5" and 2.5" with RAIDz1 the other is 3 identical 2.5" HDDs in RAIDz1. Other than having SAMBA configured on them, they are currently backups of my main server, which is also my desktop workstation. I have recently been getting into ZFS send/recieve for backups, but have more experience with rsync. ZFS is an improvement but still tedious. Simple commands like ls to list files in a directory will consume CPU cycles to the point of nearly halting the system, and I'm thinking having a scheme of some sort will improve this, that is durable across reboots.

The only reference I found to NAS optimizations on this forum was from a user who appears to no longer be active and they only referenced caching, but did not describe in detail what they did.

[foresto]

I don't use ZFS and I suspect my use case is much lighter than yours, so I don't think I can help, but a couple things come to mind:

Are you encrypting your disks? If so, could some of that work be offloaded to the RK3399's ARMv8 cryptography extensions by using the right configs & linux kernel module?

Could the driver for your PCIe/SATA card be contributing to CPU load? If so, perhaps switching to a card with a different chip could help? (Mine has a Marvell 88SE9235.)

[MNtinkerer]

I don't use ZFS and I suspect my use case is much lighter than yours, so I don't think I can help, but a couple things come to mind:

Are you encrypting your disks? If so, could some of that work be offloaded to the RK3399's ARMv8 cryptography extensions by using the right configs & linux kernel module?

Could the driver for your PCIe/SATA card be contributing to CPU load? If so, perhaps switching to a card with a different chip could help? (Mine has a Marvell 88SE9235.)

Encryption is a good point. I don't understand enough about it at the kernel level, just how to use LUKS to lock an entire partition or ZFS encryption for a dataset.

Drivers seemingly are not the issue on PCIe as behavior is consistent despite operating system or any of the PCIe chipsets I have. This includes Marvell 88SE92xx which has had mainline Linux kernel support since kernel version 5. There are compile-time options for many drivers which is also worth exploring. Just not there yet with my Linux skill set.

[enorbet2]

Hello
I'm running the RockPro64 on several systems (OpenVault and SuSeTumbleweed) but my main "go to" system is Arch64Slackware. I'm using a Marvell chipset 4 port PCIe SATA controller supporting an SATA SSD for /root as well as 2 Seagate spinners with BTRFS in RAID1. In Slackware rsync never pegs CPU or RAM but I am getting some sort of bottleneck as transfer rates are not what I expect. I suspect it has something to do with the ethernet controller since I get the same behaviour on every system. When I run a simple online fiber benchmark I get odd results in which Upload is higher than Download, the opposite of what it should be and is on 5 other PCs.

I did get some minor benefits from tweaks I tried from - https://www.nas.nasa.gov/hecc/support/kb...x_138.html but I can't say it was substantial, again likely from some problem with the ethernet controller. YMMV though... worth a shot

[vongillus]

I would say that probably ZFS will be very CPU intensive for the rockpro64 but I never used it myself. I plan to use mdraid 1 +ext4 on a 2 ports SATA controller with Debian running on the emmc. The issue right now for me, is that I was not yet able to make a successful boot into the OS. I have posted another thread about this, trying to install u-boot in spi or tow-boot but somehow it doesn't properly recognize the emmc. I just did a normal Debian install booting from the sd card and the installation process looks good. Any hint welcome! Like this I could contribute to this thread with my configuration

[enorbet2]

Part of the installation process for Aarch64Slackware replaces the initial boot process. With it I can edit the boot menu to show multiple boot options or simply allow boot priority on RockPro64 to boot emmc 1st or default to a different MicroSD card system if no emmc is p[resent. It's very flexible. Currently I can boot emmc, MicroSD, or MicroSD handoff to SATA. I now have 4 systems installed I can easily boot.

Aarch64Slackware has a very helpful YouTube channel that will walk you through the process. You can elect to install the entire system or just the boot replacement but I suggest you go all the way since by default Aarch64 Slack has the newest full support kernel I've yet seen. It's very good.

[MNtinkerer]

Both the NASA networking optimizations and the Aarch64Slackware are interesting, and will try both suggestions out! Thank you.

Other than the indexing issue described above, I haven't experienced filesystem overhead that has especially taxed my RockPro64s. ZFS can be memory hungry but hasn't topped out the 4G of memory yet. Then again I do not have an exotic setup and use case. Of course I haven't done hard bookmarking, rather just monitored system resources from time to time when noticing lags.

[enorbet2]

Heya MNtinkerer! I thought it might be useful to mention that the 2 main guys developing. distributing, and supporting Aarch64Slackware (the head guy is Stuart whom you may see on the YouTube channel) boith post and respond on the LQN Linux ( https://www.linuxquestions.org/questions...e-arm-108/ ) message boards as well as the YouTube channel.

Stuart is in the UK and the second dev, who goes under "mralk3" on LQN (i sadly forget his name, maybe Brent?) from the US, are both extremely experienced and educated as well as helpful. There is a competing platform called SLARM that is easier to install but less flexible, but his packages work a treat even on Aarch64Slack!

Regarding installation, at first I was hesitant to install Aarch64 Slack because of so many more steps but Stuart did a great job of step-by-step instructions that can be copied and pasted and it goes very fast on a decent connection. It's also very easy to keep up to date whether manually or via slackpkg.

Certainly I am biased, being a 20+ year Slackware guy, but just the newest kernel alone makes Slackware on ARM a better choice than any other system I've tried, all of which required manual updates to get a new for 2022 kernel.

Good Fortune!

[MNtinkerer]

Thank you enorbet2. You sure are active over in the Linuxquestions forum! I haven't gotten very far with slackware, it is throwing the error included below. Not sure how to proceed but will go to sleep and work on this sometime tomorrow. I did ask in the LQ Slackware for Arm forum, but am pending moderator approval of the post. Good day to you!


rsync -PavL $SLKSRV/platform/aarch64/bootware/installer/slackwareaarch64-${SLKVER}/rk3399_generic.sdimg_latest.img.xz .
rsync -PavL $SLKSRV/platform/aarch64/bootware/installer/slackwareaarch64-${SLKVER}/rk3399_generic.sdimg_latest.img.xz.asc .
--------------------------------------------------------------
The features compression (-z) and checksums (-c) are disabled.

More information about this server is available via HTTPs:
https://ftp.halifax.rwth-aachen.de/
--------------------------------------------------------------

receiving incremental file list
rsync: [sender] change_dir "/platform/aarch64/bootware/installer/slackwareaarch64-current~" (in slackwarearm) failed: No such file or directory (2)

sent 8 bytes received 168 bytes 117.33 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1835) [Receiver=3.2.4]
--------------------------------------------------------------
The features compression (-z) and checksums (-c) are disabled.

More information about this server is available via HTTPs:
https://ftp.halifax.rwth-aachen.de/
--------------------------------------------------------------

receiving incremental file list
rsync: [sender] change_dir "/platform/aarch64/bootware/installer/slackwareaarch64-current~" (in slackwarearm) failed: No such file or directory (2)

sent 8 bytes received 168 bytes 117.33 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1835) [Receiver=3.2.4]

[enorbet2]

Heh! Just looks that way. I've been there a very long time. I don't recognize those errors and I can't recall if I contacted Stuart originally through YouTube comments or via some link he posted, but I do recall he helped me poste haste. Later I asked for help on a number of concerns (I'm just so n00b at ARM) and one concern was that at one time USB was disabled briefly during startup of UBoot to solve some conflict but of course that put the kaibosh on any way to select a boot entry in a multiboot environment. It turned out I actually didn't need the menu as I could select devices just by whether or not I plugged in an emmc drive. Despite that, the US fellow (the guy I thought might be named something like Brent, mralk3) not only gave me great information he modded the UBoot to support USB continuously (the starter concern was no longer a concern) and uploaded it to me.

Needless to say, I strongly suspect you will get solid quick support. I experience a couple errors with first install but after Stuart contacted me letting me know there was an updated location, everything was smooth sailing. It really is a terrific community.