ROCKPro64 with 16 ports SATA controller

[ZeblodS]

Hi all,

I want to use that ROCKPro64 system, with an Ubuntu server or Debian, as a 10 to 12 SATAIII harddrives NAS.


The idea is to use this controller card, which is a PCI-e 2.0 4x card with Marvell 88SE9215 chipset (1 PCI-e lane to 4 SATAIII per chipset).

I know that technically, the Marvell 88SE9215 is working on the ROCKPro64, but this controller board also use another type of chip, and I cannot find-out what they are exactly...

I you look at a close-up picture, you can clearly see that each of the 4 PCI-e lanes goes to the bigger chip on the center, then to the smaller four chips and then to the mini-SAS connectors (and then mini-SAS to 4 SATAII). But the 88SE9215 chip only takes a single PCI-e lane according to its datasheet, so I suspect that board to use four 88SE9215 chips and some kind of PCI-E lanes splitter in between to give each 88SE9215 a single PCI-e lane.

Does anyone have more information? Am I wrong? And more importantly, does this card works on the ROCKPro64?


For more info, I want to replace my old Core2duo based NAS, and I currently have 10 harddrives of 3TB to manage. I use ZFS-on-linux, which seem to work on ROCKPro64, with a 10 discs RAID-Z2. I know the datarate won't be extraordinary high, but I only need it to saturate the gigabit Ethernet with Samba shares, so about 100MB/s top.


[EDIT] Title and typos...

[stuartiannaylor]

http://www.iocrest.com/en/product_details440.html

Prob a picex4 to pciex1 packet switch with basically 4x Marvell 88SE9215

With most decent large HD having approx 200MBs still quite a bottleneck and with the overhead of 10 disk RAIDZ2 I doubt you will even get 100MBs per disk.
That is if its recognised and works.

[ZeblodS]

http://www.iocrest.com/en/product_details440.html

Prob a picex4 to pciex1 packet switch with basically 4x Marvell 88SE9215

With most decent large HD having approx 200MBs still quite a bottleneck and with the overhead of 10 disk RAIDZ2 I doubt you will even get 100MBs per disk.
That is if its recognised and works.

I understand very well that I'd probably have less than 100MB/s per disk with that controller, but as I said it's a RAIDZ2 (stripped data on 8 disks with two redundancy) so it still gives me about 800MB/s data speed on the ZFS datapool (if the RK3399 is powerful enough to deal with that much data computing the redundancy and blocs checksums...).

It will be a NAS, and only a NAS as every other services such as webserver will be deported onto several dedicated Raspberry Pi 4, connected on my gigabit home network. That means I technically only need about 100MB/s data speed on the datapool. So the controller will be more than enough to sustain gigabit data transfers.

The real question is, is that specific SATA controller working properly on a ROCKPro64? Or have someone a better option for connecting 10 to 12 SATA on the ROCKPro64? I'd like to be sure it works before spending several hundreds of dollars on that setup... 


For information, my current setup is an old (from 2007) Tyan motherboard with a PCI-X SATA controller supporting all the drives, which has a total bandwidth four times smaller that the PCI-E 2.0 4x... The local access onto the ZFS datapool is about 250MB/s, and it's still enough to saturate my gigabit link. The setup with the ROCKPro64 can only be better than that, even with the bottleneck of the SATA controller.

[ZeblodS]

Well, I just ordered a 4GB ROCKPro64 with a 64GB eMMC module and a few accessories (heatsink, USB adapter, CMOS battery). I guess the only way to be completely certain it works is to try for myself... Now I hope it won't take forever to receive the order from Pine64 store.

[stuartiannaylor]

Pretty sure there is an 80% chance that card will work from the box. The resultant speed of all that individual sata control and stripe hashing means I haven't got a clue of performance.
Pretty sure the is an 100% chance that with a kernel recompile that card will work as the RK3399 has a complete root hub and there are quite a few PCIe configs that could be bumped to modules especially a kernel with debug ones.
I really don't know why we don't have in the packages a debug kernel and headers alternative for those who want to blaze a path without being so blind.

The card is basically a x4 to 4x x1 lane pcie packet switch with x4 marvell 9215 sata controllers on it.
Its just basically 4 cards in one and we know the Marvell card works well.

Strangely I have just been having a conversation that what we need is a PCIe x4 to 4x PCIe x1 packet switch as the actually packet switch silicon costs little more than the $15 bitminer ones you see on ebay.
https://www.ebay.co.uk/itm/New-4-Ports-P...3442792757
Just the bandwidth on those sucks as 1 lane is split into 4 but the difference silicon wise is about $5 for the x4 lane to 4x x1.

We are in a strange position as the PC world with many slots doesn't apply value to a PCIe packet switch as they just use another slot.
Also with the advent of pcie4/5 who wants cheap automotive pcie2.0 packet switches.

I completely believe the ArmSoc SBC world absolutely needs a x4 to 4x x1 PCIe packet switch so that single card can be use with multiples of the common PCIe x1 cards we know work.
In the SBC embedded work we need dumb bios-less simple cards and we are onto a winner and its a problem because often we are excluded from pciex2/x4 cards because they are not as dumb and simple as their pciex1 brothers with the inclusion of bios or spi configs.

If that card has a bios or spi config then the answer is no, but for £160 as I have seen them on sale for its likely its just a standard PCIe packet switch which is near identical in operation to a network switch in operation.
They are seamless  operation and if it does have a bios there is very little it can control and you have been extremely unlucky.

The pcieroot hub and all pcie root hubs are a collection of lanes aggregated into a LAG (Link aggregation group) and apart from with networks you usually group ports to a LAG with PCIE you just separate a LAG into individual ports. Essentially the same transparent to the protocol layer but networks build into LAGs whilst a PCIe slot is a LAG that needs to be split to ports.

PINE!!! we need a PCIe x4 to 4x PCIe packet switch that can be used with multiples of any cheap PCIe x1 card from Sata, USB to Ethernet or whatever crazy idea you have.
That packet switch has a lot of value as it can be used in multiple configurations and negates the need for rare specialist cards like the above sata which purely because they are rare and specialist hold high cost.

There is another way to go and at first sounds a bit bat shit crazy but far less likely to have bios / spi config and work from the box.
At half the price of the sata card you can get a USB 4 port
https://www.scan.co.uk/products/4-port-s...ta-lp4-pow

or a 2 port for as cheap as $25 and use that with the x2 USB 3.0 that the RockPro64 comes with.

If your storage is for a Media server of large relatively static files then actually this is an extremely valid method and no less than Zraid2.

Set up SnapRaid with 2x parity disks put those on the RockPro64 ports split your data disks via 2x hubs and use OMV (Open Media Vault) SnapRaid & UnionFS to create an asynchronously applied parity system of a JBOD MergerFS volume.
https://michaelxander.com/diy-nas/
https://www.youtube.com/watch?v=FYkdPyCt5FU

Its a definate no to synchronous RAID of file server / database / web server use but for large static file stores its as good as any and actually is cheaper and less prone to hit the bios / spi config barrier of cards with internal setup function that likely do not have linux drivers.

PINE!!! we need a PCIe x4 to 4x PCIe x1 packet switch!!!

[ZeblodS]

I have looked for a PCI-E x4 to 4x PCI-E x1, or even PCI-E x4 to 2x PCI-E x2, I think I looked about everywhere for several days. Indeed, those kind of cards just don't exist at all...

I have also looked for used PCI-E SATA controller cards, but all models with 12 to 16 SATA ports are RAID cards with a BIOS (and RAM and battery) that won't work for sure here.
Except for this specific IOCrest SI-PEX40097 which looks like a dumb SATA controller, but seems to be too recent to be on the used market yet.

I guess, I'll end up buying one from Amazon, it's a bit more expensive but at least if it doesn't work properly I can still send it back, no question asked.

Anyway, the build won't start before the ROCKPro64 arrives, which seem to be within 30 to 45 days according to Pine64 store shipping policy.

That leaves me plenty of time for planning the build on itself.
Right now, I'm thinking going the DIY way with my 3D printer for most part of the casing.
Using something like twelve ORICO 1106SS and printing independent stackable enclosures to put these in, open on the back for airflow.
Stacking the hard drives 2 wide and 6 high (about 300mm wide, 260mm high and 200mm deep), and putting a regular ATX power supply on top on one side and the ROCKPro64 and SATA controller on the other side (an extra 100mm high maybe) on a specific printed enclosure too.
That would allow me to use six 120mm fans on the back to cool the entire system (2 fans wide, and 3 fans high, which represent 360mm high). I could put the connections on the sides with recessed connectors, or not I don't know yet.
And I'll need to put together some kind of front panel with deported buttons to start/stop the power supply, the ROCKPro64, etc. EasyEDA and JLCPCB are most likely my friends on this one.
I'll try to 3D model a few ideas with Fusion360 in the next few days/weeks to see how those ideas could be implemented.

[stuartiannaylor]

I have looked for a PCI-E x4 to 4x PCI-E x1, or even PCI-E x4 to 2x PCI-E x2, I think I looked about everywhere for several days. Indeed, those kind of cards just don't exist at all...

I have also looked for used PCI-E SATA controller cards, but all models with 12 to 16 SATA ports are RAID cards with a BIOS (and RAM and battery) that won't work for sure here.
Except for this specific IOCrest SI-PEX40097 which looks like a dumb SATA controller, but seems to be too recent to be on the used market yet.

I guess, I'll end up buying one from Amazon, it's a bit more expensive but at least if it doesn't work properly I can still send it back, no question asked.

Anyway, the build won't start before the ROCKPro64 arrives, which seem to be within 30 to 45 days according to Pine64 store shipping policy.

That leaves me plenty of time for planning the build on itself.
Right now, I'm thinking going the DIY way with my 3D printer for most part of the casing.
Using something like twelve ORICO 1106SS and printing independent stackable enclosures to put these in, open on the back for airflow.
Stacking the hard drives 2 wide and 6 high (about 300mm wide, 260mm high and 200mm deep), and putting a regular ATX power supply on top on one side and the ROCKPro64 and SATA controller on the other side (an extra 100mm high maybe) on a specific printed enclosure too.
That would allow me to use six 120mm fans on the back to cool the entire system (2 fans wide, and 3 fans high, which represent 360mm high). I could put the connections on the sides with recessed connectors, or not I don't know yet.
And I'll need to put together some kind of front panel with deported buttons to start/stop the power supply, the ROCKPro64, etc. EasyEDA and JLCPCB are most likely my friends on this one.
I'll try to 3D model a few ideas with Fusion360 in the next few days/weeks to see how those ideas could be implemented.

I saw one of your cards on aliexpress I think for £160 Arrgh but cheaper than I have seen em.

Going cheap I go for these https://www.ebay.co.uk/itm/Hard-drive-Ca...4116108483
I use these sata power chains https://www.ebay.co.uk/itm/1Pc-4pin-ide-...4339041268
Use the 12v 60watt I have for the rock and splitters ,these are tiny bucks and go inline with the 12v on the 5 way chain. https://www.aliexpress.com/item/32742116421.html
With that amount of disks without the math but guestimate you prob better off with 90watt, maybe more. 2amp short write? 3.5"? totally forgot could be well out.
Run those off splitters via https://www.ebay.co.uk/itm/10-Female-5-5...2275763712
https://www.ebay.co.uk/itm/DC-Verteiler-...3757922865

This was the cheapest I noticed https://www.aliexpress.com/item/32546377052.html

But you do always have the USB option if for media?

"PCI-E x4 to 4x PCI-E x1, or even PCI-E x4 to 2x PCI-E x2"
They do exist but problem is they are end of line server components offered now as speciality components at high cost.
PCIe 2.0 with 4/5 now coming mainstream and PCs generally always having another slot means they are going to stay that way.
I really do feel just like the automotive industry who are using PCIe 2.0 as cheap proven high speed datbus.
The silicon is out there the design is simple and once more Pine!!! we need a PCIex4 to 4x PCIex1 packet switch!!!  

If you look all this stuff is no EOL http://www.ioi.com.tw/products/prodcat_a...ostID=2095
The silicon is nothing and all is needed is a card and with so many ArmSoc SBC looking like PCIe2.0 x4 could be a common form for some time.
http://amfeltec.com/products/flexible-x4...-splitter/ shelf hoarders trying to get a buck from those who are stuck.

[ZeblodS]

Thanks for the links, I'll look into it.

USB is the fall back solution. I'd rather stay with what I already know works well.

I've been using my current NAS with ZFS for over a decade (I've built it late 2008... running with OpenSolaris at the time when Sun Microsystems still existed).
It has survived several disks failures, power supply failure, power outages, etc. without loosing a single data.
I know ZFS with RAID-Z2 is rock solid at keeping data, even through several architectures: used on SunOS, FreeBSD, illumOS, Linux over the years, still the same data-pool created in 2008, imported and upgraded with larger disks every 3 to 4 years.

Daily snapshots (keeping one for the last 7 days, last 7 weeks and last 7 months) saved my life more than once, digging up a deleted file on the fly with very little wasted space. Copy-on-write also saved me more than once with power outage during modification on files. The NAS has been ON 24/7 since I first started it up, except for maintenance and replacing power supply once.

And I want to upgrade the hardware to something newer (and less electricity hungry) but still keep the same usability as the last ten years. I'm not willing to spend thousands, but a few hundreds is fine if it last me for the next decade.

[stuartiannaylor]

I would have a think about the USB solution if I was you as Snapraid isn't raid its just a bunch of disks with specific disks for parity that essential is as redundant as Raid.
I think if I knew more about RAID its just asynchronous distributed RAID4  that has had some good development.

The thing about USB in conjunction with Snapraid and static large media files is that each disk is separate so they are not all spinning at the same time, just the ones you are accessing or the ones doing a parity sync usually scheduled for a late night.
Not having the striping and all the enterprise wonderfulness of ZFS Zraid2 for large files doesn't mean its has any advantage.

For file, web & database servers then ZFS Zraid or other Raid mechanisms are essential and USB is an absolute no.
Everything has a fit for purpose and energy wise and many of distribution for static large media file repositories is perfect for USB and actually at least energy wise is pretty damn bad with ZFS.

12v 3.5" UAS USB3.0 adapters are like $5.00 and add a card with 4 ports and the 2 you have you would have a similar arrangement of 2x parity disks and 8x data disks that only spin up when they are specifically accessed so its massively more energy efficient by simplicity of design.

That https://www.cclonline.com/product/141126/PEXUSB3S42V/USB-Firewire-Cards/StarTech-com-4-Port-Dual-Bus-PCI-Express-PCIe-SuperSpeed-USB-3-0-Card-Adaptor-with-UASP-SATA/LP4-Power/CNT0278/ is still £70 but far more inline with the RockPro64 price, but 2 disks on each port would be no problem at all and try to distribute.
I would run up OMV install the Snapraid and unionfs plugins from omvextras and it will not make sense but when tried there is a brilliant simplicity about its manner that since I came across it its changed my perspective on at least large home media stores.

The startech seems to be another only alternative that is the USB alternative to yuour pcie x4 sata.

[ZeblodS]

@stuartiannaylor: Just to answer a few of your talking points.

I was thinking of using a regular ATX power supply for several reasons, the main one is that it's cheaper to get 12V and 5V at pretty high amperage.
For instance I can find an brand new  EVGA 600W with a single 12V49A rail and a 5V20A rail for less than 50€. More than enough to power safely the ROCKPro64 and 10 to 12 hard-drives.
Plus it's rated 80+, meaning less electricity wasted. And usually brand names power supply don't destroy all your hardware when they fail (which all power supply end up doing eventually, even brand names one as I have already experienced it after 7 years 24/7 powered on).
It has 6 SATA power connector, that can be doubled using adapters like these.
Power supply, for machines running 24/7, is really not something you can really spare expenses on.

I looked over for a USB3.0 solution, using a PCI-E like the one you linked. UASP adapter with 12V input for 3.5" harddrives are not that cheap either, I have found models like this one for instance. Plus I'd need a few USB3.0 hub to connect 2 to 3 UASP adapters per USB ports on the controller, with maybe 5V external connection to not overpower the PCI-E controller with that much disks.
In the end it won't be cheaper than the IOCrest SI-PEX40097 SATA controller, and it's a less clean solution, but it's a fallback solution for sure.

For the casing, I'd still go the individual hot-swap from the front route as it's way more practical IRL, even if it cost more. The Orico one can be found for 8.75€ a pop when bought by 10 or more.

Software wise, I'd need to do some more research into it, and some testing before going toward a different solution than what I already use. But on first glance Snapraid seem to have many advantages indeed.


EDIT: the USB solution could be cheaper using something like these:
https://m.fr.aliexpress.com/item/32897789991.html
https://m.fr.aliexpress.com/item/32829472708.html