PCI-E add-on Success & Failures

[nuumio]

Little bit more info to here:

LSI 9201-8i (SAS2008 chip, PCIe x8 card) is NOT working. Tried with ayufan's 4.4.154-1126, 4.20.0-rc6-1075 (with patch to workaround emmc with cmdq) and my own 4.20-rc6 build. Card info from x86 PC where it works in both x1 and x16 slots:

Code:

05:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)

I hope that some day it will work. Related Github issues:

• https://github.com/rockchip-linux/kernel/issues/116
  
• https://github.com/ayufan-rock64/linux-build/issues/254
  

Syba SI-PEX40064 card (Marvell 88SE9215) seems to work just fine.

Code:

$ lspci -nn
00:00.0 PCI bridge [0604]: Rockchip Inc. RK3399 PCI Express Root Port Device [1d87:0100]
01:00.0 SATA controller [0106]: Marvell Technology Group Ltd. Device [1b4b:9215] (rev 11)

$ lsblk -S
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sda  0:0:0:0    disk ATA      WDC WD60EFRX-68L 0A82 sata
sdb  1:0:0:0    disk ATA      WDC WD60EFRX-68L 0A82 sata

[gokuz]

I'm curious, how do you guys power up more than 2 3.5" HDD?

I assume a 12V8.33A power supply >> 4 pin onboard to 4 sata power

and use a circuit to piggyback 5V for 5V rail sata.

Has anyone done this?

[nuumio]

I'm curious, how do you guys power up more than 2 3.5" HDD?

I'm using Seasonic Focus Gold 450W ATX PSU to power 6 HDDs and 2 RP64s.

For more PSU related discussion maybe these threads would be better (or create a new one)?
https://forum.pine64.org/showthread.php?tid=6639
https://forum.pine64.org/showthread.php?tid=6941

[flocke196]

I have been able run  a sata card based on Marvell 88SE9230, with 4 ports.

Kernel:
Linux rockpro64 4.18.0-rc8-1060-ayufan-g45828c4db727 #1 SMP PREEMPT Thu Aug 9 19:41:21 UTC 2018 aarch64 GNU/Linux

...

To get the rp64 I had to manually bind the ahci driver to the pcie card, adding a udev rule:

Code:

root@rockpro64:~# cat /etc/udev/rules.d/99-marvell.rules
RUN+="/bin/bash -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/ahci/bind'"

I have only tried with one drive yet, the second one is on its way 

I have Delock's 4 port SATA controller which is based on Marvell 88SE9230 chip too (https://www.delock.com/produkt/89395/merkmale.html). Got it working after adding binding to udev rules like above (thanks xtract). Having 4 WD Reds running so far without problems.

88SE9230 does get really hot though (at least on this card). Its little heat sink gets too hot to touch. I'm considering a cooling fan for it.

Code:

$ uname -a
Linux rockpro64 4.19.0-1073-ayufan-ga6e013135a6e #1 SMP PREEMPT Tue Oct 23 09:19:54 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

$ lspci -nn
00:00.0 PCI bridge [0604]: Rockchip Inc. RK3399 PCI Express Root Port Device [1d87:0100]
01:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller [1b4b:9230] (rev 11)

$ lsblk -S
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sda  0:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata
sdb  1:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata
sdc  2:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata
sdd  3:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata

$ cat /etc/udev/rules.d/99-local-last.rules 
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x1b4b", ATTR{device}=="0x9230", RUN+="/bin/bash -c 'echo %k > /sys/bus/pci/drivers/ahci/bind'"

I can confirm that this card works using the instructions mentioned above.

@nuumio:
How do you use the RAID capabilities of this card? Is it supporting HW-Raid1 and how can i setup a RAID? Is there a "bios key"?

greetings

[nuumio]

@nuumio:
How do you use the RAID capabilities of this card? Is it supporting HW-Raid1 and how can i setup a RAID? Is there a "bios key"?

Nope. I'm using SW raid, more specifically Btrfs raid1. With SW raid you can use any controller that does "Just a Bunch Of Disks" (JBOD). So replacing the controller is not a problem. With HW raid you typically have to use exactly same controller. That, and I don't really trust that HW raid on cheap controllers really works that well

[flocke196]

Thanks, nuumio.
Your reasons for better using sw raid1 are conclusive.

Regarding the power source question:
I am currently running 2x 3.5 HDD and 2x 2.5 SSD just with the power the board provides from the 12V socket, using the NASCase adapter added by this adapter: https://www.amazon.de/gp/product/B01F24P...UTF8&psc=1 .

This in combination with a powerful psu(12V,7.5A) is running fine on my side.
I know, it´s a bit risky. Using a picoPSU or an ATX psu would be safer...

greetings

[nuumio]

88SE9230 does get really hot though (at least on this card). Its little heat sink gets too hot to touch. I'm considering a cooling fan for it.

Code:

$ uname -a
Linux rockpro64 4.19.0-1073-ayufan-ga6e013135a6e #1 SMP PREEMPT Tue Oct 23 09:19:54 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

$ lspci -nn
00:00.0 PCI bridge [0604]: Rockchip Inc. RK3399 PCI Express Root Port Device [1d87:0100]
01:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller [1b4b:9230] (rev 11)

$ lsblk -S
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sda  0:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata
sdb  1:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata
sdc  2:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata
sdd  3:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata

$ cat /etc/udev/rules.d/99-local-last.rules 
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x1b4b", ATTR{device}=="0x9230", RUN+="/bin/bash -c 'echo %k > /sys/bus/pci/drivers/ahci/bind'"

I can confirm that this card works using the instructions mentioned above.

Kinda bad news related to my 9230 controller: I just checked logs and found one of those infamous "ata exceptions / failed commands". Seems it occurred last Saturday night when I was running btrfs scrub. I tried scrub 3 times again and from 2 of those runs I got one failed command more per run.

I think I have to try that 9230 on PC to check if the controller itself works or not.

[flocke196]

No problems with the controller so far. Rsynced 3.8 TB of data without any dmesg errors so far.
I think the btrfs scrub creates much more io load on the controller.

I will do some stress testing (copy several hundred GB between the connected ssds/hdd, may be in parallel) and let you know if there are errors.
Do you know a good method of stress testing a Sata-Controller?

greetings

Edit:
While doing some stress testing (copy 500GB from sw-raid1 to both ssds in parallel), the ugly error messages pop up several times in dmesg:

Code:

[177359.815846] ata1.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x6 frozen
[177359.818370] ata1.00: failed command: WRITE FPDMA QUEUED
[177359.820792] ata1.00: cmd 61/78:08:70:a3:04/00:00:1d:00:00/40 tag 1 ncq dma 6                                                                                 1440 out
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (time                                                                                 out)
[177359.826247] ata1.00: status: { DRDY }
[177359.828502] ata1: hard resetting link
[177360.145994] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[177360.150489] ata1.00: supports DRM functions and may not be fully accessible
[177360.153733] ata1.00: disabling queued TRIM support
[177360.156187] ata1.00: supports DRM functions and may not be fully accessible
[177360.159228] ata1.00: disabling queued TRIM support
[177360.160830] ata1.00: configured for UDMA/133
[177360.163081] ata1: EH complete
[177394.635885] ata1.00: exception Emask 0x0 SAct 0x40000 SErr 0x0 action 0x6 fr                                                                                 ozen
[177394.638381] ata1.00: failed command: READ FPDMA QUEUED
[177394.640550] ata1.00: cmd 60/08:90:08:08:c0/00:00:00:00:00/40 tag 18 ncq dma                                                                                  4096 in
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (time                                                                                 out)
[177394.645314] ata1.00: status: { DRDY }
[177394.647351] ata1: hard resetting link
[177394.963135] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[177394.967475] ata1.00: supports DRM functions and may not be fully accessible
[177394.970766] ata1.00: disabling queued TRIM support
[177394.973283] ata1.00: supports DRM functions and may not be fully accessible
[177394.979736] ata1.00: disabling queued TRIM support
[177394.981241] ata1.00: configured for UDMA/133
[177394.986826] ata1: EH complete

The SSDs recover from that and the cp processes are still going on.

Write speeds are limited by read speed of md0 (2xWD Red 6TB):

Code:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          0.88    0.00   26.83   23.29    0.00   49.00

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
mtdblock0         0.00         0.00         0.00          0          0
mtdblock1         0.00         0.00         0.00          0          0
mtdblock2         0.00         0.00         0.00          0          0
mmcblk1          35.00         0.15         0.00          0          0
mmcblk1boot1      0.00         0.00         0.00          0          0
mmcblk1boot0      0.00         0.00         0.00          0          0
mmcblk0           0.00         0.00         0.00          0          0
sdc             202.20         0.00       102.58          0        512
sdd             413.20       102.85         0.00        514          0
sdb             454.00       112.99         0.00        564          0
sda             372.20         0.00       118.66          0        593
md0             864.60       215.84         0.00       1079          0

The write speed drops down to zero, when a hard reset of the link happens.

This happens on both connected SSDs,not in parallel and not on the HDDs so far.

greetings

[nuumio]

I think firmware update might have gotten my Marvell 9230 (manufactured by Delock) to work correctly. Old version was 1.0.0.1024 bios + 2.3.0.1065 fw. The one I flashed was 1.0.0.1027 bios + 2.3.0.1076 fw. I used x86_64 pc + UEFI shell to update it. Googling "marvell 9230 firmware download" gave me few helpful links:

• https://homeservershow.com/forums/topic/...ent=142560
  and from there: https://www.startech.com/Cards-Adapters/...PEXSAT34RH
  
• https://homeservershow.com/forums/topic/...ent=148171
  and from there: https://www.digitus.info/en/products/com...s-30104-1/
  
• ftp://ftp.supermicro.com/ISO_Extracted/C...efi64/bin/
  

Those homerservershow posts gave links to one FW which should be the same as I have and then a newer one. I wasn't exactly sure if one manufacturers FW is combatible with another so I checked md5 from FW downloaded from Startech site and from FW read from my 9230 and they matched. So I gave the newer FW from Digitus a go and it seems to be working. More than 10 Btrfs scrubs in a row without any problems and it also seems to perform a bit better as scrubs are faster.

The link to supermicro has EFI program for doing the update. Packages from Startech and Digitus seemed to be Dos only. I already had UEFI shell from https://github.com/tianocore/edk2 (used one from tag edk2-stable201811) on one USB drive to I just copied files from Supermicro and Digitus there.

Don't try this yourself if you don't know what you're doing!  Read instructions from packages you download! Take backups! I didn't use the update script myself (go.nsh) because I didn't get it to do backup (maybe I was using it the wrong way). I just used commands seen in the script in command prompt. Dos boot + go.bat may be easier way.

My steps in short (read go.nsh/go.bat for details or just the go.* script):
- Boot to UEFI BIOS and start shell (correct way may depend on UEFI BIOS manufacturer)
- Select USB filesystem and cd to correct dir if needed
- Read paramfile from 9230: mvf.efi -getparm 9230.par -cdid 9230 -adid 0
- Check paramfile (RevId 10=ImageA0.bin, 11=ImageA1.bin): type 9230.par
- Backup: mvf.efi -rfb 9230-backup.bin -addr 0 -sz 0x7b000 -cdid 9230 -adid 0
- Full backup (check size from paramfile): mvf.efi -rfb 9230-backup-full.bin -addr 0 -sz 0x80000 -cdid 9230 -adid 0
- Flash (check RevId/image from paramfile): mvf.efi -wbf ImageA1.bin -addr 0 -cdid 9230 -adid 0

Maybe that older FW had some bug which just happened to occur more frequently with RP64 than in x86(_64) PCs.

[flocke196]

Hey nuumio,


thank you so much for this update.
I managed to update the controller firmware successfully using the following firmware:

http://ftp.assmann.com/pub/DS-/DS-30104-...181221.zip .
The one I flashed was 1.0.0.1027 bios + 2.3.0.1076 fw. The same version you are using.
It is NOT working with the newest firmware file i found, 1.0.0.1028 bios + 2.3.0.1078 fw. The discs are not recognized anymore and dmesg complains about the controller init going wrong.

So, downgrading the firmware was working for me.

What i have done:

1. Put the controller(mine is a DeLOCK 89395)in a x64/x86 machine
2. Prepare USB Drive with FreeDos and put the files from the archive, mentioned above, in its root.
3. Boot FreeDos from USB Drive
4. Execute "go.bat -y"
5. Wait for flashing to be finished


Attention: THESE STEPS CONTAIN NO BACKUP OF YOUR EXISTING FIRMWARE!
Even with the "go.bat" method, the backup of the SPI flash is not correctly working, so do it at your own risk.



Will do some stress testing and report here.

Edit: First stress test (2x cp from sw-raid1 -> both connected ssds in parallel, results in using all 4 ports of the controller) produced no errors in dmesg.
That was the test, that produced those ugly errors (see above) before on my side.