PCI-E add-on Success & Failures

[macross_dyrl]

Good day,

I purchased two of the Rockpro64 Boards and NAS case. 
So far I really like the unit. 
I plan on donating second board to freebsd project.  waiting on them to tell me where to send it.

I am starting to try a few PCI boards boards in the rockpro64. 

I was able to get High point RAId card that comes with san digital to work after upgrade of the kernel.
Please post any PCI devices that work here and anything you need to do to get it recognized.

Code:

uname -a
Linux rockpro64 4.18.0-rc8-1060-ayufan-g45828c4db727 #1 SMP PREEMPT Thu Aug 9 19:41:21 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

Code:

lspci -v
00:00.0 PCI bridge: Rockchip Inc. RK3399 PCI Express Root Port Device 0100 (prog-if 00 [Normal decode])
    Flags: bus master, fast devsel, latency 0, IRQ 255
    Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
    I/O behind bridge: 00000000-00000fff
    Memory behind bridge: fa000000-fa0fffff
    Prefetchable memory behind bridge: 00000000-000fffff
    Capabilities: <access denied>

01:00.0 RAID bus controller: HighPoint Technologies, Inc. RocketRAID 642L SATA-III Controller (2 eSATA ports + 2 internal SATA ports) (rev 01)
    Subsystem: HighPoint Technologies, Inc. RocketRAID 642L SATA-III Controller (2 eSATA ports + 2 internal SATA ports)
    Flags: bus master, fast devsel, latency 0, IRQ 231
    I/O ports at 0000
    I/O ports at 0000
    I/O ports at 0000
    I/O ports at 0000
    I/O ports at 0000
    Memory at fa010000 (32-bit, non-prefetchable) [size=2K]
    Expansion ROM at fa000000 [disabled] [size=64K]
    Capabilities: <access denied>
    Kernel driver in use: ahci

Code:

lsscsi -v
[0:0:0:0]    disk    ATA      TOSHIBA HDWN160  FS1M  /dev/sda
 dir: /sys/bus/scsi/devices/0:0:0:0  [/sys/devices/platform/f8000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/ata1/host0/target0:0:0/0:0:0:0]
[0:1:0:0]    disk    ATA      TOSHIBA HDWE160  FS2A  /dev/sdb
 dir: /sys/bus/scsi/devices/0:1:0:0  [/sys/devices/platform/f8000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/ata1/host0/target0:1:0/0:1:0:0]
[0:2:0:0]    disk    ATA      TOSHIBA HDWN160  FS1M  /dev/sdc
 dir: /sys/bus/scsi/devices/0:2:0:0  [/sys/devices/platform/f8000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/ata1/host0/target0:2:0/0:2:0:0]
[0:3:0:0]    disk    ATA      TOSHIBA HDWE160  FS2A  /dev/sdd
 dir: /sys/bus/scsi/devices/0:3:0:0  [/sys/devices/platform/f8000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/ata1/host0/target0:3:0/0:3:0:0]
[2:0:0:0]    disk    ATA      TOSHIBA HDWE150  FP2A  /dev/sde
 dir: /sys/bus/scsi/devices/2:0:0:0  [/sys/devices/platform/f8000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/ata3/host2/target2:0:0/2:0:0:0]
[3:0:0:0]    disk    ATA      TOSHIBA HDWE150  FP2A  /dev/sdf
 dir: /sys/bus/scsi/devices/3:0:0:0  [/sys/devices/platform/f8000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/ata4/host3/target3:0:0/3:0:0:0]

Code:

/dev/sda:
Timing cached reads:   1976 MB in  2.00 seconds = 987.65 MB/sec
Timing buffered disk reads: 590 MB in  3.00 seconds = 196.52 MB/sec

/dev/sdb:
Timing cached reads:   1980 MB in  2.00 seconds = 989.73 MB/sec
Timing buffered disk reads: 586 MB in  3.01 seconds = 194.69 MB/sec

/dev/sdc:
Timing cached reads:   1990 MB in  2.00 seconds = 995.44 MB/sec
Timing buffered disk reads: 586 MB in  3.01 seconds = 194.71 MB/sec

/dev/sdd:
Timing cached reads:   1986 MB in  2.00 seconds = 992.67 MB/sec
Timing buffered disk reads: 468 MB in  3.00 seconds = 155.81 MB/sec

/dev/sde:
Timing cached reads:   1948 MB in  2.00 seconds = 973.81 MB/sec
Timing buffered disk reads: 402 MB in  3.00 seconds = 133.85 MB/sec

[unregisteredidiot]

I've been on a similar adventure, just with a bit more limited scope.

As of 0.7.9, the linux images from ayufan (huge thanks to him, btw) only support the pine64-supplied pci-e sata controller if you go with one of the Bionic minimal releases. From my experiments so far, it seems that if you really want a window manager your best bet is i3. There's a script:
`sudo /usr/local/sbin/install_desktop.sh i3`

For whatever reason (which dependency /is/ it? I don't know yet), installing lxde, mate, or xfce cause some kind of conflict that results in a total lockup when you access the pine64 pci-e sata controller. Will you please test this with the various pci-e cards you are experimenting with? More info might help figure out exactly where the bug lies in those builds.

[xtract]

I have been able run  a sata card based on Marvell 88SE9230, with 4 ports.

Kernel:
Linux rockpro64 4.18.0-rc8-1060-ayufan-g45828c4db727 #1 SMP PREEMPT Thu Aug 9 19:41:21 UTC 2018 aarch64 GNU/Linux

Code:

01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller (rev 11) (prog-if 01 [AHCI 1.0])
    Subsystem: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller
    Flags: bus master, fast devsel, latency 0, IRQ 231
    I/O ports at 0000
    I/O ports at 0000
    I/O ports at 0000
    I/O ports at 0000
    I/O ports at 0000
    Memory at fa010000 (32-bit, non-prefetchable) [size=2K]
    Expansion ROM at fa000000 [disabled] [size=64K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
    Capabilities: [70] Express Legacy Endpoint, MSI 00
    Capabilities: [e0] SATA HBA v0.0
    Capabilities: [100] Advanced Error Reporting
    Kernel driver in use: ahci

To get the rp64 I had to manually bind the ahci driver to the pcie card, adding a udev rule:

Code:

root@rockpro64:~# cat /etc/udev/rules.d/99-marvell.rules
RUN+="/bin/bash -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/ahci/bind'"

I have only tried with one drive yet, the second one is on its way 

[guannais]

Another SATA adapter that works:

Code:

jinks ~ # lspci
00:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 0100
01:00.0 SATA controller: ASMedia Technology Inc. Device 0625 (rev 01)
jinks ~ # lspci -n
00:00.0 0604: 1d87:0100
01:00.0 0106: 1b21:0625 (rev 01)

But with issues. I consider this chipset buggy, because it gave me problems on both ARM and x86_64. It does not like my DVD writer, and I had to try different port connection configurations to make it to stop giving errors like "exception Emask ..." (like this: https://www.spinics.net/lists/linux-ide/msg54099.html), mainly around a WD Purple disk.

[url=https://www.spinics.net/lists/linux-ide/msg54099.html][/url]

Code:

jinks ~ # uname -a
Linux jinks 4.4.154 #8 SMP Fri Oct 12 00:42:01 CEST 2018 aarch64 GNU/Linux

Code:

jinks ~ # for x in a b c d e; do hdparm -I /dev/sd${x} | grep 'Model'; done
       Model Number:       WDC WD20EARS-00MVWB0                    
       Model Number:       Samsung SSD 860 EVO 250GB              
       Model Number:       TOSHIBA DT01ABA300                      
       Model Number:       Corsair Force 3 SSD                    
       Model Number:       WDC WD60PURX-64LZMY0

Code:

jinks ~ # for x in a b c d e; do hdparm -t /dev/sd${x}; done

/dev/sda:
Timing buffered disk reads: 352 MB in  3.01 seconds = 116.93 MB/sec

/dev/sdb:
Timing buffered disk reads: 1418 MB in  3.00 seconds = 472.48 MB/sec

/dev/sdc:
Timing buffered disk reads: 462 MB in  3.01 seconds = 153.50 MB/sec

/dev/sdd:
Timing buffered disk reads: 704 MB in  3.01 seconds = 234.15 MB/sec

/dev/sde:
Timing buffered disk reads: 472 MB in  3.01 seconds = 156.99 MB/sec

[romtorwator]

I also ordered a controller on a Marvell 88SE9235 chip (4*SATA, no raid).
I wonder if it works on the image stretch-minimal-rockpro64-0.7.9-1067-arm64,
with updated kernel Linux rockpro64 4.4.138-1094-ayufan-gf13a8a9a4eee ?

[nuumio]

I have been able run  a sata card based on Marvell 88SE9230, with 4 ports.

Kernel:
Linux rockpro64 4.18.0-rc8-1060-ayufan-g45828c4db727 #1 SMP PREEMPT Thu Aug 9 19:41:21 UTC 2018 aarch64 GNU/Linux

...

To get the rp64 I had to manually bind the ahci driver to the pcie card, adding a udev rule:

Code:

root@rockpro64:~# cat /etc/udev/rules.d/99-marvell.rules
RUN+="/bin/bash -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/ahci/bind'"

I have only tried with one drive yet, the second one is on its way 

I have Delock's 4 port SATA controller which is based on Marvell 88SE9230 chip too (https://www.delock.com/produkt/89395/merkmale.html). Got it working after adding binding to udev rules like above (thanks xtract). Having 4 WD Reds running so far without problems.

88SE9230 does get really hot though (at least on this card). Its little heat sink gets too hot to touch. I'm considering a cooling fan for it.

Code:

$ uname -a
Linux rockpro64 4.19.0-1073-ayufan-ga6e013135a6e #1 SMP PREEMPT Tue Oct 23 09:19:54 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

$ lspci -nn
00:00.0 PCI bridge [0604]: Rockchip Inc. RK3399 PCI Express Root Port Device [1d87:0100]
01:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller [1b4b:9230] (rev 11)

$ lsblk -S
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sda  0:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata
sdb  1:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata
sdc  2:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata
sdd  3:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata

$ cat /etc/udev/rules.d/99-local-last.rules 
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x1b4b", ATTR{device}=="0x9230", RUN+="/bin/bash -c 'echo %k > /sys/bus/pci/drivers/ahci/bind'"

[huey]

I have been able run  a sata card based on Marvell 88SE9230, with 4 ports.

Kernel:
Linux rockpro64 4.18.0-rc8-1060-ayufan-g45828c4db727 #1 SMP PREEMPT Thu Aug 9 19:41:21 UTC 2018 aarch64 GNU/Linux

...

To get the rp64 I had to manually bind the ahci driver to the pcie card, adding a udev rule:

Code:

root@rockpro64:~# cat /etc/udev/rules.d/99-marvell.rules
RUN+="/bin/bash -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/ahci/bind'"

I have only tried with one drive yet, the second one is on its way 

I have Delock's 4 port SATA controller which is based on Marvell 88SE9230 chip too (https://www.delock.com/produkt/89395/merkmale.html). Got it working after adding binding to udev rules like above (thanks xtract). Having 4 WD Reds running so far without problems.

88SE9230 does get really hot though (at least on this card). Its little heat sink gets too hot to touch. I'm considering a cooling fan for it.

Code:

$ uname -a
Linux rockpro64 4.19.0-1073-ayufan-ga6e013135a6e #1 SMP PREEMPT Tue Oct 23 09:19:54 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

$ lspci -nn
00:00.0 PCI bridge [0604]: Rockchip Inc. RK3399 PCI Express Root Port Device [1d87:0100]
01:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller [1b4b:9230] (rev 11)

$ lsblk -S
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sda  0:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata
sdb  1:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata
sdc  2:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata
sdd  3:0:0:0    disk ATA      WDC WD30EFRX-68N 0A82 sata

$ cat /etc/udev/rules.d/99-local-last.rules 
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x1b4b", ATTR{device}=="0x9230", RUN+="/bin/bash -c 'echo %k > /sys/bus/pci/drivers/ahci/bind'"

Using the same card but from IOCREST (same id's) but can not get it working. Followed same steps I think: SD Card with minimal bionic 7.9.0 (?) and switched mainline kernel 4.19.0-1073. But with and without udev rule it will not see my disk. I checked voltage on the power...
If I try to run eMMC image and switch to mainline kernel my rockpro does not boot anymore... keeps complaining and looping about EQC error? 
Tried to use serial console to see boot from the start, but that only gives me garbled code??

Is the pcie only visible with the mainline kernel or does someone have an easier route to a working adapter for me available?
Pfff, not sure what to do to understand whats wrong.

Regards

[nuumio]

Using the same card but from IOCREST (same id's) but can not get it working. Followed same steps I think: SD Card with minimal bionic 7.9.0 (?) and switched mainline kernel 4.19.0-1073. But with and without udev rule it will not see my disk. I checked voltage on the power...
If I try to run eMMC image and switch to mainline kernel my rockpro does not boot anymore... keeps complaining and looping about EQC error? 
Tried to use serial console to see boot from the start, but that only gives me garbled code??

Is the pcie only visible with the mainline kernel or does someone have an easier route to a working adapter for me available?
Pfff, not sure what to do to understand whats wrong.

Regards

I'm assuming that the controller is working and you have checked vendor and device id with lspci. You've probably already done it but just to be sure: Check power and data cable connections once more on both RP64 and the drive. Check the data cable and drive on PC or some other device if possible. If none of these help I'm out of ideas.


About eMMC card I'd guess you're having 32GB Foresee (so some other card that supports command queueing). In that case the problem is that command queueing doesn't work and has to be disabled with a kernel patch. See discussion on another thread and couple of link to different patches:
- https://forum.pine64.org/showthread.php?...2#pid40442
- https://github.com/nuumio/ayufan-rock64-...f97c82e2a3
- https://github.com/nuumio/ayufan-rock64-...582380766d


For serial console my guess would be wrong settings. They should be 1500000 8N1. I made a mistake the first time and missed one zero

[huey]

Hi, thx for the response. Will check out the patches, but have ordered the pine sata adapter to see if I can get that one working at least. Will make it a bit easier I hope to troubleshoot...
Will have to wait some weeks for it to arrive though. Pity.

[gokuz]

How do you guys power up an additional 2 (3.5 inch) drives? 60W is only sufficient for 2 (3.5) drives.

An ATX PSU would be a waste powering up just 2 drives, not to mention the efficiency?
The only other idea is a picopsu with 36W adapter. Ideas?