Rockpro64 PCI-Express Issue.

[kuleszdl]

@t4_4t @pgwipeout Thank you for sharing these interesting findings!

My board is v2.1 2018-07-02 and it does NOT have the mentioned capacitors. I am not sure about the jumpers you mentioned but I don't see any wires there. Would it make sense to add them?

Here's a picture of my unit, I will try to supply a better picture later:


  rockpro64-nocapacitors2.jpg (Size: 464.56 KB / Downloads: 553)

Also, could you please describe the performance improvement you gained? Currently, I am getting only 560-580 MB/s in `hdparm --direct -t` test with a Samsung 970 Evo that is supposed to be much faster as well as a kernel panic when I insert a Dualport NIC (both cards are PCIe x4).

[pgwipeout]

@t4_4t @pgwipeout Thank you for sharing these interesting findings!

My board is v2.1 2018-07-02 and it does NOT have the mentioned capacitors. I am not sure about the jumpers you mentioned but I don't see any wires there. Would it make sense to add them?

Here's a picture of my unit, I will try to supply a better picture later:



Also, could you please describe the performance improvement you gained? Currently, I am getting only 560-580 MB/s in `hdparm --direct -t` test with a Samsung 970 Evo that is supposed to be much faster as well as a kernel panic when I insert a Dualport NIC (both cards are PCIe x4).

Okay, your board does not have the resistors pairs, but it does have the resistor/capacitor pairs in line of the data lines.
Here's my board after I removed the resistors and added the jumpers around the resistor/capacitor pairs. 
Don't pick on my soldering job, please.

  Annotation 2020-04-22 105427.jpg (Size: 217.19 KB / Downloads: 563)

[pgwipeout]

@t4_4t @pgwipeout Thank you for sharing these interesting findings!

My board is v2.1 2018-07-02 and it does NOT have the mentioned capacitors. I am not sure about the jumpers you mentioned but I don't see any wires there. Would it make sense to add them?

Here's a picture of my unit, I will try to supply a better picture later:



Also, could you please describe the performance improvement you gained? Currently, I am getting only 560-580 MB/s in `hdparm --direct -t` test with a Samsung 970 Evo that is supposed to be much faster as well as a kernel panic when I insert a Dualport NIC (both cards are PCIe x4).

Unfortunately I didn't have a card that hit the upper bandwidth until after I modified it.

What I did notice is under heavy data load my network card would occasionally fault and require a board reset to recover completely.
The kernel also pogos less while enumerating the card, and pcie init completes about a second faster.
Can't guarantee if the jumpers will help you, since I also had to remove the resistors that are already gone from your board.

I did a lot of research into pcie design before deciding to go all the way, because removing components is easy, putting components on is a lot harder.

By the way, what is the exact kernel version you are using? <uname -a>

[t4_4t]

> rockpro64-nocapacitors2.jpg
I saw the jpg you posted, it's probably a board that's already fixed.
I think that the resistance "R89506 / R89507" was changed to 0Ω and the capacitor "C50007 / C50008" was left as it was.
This is exactly what I did.

The impedance of the PCIE transmission line is specified as 50Ω.
For this standard, the effect of the capacitor "C50007/C50008" is 1Ω or less, and even if there is, it has almost no effect.
On the other hand, the resistors has an error of +50Ω or more with respect to the standard value of 50Ω, which is an unacceptable error.

You can check whether it is working properly by the following method.

Code:

# lspci -vvv | grep Speed
               LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L0s <256ns, L1 <8us
=>             LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
               LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
               LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <1us, L1 <8us
=>             LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
               LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

> I am getting only 560-580 MB/s in `hdparm --direct -t` test with a Samsung 970 Evo that...
Please refer to the post below.
The Performance varies depending on the kernel type and test method.

https://forum.pine64.org/showthread.php?...1#pid39681

[pgwipeout]

> rockpro64-nocapacitors2.jpg
I saw the jpg you posted, it's probably a board that's already fixed.
I think that the resistance "R89506 / R89507" was changed to 0Ω and the capacitor "C50007 / C50008" was left as it was.
This is exactly what I did.

The impedance of the PCIE transmission line is specified as 50Ω.
For this standard, the effect of the capacitor "C50007/C50008" is 1Ω or less, and even if there is, it has almost no effect.
On the other hand, the resistors has an error of +50Ω or more with respect to the standard value of 50Ω, which is an unacceptable error.

You can check whether it is working properly by the following method.

Code:

# lspci -vvv | grep Speed
               LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L0s <256ns, L1 <8us
=>             LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
               LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
               LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <1us, L1 <8us
=>             LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
               LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

> I am getting only 560-580 MB/s in `hdparm --direct -t` test with a Samsung 970 Evo that...
Please refer to the post below.
The Performance varies depending on the kernel type and test method.

https://forum.pine64.org/showthread.php?...1#pid39681

The Rockpro64 can do 5GT/s?

Currently the rk3399 pcie controller is limited to pcie gen 1 in the dtsi.
`max-link-speed = <1>;`

[kuleszdl]

Thank you a lot for the feedback! I am running kernel 5.5.17 without patches:

https://packages.debian.org/sid/linux-image-arm64

I will try with a patched kernel (5.4-stable and 5.7-rc2 from Armbian) and do the proposed lspci call before trying further steps.

@Luke @tllim My impression is that the provided schematics are outdated as the changes made in the revision "v2.1 2018-07-02" are not covered. I would appreciate it if you could provide us with updated schematics.

[pgwipeout]

So, I just patched my device tree to 

Code:

max-link-speed = <2>;

Now, I've got this:

Code:

root@rockpro64:~# lspci -vvv | grep Speed
               LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
               LnkSta: Speed 5GT/s (ok), Width x4 (ok)
               LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
               LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
               LnkSta: Speed 5GT/s (ok), Width x4 (downgraded)
               LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
               LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
               LnkSta: Speed 5GT/s (ok), Width x4 (downgraded)

So yes, mainline needs to be fixed in a few places (I'll submit patches here shortly).

[kuleszdl]

I am getting the following with the unpatched Debian unstable kernel and an ASMedia-based 4-Port SATA card inserted:

Code:

root@devuan:~# lspci -vvv|grep Speed
               LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L1, Exit Latency L0s <256ns, L1 <8us

               LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
               LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
               LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM not supported, Exit Latency L0s unlimited, L1 <2us
               LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-



               LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-

Edit: Tried with latest 5.7-rc2 kernel from Armbian: same result
Edit2: Tried with 5.4.32 kernel with rockchip-specific patches from Armbian: same result

Both kernels hang with the inserted PCIe x4 gigabit card, and both also boot fine with the inserted NVMe card (just like the regular Debian unstable kernel).

@pgwipeout Patches highly appreciated, but if it's just the single change you mentioned this should be already sufficient.

[pgwipeout]

So apparently the rk3399 has an errata that caused the rockchip group to drop it down to gen 1.
I'm raising a few arguments how this is not correct.
You can follow it here:
https://patchwork.kernel.org/patch/11505857/

[kuleszdl]

Thanks for bringing this upstream. I wonder if it would be possible to change this behavior via a kernel command line parameter, so people with boards where it works could enable it. Any ideas regarding this?

Btw.: I've been busy setting up the compile environment and had already figured applied the change (but was not sure whether it was the only one that was needed). I am now recompiling and will report back if it worked - especially if it worked on my obviously revised v2.1 2018-07-02 board.

Edit: There is a separate dts for the v2 version of the rockpro64. Maybe creating an alternative one based on this for boards with the newer revision could work?