+- PINE64 (https://forum.pine64.org)
+-- Forum: ROCKPRO64 (https://forum.pine64.org/forumdisplay.php?fid=98)
+--- Forum: RockPro64 Hardware and Accessories (https://forum.pine64.org/forumdisplay.php?fid=102)
+--- Thread: Rockpro64 PCI-Express Issue. (/showthread.php?tid=8374)
RE: Rockpro64 PCI-Express Issue. - kuleszdl - 04-22-2020
@t4_4t @pgwipeout Thank you for sharing these interesting findings!
My board is v2.1 2018-07-02 and it does NOT have the mentioned capacitors. I am not sure about the jumpers you mentioned but I don't see any wires there. Would it make sense to add them?
Here's a picture of my unit, I will try to supply a better picture later:
rockpro64-nocapacitors2.jpg (Size: 464.56 KB / Downloads: 626)
Also, could you please describe the performance improvement you gained? Currently, I am getting only 560-580 MB/s in `hdparm --direct -t` test with a Samsung 970 Evo that is supposed to be much faster as well as a kernel panic when I insert a Dualport NIC (both cards are PCIe x4).
RE: Rockpro64 PCI-Express Issue. - pgwipeout - 04-22-2020
(04-22-2020, 08:56 AM)kuleszdl Wrote: @t4_4t @pgwipeout Thank you for sharing these interesting findings!
My board is v2.1 2018-07-02 and it does NOT have the mentioned capacitors. I am not sure about the jumpers you mentioned but I don't see any wires there. Would it make sense to add them?
Here's a picture of my unit, I will try to supply a better picture later:
Also, could you please describe the performance improvement you gained? Currently, I am getting only 560-580 MB/s in `hdparm --direct -t` test with a Samsung 970 Evo that is supposed to be much faster as well as a kernel panic when I insert a Dualport NIC (both cards are PCIe x4).
Okay, your board does not have the resistors pairs, but it does have the resistor/capacitor pairs in line of the data lines.
Here's my board after I removed the resistors and added the jumpers around the resistor/capacitor pairs.
Don't pick on my soldering job, please.
Annotation 2020-04-22 105427.jpg (Size: 217.19 KB / Downloads: 625)
RE: Rockpro64 PCI-Express Issue. - pgwipeout - 04-23-2020
(04-22-2020, 08:56 AM)kuleszdl Wrote: @t4_4t @pgwipeout Thank you for sharing these interesting findings!
My board is v2.1 2018-07-02 and it does NOT have the mentioned capacitors. I am not sure about the jumpers you mentioned but I don't see any wires there. Would it make sense to add them?
Here's a picture of my unit, I will try to supply a better picture later:
Also, could you please describe the performance improvement you gained? Currently, I am getting only 560-580 MB/s in `hdparm --direct -t` test with a Samsung 970 Evo that is supposed to be much faster as well as a kernel panic when I insert a Dualport NIC (both cards are PCIe x4).
Unfortunately I didn't have a card that hit the upper bandwidth until after I modified it.
What I did notice is under heavy data load my network card would occasionally fault and require a board reset to recover completely.
The kernel also pogos less while enumerating the card, and pcie init completes about a second faster.
Can't guarantee if the jumpers will help you, since I also had to remove the resistors that are already gone from your board.
I did a lot of research into pcie design before deciding to go all the way, because removing components is easy, putting components on is a lot harder.
By the way, what is the exact kernel version you are using? <uname -a>
RE: Rockpro64 PCI-Express Issue. - t4_4t - 04-23-2020
> rockpro64-nocapacitors2.jpg
I saw the jpg you posted, it's probably a board that's already fixed.
I think that the resistance "R89506 / R89507" was changed to 0Ω and the capacitor "C50007 / C50008" was left as it was.
This is exactly what I did.
The impedance of the PCIE transmission line is specified as 50Ω.
For this standard, the effect of the capacitor "C50007/C50008" is 1Ω or less, and even if there is, it has almost no effect.
On the other hand, the resistors has an error of +50Ω or more with respect to the standard value of 50Ω, which is an unacceptable error.
You can check whether it is working properly by the following method.
Code:
# lspci -vvv | grep Speed
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L0s <256ns, L1 <8us
=> LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <1us, L1 <8us
=> LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-> I am getting only 560-580 MB/s in `hdparm --direct -t` test with a Samsung 970 Evo that...
Please refer to the post below.
The Performance varies depending on the kernel type and test method.
https://forum.pine64.org/showthread.php?tid=6349&pid=39681#pid39681
RE: Rockpro64 PCI-Express Issue. - pgwipeout - 04-23-2020
(04-23-2020, 05:54 AM)t4_4t Wrote: > rockpro64-nocapacitors2.jpg
I saw the jpg you posted, it's probably a board that's already fixed.
I think that the resistance "R89506 / R89507" was changed to 0Ω and the capacitor "C50007 / C50008" was left as it was.
This is exactly what I did.
The impedance of the PCIE transmission line is specified as 50Ω.
For this standard, the effect of the capacitor "C50007/C50008" is 1Ω or less, and even if there is, it has almost no effect.
On the other hand, the resistors has an error of +50Ω or more with respect to the standard value of 50Ω, which is an unacceptable error.
You can check whether it is working properly by the following method.
Code:# lspci -vvv | grep Speed
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L0s <256ns, L1 <8us
=> LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <1us, L1 <8us
=> LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
> I am getting only 560-580 MB/s in `hdparm --direct -t` test with a Samsung 970 Evo that...
Please refer to the post below.
The Performance varies depending on the kernel type and test method.
https://forum.pine64.org/showthread.php?tid=6349&pid=39681#pid39681
The Rockpro64 can do 5GT/s?
Currently the rk3399 pcie controller is limited to pcie gen 1 in the dtsi.
`max-link-speed = <1>;`
RE: Rockpro64 PCI-Express Issue. - kuleszdl - 04-23-2020
Thank you a lot for the feedback! I am running kernel 5.5.17 without patches:
https://packages.debian.org/sid/linux-image-arm64
I will try with a patched kernel (5.4-stable and 5.7-rc2 from Armbian) and do the proposed lspci call before trying further steps.
@Luke @tllim My impression is that the provided schematics are outdated as the changes made in the revision "v2.1 2018-07-02" are not covered. I would appreciate it if you could provide us with updated schematics.
RE: Rockpro64 PCI-Express Issue. - pgwipeout - 04-23-2020
So, I just patched my device tree to
Code:
max-link-speed = <2>;Now, I've got this:
Code:
root@rockpro64:~# lspci -vvv | grep Speed
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
LnkSta: Speed 5GT/s (ok), Width x4 (ok)
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
LnkSta: Speed 5GT/s (ok), Width x4 (downgraded)
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
LnkSta: Speed 5GT/s (ok), Width x4 (downgraded)So yes, mainline needs to be fixed in a few places (I'll submit patches here shortly).
RE: Rockpro64 PCI-Express Issue. - kuleszdl - 04-23-2020
I am getting the following with the unpatched Debian unstable kernel and an ASMedia-based 4-Port SATA card inserted:
Code:
root@devuan:~# lspci -vvv|grep Speed
LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L1, Exit Latency L0s <256ns, L1 <8us
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM not supported, Exit Latency L0s unlimited, L1 <2us
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-Edit: Tried with latest 5.7-rc2 kernel from Armbian: same result
Edit2: Tried with 5.4.32 kernel with rockchip-specific patches from Armbian: same result
Both kernels hang with the inserted PCIe x4 gigabit card, and both also boot fine with the inserted NVMe card (just like the regular Debian unstable kernel).
@pgwipeout Patches highly appreciated, but if it's just the single change you mentioned this should be already sufficient.
RE: Rockpro64 PCI-Express Issue. - pgwipeout - 04-23-2020
So apparently the rk3399 has an errata that caused the rockchip group to drop it down to gen 1.
I'm raising a few arguments how this is not correct.
You can follow it here:
https://patchwork.kernel.org/patch/11505857/
RE: Rockpro64 PCI-Express Issue. - kuleszdl - 04-23-2020
Thanks for bringing this upstream. I wonder if it would be possible to change this behavior via a kernel command line parameter, so people with boards where it works could enable it. Any ideas regarding this?
Btw.: I've been busy setting up the compile environment and had already figured applied the change (but was not sure whether it was the only one that was needed). I am now recompiling and will report back if it worked - especially if it worked on my obviously revised v2.1 2018-07-02 board.
Edit: There is a separate dts for the v2 version of the rockpro64. Maybe creating an alternative one based on this for boards with the newer revision could work?