PCIe x4 mode - Lane 3 failure
#1
Hello,

Has anyone had any problems with PCIe working at x4 link width? I have two Rockpro64 boards, let's call them Board A and Board B. And I have an I350-T2 PCI Express card, which has x4 Lane support. Board A only ever achieves a x2 Lane configuration after link training. Board B usually manages x4 Lane (but not always) and quite often fails on the first Configuration Register access with an External Data Abort fault.

To diagnose this further, I have modified the PCIe PHY driver, so that at startup only one Lane is turned on, the other three remaining off (by configuring GRF_SOC_CON_5_PCIE accordingly). I then boot the board with just this one Lane enabled and observe the results - obviously the system should run successfully with a x1 Lane configuration after link training. Trying each board, with each lane enabled in turn individually, I find that:

With Board A, each of Lanes 0, 1 and 2 always achieve successful link training and subsequent access to the card never fails. However, link training on Lane 3 always fails, so the card cannot be accessed with just this lane enabled - and hence explains why it always falls back to x2 mode when all four lanes are enabled.

With Board B, again each of Lanes 0, 1 and 2 always work successfully. With just Lane 3 enabled, link training succeeds (nearly always, anyway). However,  after successful link training, the first access to a register on the card fails around 50% of the time, with an External Data Abort fault, implying that the signal over this lane is not reliable. On one occasion, the first few accesses succeeded and then it failed. The frequency of the failure does diminish if you increase the bus-scan-delay-ms period, but even with this set at 3 seconds, failures still occur.

I notice on the Rockpro64 schematic (Sheet 27) that the PCIE_RX3P / PCIE_RX3N lines have additional circuitry. It appears to consist of a potential divider to hold the DC bias of the Lane 3 output from the PCIe card at 350mV, followed by a 50ohm impedance termination and AC coupling capacitor, before the signal is fed into to RK3399. I'm no hardware engineer, so I don't quite understand what the purpose of this is - and I haven't managed to find any information about it on the internet. Does anyone know what this is for? Could it be affecting the reliability of Lane 3?

(I get the same results with a second I350-T2 card, but unfortunately don't have any alternative PCIe x4 cards to test with).

I look forward to anyone's comments regarding this,
Andy
#2
@acdkyn You might be interested in the following thread where a fix for exactly this issue is being discussed:
https://forum.pine64.org/showthread.php?tid=8374


Possibly Related Threads…
Thread Author Replies Views Last Post
  PCIE interface power on too late Slark Xiao 0 1,298 11-29-2022, 11:56 PM
Last Post: Slark Xiao
  PCIe gen1 timeout - how to debug tosi 3 5,389 03-10-2022, 05:06 PM
Last Post: db579
  Anyone try an NVIDIA GPU in the PCIe slot? Paraplegic Racehorse 0 1,954 04-15-2021, 11:46 AM
Last Post: Paraplegic Racehorse
  Never ever mix up CPU board connector with 8-pin PCIE connector LMM 0 1,996 02-11-2021, 10:18 AM
Last Post: LMM
  PCIe Armbian and Ayufan for nvme ssd LMM 1 3,081 02-03-2021, 04:38 PM
Last Post: LMM
  USB Device Mode R0b0t1 3 17,013 10-30-2020, 07:34 PM
Last Post: guannais
  fan continously runs on kernel 5.8 but I need to use 5. kernel for PCIe sata card GreyLinux 4 6,457 10-20-2020, 10:52 AM
Last Post: GreyLinux
  Failure to boot atcwells 1 3,699 05-10-2020, 03:06 PM
Last Post: atcwells
  Considering buying and question on the PCIe slot MobileJAD 25 47,882 11-07-2019, 03:10 PM
Last Post: hmuller
  RockPro64 pcie Gpu Davidos 1 3,636 06-09-2019, 02:09 PM
Last Post: guannais

Forum Jump:


Users browsing this thread: 1 Guest(s)