12-02-2019, 05:30 AM
Hello,
Has anyone had any problems with PCIe working at x4 link width? I have two Rockpro64 boards, let's call them Board A and Board B. And I have an I350-T2 PCI Express card, which has x4 Lane support. Board A only ever achieves a x2 Lane configuration after link training. Board B usually manages x4 Lane (but not always) and quite often fails on the first Configuration Register access with an External Data Abort fault.
To diagnose this further, I have modified the PCIe PHY driver, so that at startup only one Lane is turned on, the other three remaining off (by configuring GRF_SOC_CON_5_PCIE accordingly). I then boot the board with just this one Lane enabled and observe the results - obviously the system should run successfully with a x1 Lane configuration after link training. Trying each board, with each lane enabled in turn individually, I find that:
With Board A, each of Lanes 0, 1 and 2 always achieve successful link training and subsequent access to the card never fails. However, link training on Lane 3 always fails, so the card cannot be accessed with just this lane enabled - and hence explains why it always falls back to x2 mode when all four lanes are enabled.
With Board B, again each of Lanes 0, 1 and 2 always work successfully. With just Lane 3 enabled, link training succeeds (nearly always, anyway). However, after successful link training, the first access to a register on the card fails around 50% of the time, with an External Data Abort fault, implying that the signal over this lane is not reliable. On one occasion, the first few accesses succeeded and then it failed. The frequency of the failure does diminish if you increase the bus-scan-delay-ms period, but even with this set at 3 seconds, failures still occur.
I notice on the Rockpro64 schematic (Sheet 27) that the PCIE_RX3P / PCIE_RX3N lines have additional circuitry. It appears to consist of a potential divider to hold the DC bias of the Lane 3 output from the PCIe card at 350mV, followed by a 50ohm impedance termination and AC coupling capacitor, before the signal is fed into to RK3399. I'm no hardware engineer, so I don't quite understand what the purpose of this is - and I haven't managed to find any information about it on the internet. Does anyone know what this is for? Could it be affecting the reliability of Lane 3?
(I get the same results with a second I350-T2 card, but unfortunately don't have any alternative PCIe x4 cards to test with).
I look forward to anyone's comments regarding this,
Andy
Has anyone had any problems with PCIe working at x4 link width? I have two Rockpro64 boards, let's call them Board A and Board B. And I have an I350-T2 PCI Express card, which has x4 Lane support. Board A only ever achieves a x2 Lane configuration after link training. Board B usually manages x4 Lane (but not always) and quite often fails on the first Configuration Register access with an External Data Abort fault.
To diagnose this further, I have modified the PCIe PHY driver, so that at startup only one Lane is turned on, the other three remaining off (by configuring GRF_SOC_CON_5_PCIE accordingly). I then boot the board with just this one Lane enabled and observe the results - obviously the system should run successfully with a x1 Lane configuration after link training. Trying each board, with each lane enabled in turn individually, I find that:
With Board A, each of Lanes 0, 1 and 2 always achieve successful link training and subsequent access to the card never fails. However, link training on Lane 3 always fails, so the card cannot be accessed with just this lane enabled - and hence explains why it always falls back to x2 mode when all four lanes are enabled.
With Board B, again each of Lanes 0, 1 and 2 always work successfully. With just Lane 3 enabled, link training succeeds (nearly always, anyway). However, after successful link training, the first access to a register on the card fails around 50% of the time, with an External Data Abort fault, implying that the signal over this lane is not reliable. On one occasion, the first few accesses succeeded and then it failed. The frequency of the failure does diminish if you increase the bus-scan-delay-ms period, but even with this set at 3 seconds, failures still occur.
I notice on the Rockpro64 schematic (Sheet 27) that the PCIE_RX3P / PCIE_RX3N lines have additional circuitry. It appears to consist of a potential divider to hold the DC bias of the Lane 3 output from the PCIe card at 350mV, followed by a 50ohm impedance termination and AC coupling capacitor, before the signal is fed into to RK3399. I'm no hardware engineer, so I don't quite understand what the purpose of this is - and I haven't managed to find any information about it on the internet. Does anyone know what this is for? Could it be affecting the reliability of Lane 3?
(I get the same results with a second I350-T2 card, but unfortunately don't have any alternative PCIe x4 cards to test with).
I look forward to anyone's comments regarding this,
Andy