PCIe Inconsistency Problems - Printable Version

PCIe Inconsistency Problems - Printable Version

+- PINE64 (https://forum.pine64.org)
+-- Forum: ROCKPRO64 (https://forum.pine64.org/forumdisplay.php?fid=98)
+--- Forum: Linux on RockPro64 (https://forum.pine64.org/forumdisplay.php?fid=101)
+--- Thread: PCIe Inconsistency Problems (/showthread.php?tid=6685)

Pages: 1 2

PCIe Inconsistency Problems - jja2000 - 10-21-2018

Hi,

I've posted this in another thread:

(10-20-2018, 05:36 AM)jja2000 Wrote:
(10-15-2018, 05:28 AM)jja2000 Wrote: I've been having trouble getting my Pine bought PCIe SATA card to work aswell.

It used to show up before but then totally disappear. The HDD spins up but does not show up in lsblk or in dmesg. There's no LED burning on the card. The HDD works aswell.

Doing a dmesg | grep pci gives me this:

Code:
rock64@rockpro64:~$ dmesg | grep pci [ 0.498071] of_get_named_gpiod_flags: parsed 'gpio' property of node '/vcc3v3-pcie-regulator[0]' - status (0) [ 0.498129] reg-fixed-voltage vcc3v3-pcie-regulator: Looking up vin-supply from device tree [ 0.498167] vcc3v3_pcie: supplied by dc_12v [ 0.498232] vcc3v3_pcie: 3300 mV [ 0.498386] reg-fixed-voltage vcc3v3-pcie-regulator: vcc3v3_pcie supplying 3300000uV [ 1.177078] ehci-pci: EHCI PCI platform driver [ 2.656712] vcc3v3_pcie: disabling [ 3.114214] phy phy-pcie-phy.9: Looking up phy-supply from device tree [ 3.114218] phy phy-pcie-phy.9: Looking up phy-supply property in node /pcie-phy failed [ 3.292837] rockchip-pcie f8000000.pcie: GPIO lookup for consumer ep [ 3.292852] rockchip-pcie f8000000.pcie: using device tree for GPIO lookup [ 3.292902] of_get_named_gpiod_flags: parsed 'ep-gpios' property of node '/pcie@f8000000[0]' - status (0) [ 3.293278] rockchip-pcie f8000000.pcie: Looking up vpcie3v3-supply from device tree [ 3.293470] rockchip-pcie f8000000.pcie: Looking up vpcie1v8-supply from device tree [ 3.293483] rockchip-pcie f8000000.pcie: Looking up vpcie1v8-supply property in node /pcie@f8000000 failed [ 3.293518] rockchip-pcie f8000000.pcie: no vpcie1v8 regulator found [ 3.302175] rockchip-pcie f8000000.pcie: Looking up vpcie0v9-supply from device tree [ 3.302195] rockchip-pcie f8000000.pcie: Looking up vpcie0v9-supply property in node /pcie@f8000000 failed [ 3.302230] rockchip-pcie f8000000.pcie: no vpcie0v9 regulator found [ 3.313324] rockchip-pcie f8000000.pcie: invalid power supply [ 3.820353] rockchip-pcie f8000000.pcie: PCIe link training gen1 timeout! [ 3.830861] rockchip-pcie: probe of f8000000.pcie failed with error -110

Okay so I installed mainline and I get a similar error:

Code:
rock64@rockpro64:~$ dmesg | grep pci [ 1.709157] vcc3v3_pcie: supplied by dc_12v [ 2.994902] ehci-pci: EHCI PCI platform driver [ 3.051118] ohci-pci: OHCI PCI platform driver [ 3.468181] vcc3v3_pcie: disabling [ 6.753355] rockchip-pcie f8000000.pcie: no vpcie12v regulator found [ 6.753416] rockchip-pcie f8000000.pcie: Linked as a consumer to regulator.3 [ 6.753431] rockchip-pcie f8000000.pcie: no vpcie1v8 regulator found [ 6.753441] rockchip-pcie f8000000.pcie: no vpcie0v9 regulator found [ 7.308970] rockchip-pcie f8000000.pcie: PCIe link training gen1 timeout! [ 7.309707] rockchip-pcie f8000000.pcie: Dropping the link to regulator.3 [ 7.309778] rockchip-pcie: probe of f8000000.pcie failed with error -110

But it happens a lot less consistently, mostly reboots and a bit of reseating of the cables makes it function again.

When it does work and I try to access the harddrive (either by mounting or by opening a partition manager via ssh) the board (presumeably, because I have no display output on Mainline and no way to read the GPIO pins for serial) kernel panics.

It kinda sucks that I can't use this device as intended. Anyone have a suggestion to make this work?

RE: PCIe Inconsistency Problems - Bullet64 - 10-21-2018

No problems here with two 2,5'' HDD. Works fine for me.

Code:
Linux rockpro64v_2_1 4.19.0-rc4-1071-ayufan-g10a63ec6c2a2 #1 SMP PREEMPT Mon Oct 1 07:33:40 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

Ok, now your problem. Whats your setup?

Image / Kernel
which hdd's
power supply
sd-card / eMMC (boot medium)

And please give an output from

Code:
sudo lspci -vvv

RE: PCIe Inconsistency Problems - jja2000 - 10-22-2018

(10-21-2018, 11:29 AM)Bullet64 Wrote: No problems here with two 2,5'' HDD. Works fine for me.

Code:
Linux rockpro64v_2_1 4.19.0-rc4-1071-ayufan-g10a63ec6c2a2 #1 SMP PREEMPT Mon Oct 1 07:33:40 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

Ok, now your problem. Whats your setup?

Image / Kernel

which hdd's

power supply

sd-card / eMMC (boot medium)

And please give an output from

Code:
sudo lspci -vvv

Okay so I'm running bionic-lxde-rockpro64-0.7.10-1072-arm64 with the mainline kernel (so the same one as you). I have a single 3.5" HDD from Seagate (Seagate IronWolf ST4000VN008). The powersupply is the 12V 5A model from the store. I'm booting off a 32GB SD card from sandisk.

Just like mounting/partitioning the HDD, lspci makes the board hang. If it's a kernel panic or not I don't know. I can't seem to get display to work on HDMI.

RE: PCIe Inconsistency Problems - Bullet64 - 10-22-2018

I am using this release -> https://github.com/ayufan-rock64/linux-build/releases/download/0.7.10/bionic-minimal-rockpro64-0.7.10-1072-arm64.img.xz

Only minimal, its enough for an server Smile

Please use an uart adapter so we can find the error. We need logs from lspci -vvv and dmesg.

RE: PCIe Inconsistency Problems - jja2000 - 10-22-2018

(10-22-2018, 11:06 AM)Bullet64 Wrote: I am using this release -> https://github.com/ayufan-rock64/linux-build/releases/download/0.7.10/bionic-minimal-rockpro64-0.7.10-1072-arm64.img.xz

Only minimal, its enough for an server Please use an uart adapter so we can find the error. We need logs from lspci -vvv and dmesg.

Any idea where I can find the pin layout for serial? Is there anything else I need to enable before it works?

RE: PCIe Inconsistency Problems - Bullet64 - 10-22-2018

I think this post help you. https://forum.pine64.org/showthread.php?tid=6387

RE: PCIe Inconsistency Problems - jja2000 - 10-22-2018

(10-22-2018, 11:23 AM)Bullet64 Wrote: I think this post help you. https://forum.pine64.org/showthread.php?tid=6387

Code:
rock64@rockpro64:~$ sudo lspci -vvv

[sudo] password for rock64:

[   79.585964] SError Interrupt on CPU4, code 0xbf000002 -- SError

[   79.585966] CPU: 4 PID: 2743 Comm: lspci Not tainted 4.19.0-rc4-1071-ayufan-g10a63ec6c2a2 #1

[   79.585967] Hardware name: Pine64 RockPro64 (DT)

[   79.585968] pstate: 60000085 (nZCv daIf -PAN -UAO)

[   79.585969] pc : rockchip_pcie_rd_conf+0x1a4/0x210 [pcie_rockchip_host]

[   79.585970] lr : rockchip_pcie_rd_conf+0x190/0x210 [pcie_rockchip_host]

[   79.585970] sp : ffff00000ee63bf0

[   79.585971] x29: ffff00000ee63bf0 x28: ffff8000efb26400

[   79.585973] x27: 0000000000000000 x26: ffff8000efb26400

[   79.585975] x25: 000000000000000f x24: 0000000000000040

[   79.585977] x23: ffff00000ee63ce4 x22: ffff00000ee63c64

[   79.585979] x21: ffff8000ef186b98 x20: 0000000000000004

[   79.585980] x19: 0000000000000000 x18: 0000000000000000

[   79.585982] x17: 0000000000000000 x16: 0000000000000000

[   79.585984] x15: 0000000000000000 x14: 0000000000000000

[   79.585985] x13: 0000000000000000 x12: 0000000000000000

[   79.585987] x11: 0000000000000000 x10: 0000000000000000

[   79.585989] x9 : 0000000000000000 x8 : 0000000000000000

[   79.585990] x7 : 0000000000000000 x6 : 0000000000000000

[   79.585992] x5 : 0000000000000000 x4 : 0000000000c00008

[   79.585994] x3 : ffff000013000000 x2 : 000000000080000a

[   79.585995] x1 : ffff000013c00008 x0 : ffff000010000000

[   79.585997] Kernel panic - not syncing: Asynchronous SError Interrupt

[   79.585998] CPU: 4 PID: 2743 Comm: lspci Not tainted 4.19.0-rc4-1071-ayufan-g10a63ec6c2a2 #1

[   79.585999] Hardware name: Pine64 RockPro64 (DT)

[   79.586000] Call trace:

[   79.586000]  dump_backtrace+0x0/0x180

[   79.586001]  show_stack+0x14/0x20

[   79.586002]  dump_stack+0x90/0xb4

[   79.586002]  panic+0x120/0x294

[   79.586003]  nmi_panic+0x6c/0x70

[   79.586004]  arm64_serror_panic+0x74/0x88

[   79.586004]  do_serror+0xbc/0xc0

[   79.586005]  el1_error+0x7c/0xdc

[   79.586006]  rockchip_pcie_rd_conf+0x1a4/0x210 [pcie_rockchip_host]

[   79.586006]  pci_user_read_config_dword+0x7c/0x110

[   79.586007]  pci_read_config+0xe8/0x258

[   79.586008]  sysfs_kf_bin_read+0x50/0x90

[   79.586008]  kernfs_fop_read+0x98/0x1d8

[   79.586009]  __vfs_read+0x30/0x158

[   79.586010]  vfs_read+0x90/0x168

[   79.586010]  ksys_pread64+0x70/0xb0

[   79.586011]  __arm64_sys_pread64+0x1c/0x28

[   79.586012]  el0_svc_common+0x60/0xe8

[   79.586012]  el0_svc_handler+0x2c/0x80

[   79.586013]  el0_svc+0x8/0xc

[   79.586027] SMP: stopping secondary CPUs

[   79.586027] Kernel Offset: disabled

[   79.586028] CPU features: 0x0,2180600c

[   79.586029] Memory Limit: none

As I thought, kernel panic :/

Edit: I tried it again (just a regular "lspci")

Now I don't get a kernel panic:

Code:
[ 2914.597297] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2915.621292] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2916.645295] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2917.669304] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2918.693301] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2919.717306] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2919.717986] cpu cpu4: _set_opp_voltage: failed to set voltage (1200000 1200000 1200000 mV): -110

[ 2919.718956] cpufreq: __target_index: Failed to change cpu frequency: -110

[ 2920.741313] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2921.765310] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2922.789315] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2923.813320] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2924.837320] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2925.861324] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2925.862002] cpu cpu4: _set_opp_voltage: failed to set voltage (1200000 1200000 1200000 mV): -110

[ 2925.862970] cpufreq: __target_index: Failed to change cpu frequency: -110

[ 2926.885328] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2927.909329] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2928.933332] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2929.957336] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2930.325296] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:

[ 2930.326003] rcu:     3-...0: (0 ticks this GP) idle=24e/1/0x4000000000000000 softirq=73507/73507 fqs=2626

[ 2930.327029] rcu:     (detected by 4, t=5252 jiffies, g=538765, q=1554)

[ 2930.981340] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2931.765306] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:

[ 2931.766005] rcu:     3-...0: (1 GPs behind) idle=24e/1/0x4000000000000000 softirq=73506/73507 fqs=2625

[ 2931.767006] rcu:     (detected by 5, t=5253 jiffies, g=19657, q=2)

[ 2940.753399] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2940.754081] cpu cpu4: _set_opp_voltage: failed to set voltage (1200000 1200000 1200000 mV): -110

[ 2940.755048] cpufreq: __target_index: Failed to change cpu frequency: -110

[ 2941.761377] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[ 2942.785381] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

RE: PCIe Inconsistency Problems - jannerman - 10-22-2018

Exactly the same problem (I have post same symptoms on another topic).
Reboot the device
DONT LOGIN
press ctrl+alt+f1 to bring the terminal.
Login and mount your hdd. You can also check your hdd with a smartctl to see if you can read its properties.

You should be ok.
After that if you try to login to lxde the board will hang again. But at least you will know that the PCIe and the harddisk works ok.
It's a software problem for sure.

RE: PCIe Inconsistency Problems - Bullet64 - 10-22-2018

And now please use bionic-minimal. I think LXDE is the problem.

RE: PCIe Inconsistency Problems - jja2000 - 10-23-2018

Installed the minimal version, tried mainline and stable. First boot it showed up but it hung after doing lspci. After the next couple of reboots I got the same error as before about "pcie link training".

@ayufan can't help?