Can RockPRO64 support PCIe Endpoint?

[andrewg132]

Hello,

I have been attempting to implement PCIe endpoint support on my RockPRO64 board over the past few weeks, but have not been successful and I am starting to doubt that the board can support PCIe EP mode. I'm fairly new to Linux and I am most likely overlooking/missing something as well, but before spending too much more time on this project, I wanted to ensure that it was even possible so I can pursue another route. Has anyone had success with this setup?

At the moment, I am using a custom Armbian build (Debian Sid, no Desktop, kernel v5.19.y) as I was having difficulty setting up Yocto and other off-the-shelf images did not seem to want to boot properly with my board.

I am essentially trying to replicate this tutorial on the RockPRO64 https://docs.nxp.com/bundle/GUID-3FFCCD7...055F0.html  I am able to boot, create an endpoint function, but I am unable to bind the function to the controller as there isn't one!

If the board does truly support PCI Endpoint, I believe it is most likely due to improper configuration of the Device Tree on my end. My various attempts of patching did show some promise.

Apologies for the lack of information, I didn't want to write a lengthy post before determining what I am trying to do is even possible!

Any help would be greatly appreciated!
Thanks, and regards,
Andrew

[andrewg132]

I have made some progress on getting this working, but I am unable to figure out why the initialization of the driver is failing to reserve memory space for MSI.
I modified the kernel to disable the "host" Rockchip PCI controllers which did seem to help, but unsure if this is the proper way to do it.

See syslog output below:

Code:

Aug 31 21:37:40 rockpro64 haveged[737]: haveged: ver: 1.9.1; arch: generic; vend: ; build: (gcc 8.3.0 CTV); collect: 128K
Aug 31 21:37:40 rockpro64 haveged[737]: haveged: cpu: (VC); data: 16K (D); inst: 16K (D); idx: 11/40; sz: 15528/64688
Aug 31 21:37:40 rockpro64 haveged[737]: haveged: tot tests(BA8): A:1/1 B:1/1 continuous tests(B):  last entropy estimate 7.99992
Aug 31 21:37:40 rockpro64 kernel: [    1.561821] CPU: 4 PID: 1 Comm: swapper/0 Not tainted 5.19.6-rockchip64 #trunk
Aug 31 21:37:40 rockpro64 kernel: [    1.561845] Hardware name: Pine64 RockPro64 v2.1 (DT)
Aug 31 21:37:40 rockpro64 kernel: [    1.561860] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Aug 31 21:37:40 rockpro64 kernel: [    1.561882] pc : __ioremap_caller+0xcc/0xf0
Aug 31 21:37:40 rockpro64 kernel: [    1.561904] lr : __ioremap_caller+0x54/0xf0
Aug 31 21:37:40 rockpro64 kernel: [    1.561925] sp : ffff800009ebbad0
Aug 31 21:37:40 rockpro64 haveged[737]: haveged: fills: 0, generated: 0
Aug 31 21:37:40 rockpro64 systemd[1]: systemd-rfkill.service: Succeeded.
Aug 31 21:37:40 rockpro64 kernel: [    1.561937] x29: ffff800009ebbad0 x28: 00000000ffffffff x27: 0000000000020000
Aug 31 21:37:40 rockpro64 kernel: [    1.561971] x26: ffff0000066c7100 x25: 000000000001ffff x24: 0000000000000005
Aug 31 21:37:40 rockpro64 kernel: [    1.562002] x23: 0068000000000713 x22: ffff800008894198 x21: 0000000000000000
Aug 31 21:37:40 rockpro64 kernel: [    1.562031] x20: 0000000080000000 x19: 0000000000020000 x18: 0000000000000000
Aug 31 21:37:40 rockpro64 kernel: [    1.562059] x17: 6f7420746c756166 x16: 6564202c79616c65 x15: 00003d12a555276a
Aug 31 21:37:40 rockpro64 kernel: [    1.562088] x14: 000000000000016e x13: 000000000000013b x12: 0000000000000020
Aug 31 21:37:40 rockpro64 kernel: [    1.562116] x11: ffff0000066c7180 x10: 0000000000000005 x9 : 0000000000000020
Aug 31 21:37:40 rockpro64 systemd[1]: Finished Armbian hardware optimization.
Aug 31 21:37:40 rockpro64 kernel: [    1.562144] x8 : 00000000f7e00000 x7 : 0000000000000018 x6 : ffff800009e0b198
Aug 31 21:37:40 rockpro64 kernel: [    1.562173] x5 : ffff800009e0b198 x4 : 0000000000000001 x3 : 0000000000000001
Aug 31 21:37:40 rockpro64 kernel: [    1.562200] x2 : 00000000f8000000 x1 : 0000000000000000 x0 : 0000000000000001
Aug 31 21:37:40 rockpro64 kernel: [    1.562228] Call trace:
Aug 31 21:37:40 rockpro64 kernel: [    1.562240]  __ioremap_caller+0xcc/0xf0
Aug 31 21:37:40 rockpro64 kernel: [    1.562263]  __ioremap+0x28/0x58
Aug 31 21:37:40 rockpro64 systemd[1]: Reached target Basic System.
Aug 31 21:37:40 rockpro64 kernel: [    1.562284]  pci_epc_mem_alloc_addr+0xe0/0x168
Aug 31 21:37:40 rockpro64 kernel: [    1.562306]  rockchip_pcie_ep_probe+0x1a8/0x248
Aug 31 21:37:40 rockpro64 kernel: [    1.562326]  platform_probe+0x68/0xd8
Aug 31 21:37:40 rockpro64 kernel: [    1.562347]  really_probe+0x15c/0x3a8
Aug 31 21:37:40 rockpro64 kernel: [    1.562364]  __driver_probe_device+0x114/0x188
Aug 31 21:37:40 rockpro64 kernel: [    1.562381]  driver_probe_device+0x3c/0xe8
Aug 31 21:37:40 rockpro64 kernel: [    1.562399]  __driver_attach+0xbc/0x1d0
Aug 31 21:37:40 rockpro64 kernel: [    1.562415]  bus_for_each_dev+0x74/0xc8
Aug 31 21:37:40 rockpro64 kernel: [    1.562431]  driver_attach+0x24/0x30
Aug 31 21:37:40 rockpro64 kernel: [    1.562447]  bus_add_driver+0x1a8/0x250
Aug 31 21:37:40 rockpro64 kernel: [    1.562463]  driver_register+0x64/0x120
Aug 31 21:37:40 rockpro64 kernel: [    1.562480]  __platform_driver_register+0x28/0x38
Aug 31 21:37:40 rockpro64 kernel: [    1.562501]  rockchip_pcie_ep_driver_init+0x1c/0x28
Aug 31 21:37:40 rockpro64 kernel: [    1.562522]  do_one_initcall+0x4c/0x288
Aug 31 21:37:40 rockpro64 kernel: [    1.562540]  kernel_init_freeable+0x224/0x2ac
Aug 31 21:37:40 rockpro64 kernel: [    1.562561]  kernel_init+0x24/0x130
Aug 31 21:37:40 rockpro64 kernel: [    1.562583]  ret_from_fork+0x10/0x20
Aug 31 21:37:40 rockpro64 kernel: [    1.562601] ---[ end trace 0000000000000000 ]---
Aug 31 21:37:40 rockpro64 kernel: [    1.562620] rockchip-pcie-ep fd000000.pcie-ep: failed to reserve memory space for MSI
Aug 31 21:37:40 rockpro64 kernel: [    1.562655] rockchip-pcie-ep: probe of fd000000.pcie-ep failed with error -12
Aug 31 21:37:40 rockpro64 kernel: [    1.563283] ------------[ cut here ]------------
Aug 31 21:37:40 rockpro64 kernel: [    1.563302] Device 'fd000000.pcie-ep' does not have a release() function, it is broken and must be fixed. See Documentation/core-api/kobject.rst.
Aug 31 21:37:40 rockpro64 kernel: [    1.563363] WARNING: CPU: 4 PID: 1 at drivers/base/core.c:2244 device_release+0x80/0x90
Aug 31 21:37:40 rockpro64 kernel: [    1.563399] Modules linked in:
Aug 31 21:37:40 rockpro64 kernel: [    1.563416] CPU: 4 PID: 1 Comm: swapper/0 Tainted: G        W         5.19.6-rockchip64 #trunk
Aug 31 21:37:40 rockpro64 kernel: [    1.563439] Hardware name: Pine64 RockPro64 v2.1 (DT)

Device Tree (decompiled with fdtdump)

Code:

pcie@f8000000 {
        compatible = "rockchip,rk3399-pcie";
        reg = <0x00000000 0xf8000000 0x00000000 0x02000000 0x00000000 0xfd000000 0x00000000 0x01000000>;
        reg-names = "axi-base", "apb-base";
        device_type = "pci";
        #address-cells = <0x00000003>;
        #size-cells = <0x00000002>;
        #interrupt-cells = <0x00000001>;
        aspm-no-l0s;
        bus-range = <0x00000000 0x0000001f>;
        clocks = <0x00000008 0x000000c5 0x00000008 0x000000c4 0x00000008 0x00000147 0x00000008 0x000000a0>;
        clock-names = "aclk", "aclk-perf", "hclk", "pm";
        interrupts = <0x00000000 0x00000031 0x00000004 0x00000000 0x00000000 0x00000032 0x00000004 0x00000000 0x00000000 0x00000033 0x00000004 0x00000000>;
        interrupt-names = "sys", "legacy", "client";
        interrupt-map-mask = <0x00000000 0x00000000 0x00000000 0x00000007>;
        interrupt-map = <0x00000000 0x00000000 0x00000000 0x00000001 0x00000015 0x00000000 0x00000000 0x00000000 0x00000000 0x00000002 0x00000015 0x00000001 0x00000000 0x00000000 0x00000000 0x00000003 0x00000015 0x00000002 0x00000000 0x00000000 0x00000000 0x00000004 0x00000015 0x00000003>;
        max-link-speed = <0x00000001>;
        msi-map = <0x00000000 0x00000016 0x00000000 0x00001000>;
        phys = <0x00000017 0x00000000 0x00000017 0x00000001 0x00000017 0x00000002 0x00000017 0x00000003>;
        phy-names = "pcie-phy-0", "pcie-phy-1", "pcie-phy-2", "pcie-phy-3";
        ranges = <0x82000000 0x00000000 0xfa000000 0x00000000 0xfa000000 0x00000000 0x01e00000 0x81000000 0x00000000 0xfbe00000 0x00000000 0xfbe00000 0x00000000 0x00100000>;
        resets = <0x00000008 0x00000082 0x00000008 0x00000083 0x00000008 0x00000084 0x00000008 0x00000085 0x00000008 0x00000086 0x00000008 0x00000081 0x00000008 0x00000080>;
        reset-names = "core", "mgmt", "mgmt-sticky", "pipe", "pm", "pclk", "aclk";
        status = "okay";
        ep-gpios = <0x00000018 0x0000001c 0x00000000>;
        num-lanes = <0x00000004>;
        pinctrl-names = "default";
        pinctrl-0 = <0x00000019>;
        vpcie12v-supply = <0x0000001a>;
        vpcie3v3-supply = <0x0000001b>;
        bus-scan-delay-ms = <0x000003e8>;
        phandle = <0x000000da>;
        interrupt-controller {
            interrupt-controller;
            #address-cells = <0x00000000>;
            #interrupt-cells = <0x00000001>;
            phandle = <0x00000015>;
        };
    };
    pcie-ep@f8000000 {
        compatible = "rockchip,rk3399-pcie-ep";
        #address-cells = <0x00000003>;
        #size-cells = <0x00000002>;
        rockchip,max-outbound-regions = <0x00000010>;
        clocks = <0x00000008 0x000000c5 0x00000008 0x000000c4 0x00000008 0x00000147 0x00000008 0x000000a0>;
        clock-names = "aclk", "aclk-perf", "hclk", "pm";
        max-functions = [08];
        num-lanes = <0x00000004>;
        reg = <0x00000000 0xfd000000 0x00000000 0x01000000 0x00000000 0x80000000 0x00000000 0x00020000>;
        reg-names = "apb-base", "mem-base";
        resets = <0x00000008 0x00000082 0x00000008 0x00000083 0x00000008 0x00000084 0x00000008 0x00000085 0x00000008 0x00000086 0x00000008 0x00000081 0x00000008 0x00000080>;
        reset-names = "core", "mgmt", "mgmt-sticky", "pipe", "pm", "pclk", "aclk";
        phys = <0x00000017 0x00000000 0x00000017 0x00000001 0x00000017 0x00000002 0x00000017 0x00000003>;
        phy-names = "pcie-phy-0", "pcie-phy-1", "pcie-phy-2", "pcie-phy-3";
        pinctrl-names = "default";
        status = "okay";
        phandle = <0x000000db>;
    };

Any hints or ideas?

[andrewg132]

I managed to figure it out! The issue was with the starting address for 'mem-base' in the device tree. Setting the starting address to a valid range (0x0000_0000 in my case) enables the driver to be properly loaded and the controller is now present. Below is the device tree overlay I used in Armbian in case anyone else would like to attempt this. 

Code:

/dts-v1/;
/plugin/;
/ {

    fragment@0 {
        target-path = "/pcie@f8000000";
        __overlay__ {
               status = "disabled";
        };
    };

    fragment@1 {
                target-path = "/";
                __overlay__ {
                        pcie_ep0: pcie_ep@f8000000 {
                                compatible = "rockchip,rk3399-pcie-ep";
                                #address-cells = <0x03>;
                                #size-cells = <0x02>;
                                rockchip,max-outbound-regions = <0x10>;
                                max-functions = [08];
                                num-lanes = <0x04>;
                                reg = <0x00 0xfd000000 0x00 0x1000000 0x00 0x00000000 0x00 0x20000>;
                                reg-names = "apb-base\0mem-base";
                                pinctrl-names = "default";
                                pinctrl-0 = <0x17>;
                                phys = <0x15 0x00 0x15 0x01 0x15 0x02 0x15 0x03>;
                                phy-names = "pcie-phy-0\0pcie-phy-1\0pcie-phy-2\0pcie-phy-3";
                                resets = <0x08 0x82 0x08 0x83 0x08 0x84 0x08 0x85 0x08 0x86 0x08 0x81 0x08 0x80>;
                                reset-names = "core\0mgmt\0mgmt-sticky\0pipe\0pm\0pclk\0aclk";
                                clocks = <0x08 0xc5 0x08 0xc4 0x08 0x147 0x08 0xa0>;
                                clock-names = "aclk\0aclk-perf\0hclk\0pm";
                                status = "okay";
                         };
                };
    };
};

[rwk-]

Hello, I am trying to setup a PCIe endpoint, I was wondering about which cable you used to connect your board.
I suppose you use a male to male cable with RX/TX lines crossed right ? Could you share the cable / setup you use to connect your board (with it's female connector) to another board / mainboard that acts as a root complex.

Thank you