(04-22-2020, 09:14 AM)pgwipeout Wrote:

(04-22-2020, 07:20 AM)kuleszdl Wrote: Hi, I tried the following card:

   HP NC360T PCI Express Dual Port Gigabit Server Adapter

It uses the 82571EB chipset. I am using the Debian unstable Kernel and, unfortunately, the kernel panics during boot when the card is plugged in. However, I found a very interesting project that provides a free firmware for quad-cards with the BCM5719 chip (it replaces the blobs in the card itself):

https://github.com/meklort/bcm5719-fw

Also, these cards look MUCH more promising to me as they have just one chip instead of four like most Intel NICs and are rated at 4W maximum (instead of 10W that the Intel quad cards seem to use). I am planning to get one and see how well this works.

Edit - could this be related to the following?:

https://forum.pine64.org/showthread.php?tid=8374

Good Morning,

It may be the hardware issue, but do note there is an issue with the rk3399 pcie controller that is currently unmitigated.
See the LKML thread here : https://lore.kernel.org/linux-pci/CAMdYzYoTwjKz4EN8PtD5pZfu3+SX+68JL+dfvmCrSnLL=K6Few@mail.gmail.com/
Also see this for additional information : https://lkml.org/lkml/2020/4/6/320

TLDR: We found the rk3399 throws either a synchronous error or a SError when a pcie device sends an unknown message.
The error type is determined by which cpu cluster handles the message.
We hijacked the arm64 error handling and processed it ourselves, and that corrects the issue, but it's not a good fix.
In the end, it was determined that significant changes to how arm64 handles pcie errors in the linux kernel need to happen.

(05-01-2020, 01:45 PM)kuleszdl Wrote: Thanks a lot @pgwipeout - I had overlooked this.

I applied the hack and also enabled PCIe gen2 mode via the already discussed link-speed change in the dts and ... the kernel boots now with the PCIe card inserted!

Even without the taskset-call all NICs are now recognized (I removed the Mac addresses from the output):

Code:

root@devuan:~# lspci
00:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 0100
01:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
01:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
01:00.2 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
01:00.3 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)

root@devuan:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
   inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
   link/ether ***************** brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
   link/ether ***************** brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
   link/ether ***************** brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
   link/ether ***************** brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
   link/ether ***************** brd ff:ff:ff:ff:ff:ff

Power consumption is at 5.3 Watts with one of the four links active. This sounds quite promising, I will now try the other Dualport card I bought previously.

Update: I tried the Intel Dualport-NIC (on HP NC360T), however, the ports don't get recognized automatically.

(05-01-2020, 05:35 PM)kuleszdl Wrote: Hard to tell without recompiling. This is what I'm getting in lspci -vvv regarding the speed:

Code:

root@devuan:~# lspci -vvv|grep -i speed
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L0s <256ns, L1 <8us
                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <2us
                LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

(05-01-2020, 09:01 PM)kuleszdl Wrote: I now added the two patches discussed in this thread (the hack from the lkml and the PCIe gen2 enablement) and added the tg3 driver, then built a custom OpenWRT image. Very good news: It boots up fine!

Then I have setup a small testbed with four Lenovo Thinkpad X200(s) machines (all with an Intel NIC) and ran a couple of benchmarks and power measures (using a simple wall meter):

Code:

        | 0-Links | 1-Link | 2-Links | 3-Links | 4-Links
--------------------------------------------------------
Idle    | 5.3 W   | 5.8 W  | 6.5 W   | 7.3 W   | 8.1 W
Load    | -       | 6.4 W  | 7.6 W   | 8.6 W   | 9.4 W
Speed   | -       | 940    | 930     | 890     | 630
(Mbit/s)

The limiting factor in terms of speed seems to be the NIC or maybe thermals (mine lacks a cooler). The CPU load with 4 iperf3-tests running was still at around 30%, so lots of CPU cycles left on the rockpro64 side. Interesting note: When the speed dropped, it dropped on all four machines to the same level.

During the tests, the rockpro64 itself was very stable (no crashes or errors in dmesg).

Here are two pictures illustrating the setup:





@tllim Firewall-case welcome! Apart from a let's say smaller version of the NAS case I was also thinking about a tall case where the NIC is placed using a PCIe riser card.

As a next step I plan to put the rockpro64 in the NAS case and replace my existing router to see how stable the whole thing runs in a more realistic 24/7-setting. Another goal is to replace the proprietary firmware on the NICs by the free alternative I mentioned earlier in this thread.

(05-02-2020, 03:10 PM)kuleszdl Wrote: Cool, looks like you are running yours already in production - or are you still testing the setup?

Regarding the 3 GBit/s maximum - well this might be due to the fact that your NIC has two ports. Assuming that the observed 800 MB/s when running a single NVMe card in x4 mode is the maximum you can get from this SoC atm, this would support this theory since

2 * 3 GBit/s = 6 GBit/s =~ 750 MB/s


Personally, I rather need a setup with 3-4  one GBit/s links. Therefore, the rockpro64 seems to be a good match for that. But apart from the speed, the power consumption is not that satisfactory for me when compared to x86_64 alternatives like the APU series that are more or less on the same level.

(05-30-2020, 10:31 AM)kuleszdl Wrote: I guess all SoCs that support more than one NIC provide the second one either via USB or internally via PCIe.

(05-30-2020, 10:31 AM)kuleszdl Wrote: There are some boards build like this

(05-30-2020, 10:31 AM)kuleszdl Wrote: however, I am not sure the Pine64 forums are the right place to discuss other boards.