08-28-2016, 10:25 AM
(This post was last modified: 08-28-2016, 04:41 PM by MarkHaysHarris777.
Edit Reason: added code blocks for readability
)
Follow-up from previous post with results from testing two boards (one bad) with two different OS's
Background:
I have two PINE64 boards which run two different OS versions. One runs Ubuntu. The other, Debian.
I will refer to these boards as board A and board B regardless of which OS I run on them.
I have experienced poor communication performance from the new board B. What was unknown was whether
the performance was due to OS, network parameters defined while setting up the OS, a network issue
caused by the physical layer on the network (bad or incompatible hardware), or a faulty NIC on the
board itself. The boards are running on different legs on my network. I have swapped the SD cards
on the boards leaving all other hardware unchanged and the problem continues with the board that
originally ran Debian (board B). The problem remained on board B regardless of OS running.
Testing Methodology:
Eliminate the variables:
1) Run each board separately while testing in an isolated area on the network. Plug into
same network cable (CAT 6) and power supply swapping out the board so the only variable is the board.
2) Run test suite twice with each board. Once with Debian running and once with Ubuntu running.
Boot the Pine64 machine
Run longsleep's "pine64_tune_network.sh" script to insure NIC parameters are known.
Validate communication configuration by running "ethtool eth0" and examining output for consistency.
Run iperf3 as client connecting to know stable server running iperf3 as server and record output.
Run iperf3 as server and connect from known stable machine running iperf3 as client and record output.
Findings:
Background:
I have two PINE64 boards which run two different OS versions. One runs Ubuntu. The other, Debian.
I will refer to these boards as board A and board B regardless of which OS I run on them.
I have experienced poor communication performance from the new board B. What was unknown was whether
the performance was due to OS, network parameters defined while setting up the OS, a network issue
caused by the physical layer on the network (bad or incompatible hardware), or a faulty NIC on the
board itself. The boards are running on different legs on my network. I have swapped the SD cards
on the boards leaving all other hardware unchanged and the problem continues with the board that
originally ran Debian (board B). The problem remained on board B regardless of OS running.
Testing Methodology:
Eliminate the variables:
1) Run each board separately while testing in an isolated area on the network. Plug into
same network cable (CAT 6) and power supply swapping out the board so the only variable is the board.
2) Run test suite twice with each board. Once with Debian running and once with Ubuntu running.
Boot the Pine64 machine
Run longsleep's "pine64_tune_network.sh" script to insure NIC parameters are known.
Validate communication configuration by running "ethtool eth0" and examining output for consistency.
Run iperf3 as client connecting to know stable server running iperf3 as server and record output.
Run iperf3 as server and connect from known stable machine running iperf3 as client and record output.
Findings:
Code:
Board B has a faulty NIC or component that interacts with NIC.
#######################################
Data
#######################################
###############
BOARD "B" specs
1) Pine64+ 2Gb with wifi/BT 201-A64DB110-02 A64-DB-V1.1 2G C106358 04.10
2) RTL chip numbers: RTL8211E G3E16P1 GG12B
###############
###############
BOARD "A" specs
1) Pine64+ 2Gb with wifi/BT 201-A64DB110-02 A64-DB-V1.1 2G C106358 04.12
2) RTL chip numbers: RTL8211E G3E16P1 GG12B
###############
Network Switch used during test: TRENDnet TEG-S80g
Network cable used throughout testing is a CAT 6 cable
#############################################################################################
BOARD "A" test output running Debian
#############################################################################################
Linux pine64 3.10.102-3-pine64-longsleep #98 SMP PREEMPT Sat Aug 20 22:28:17 CEST 2016 aarch64 GNU/Linux
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: external
Auto-negotiation: on
Link detected: yes
Connecting to host 192.168.0.3, port 5201
[ 4] local 192.168.0.9 port 57805 connected to 192.168.0.3 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 108 MBytes 906 Mbits/sec 0 273 KBytes
[ 4] 1.00-2.00 sec 109 MBytes 918 Mbits/sec 0 273 KBytes
[ 4] 2.00-3.01 sec 108 MBytes 902 Mbits/sec 0 273 KBytes
[ 4] 3.01-4.01 sec 93.8 MBytes 785 Mbits/sec 0 273 KBytes
[ 4] 4.01-5.00 sec 97.6 MBytes 826 Mbits/sec 0 273 KBytes
[ 4] 5.00-6.02 sec 111 MBytes 913 Mbits/sec 0 273 KBytes
[ 4] 6.02-7.01 sec 92.5 MBytes 783 Mbits/sec 0 273 KBytes
[ 4] 7.01-8.00 sec 98.0 MBytes 828 Mbits/sec 0 273 KBytes
[ 4] 8.00-9.01 sec 101 MBytes 844 Mbits/sec 0 273 KBytes
[ 4] 9.01-10.00 sec 102 MBytes 854 Mbits/sec 0 273 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1021 MBytes 856 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1020 MBytes 856 Mbits/sec receiver
iperf Done.
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.0.3, port 54294
[ 5] local 192.168.0.9 port 5201 connected to 192.168.0.3 port 54295
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 102 MBytes 853 Mbits/sec
[ 5] 1.00-2.00 sec 112 MBytes 940 Mbits/sec
[ 5] 2.00-3.00 sec 112 MBytes 936 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 938 Mbits/sec
[ 5] 5.00-6.00 sec 112 MBytes 939 Mbits/sec
[ 5] 6.00-7.00 sec 112 MBytes 940 Mbits/sec
[ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec
[ 5] 8.00-9.00 sec 112 MBytes 942 Mbits/sec
[ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec
[ 5] 10.00-10.04 sec 3.62 MBytes 727 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.04 sec 1.09 GBytes 930 Mbits/sec sender
[ 5] 0.00-10.04 sec 1.09 GBytes 930 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
#############################################################################################
BOARD "B" test output running Debian
#############################################################################################
Linux pine64 3.10.102-3-pine64-longsleep #98 SMP PREEMPT Sat Aug 20 22:28:17 CEST 2016 aarch64 GNU/Linux
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: external
Auto-negotiation: on
Link detected: yes
Connecting to host 192.168.0.3, port 5201
[ 4] local 192.168.0.9 port 38219 connected to 192.168.0.3 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 106 MBytes 890 Mbits/sec 1 94.7 KBytes
[ 4] 1.00-2.00 sec 109 MBytes 914 Mbits/sec 0 132 KBytes
[ 4] 2.00-3.00 sec 109 MBytes 915 Mbits/sec 1 120 KBytes
[ 4] 3.00-4.00 sec 85.0 MBytes 714 Mbits/sec 2 79.2 KBytes
[ 4] 4.00-5.00 sec 108 MBytes 906 Mbits/sec 0 122 KBytes
[ 4] 5.00-6.00 sec 108 MBytes 909 Mbits/sec 2 67.9 KBytes
[ 4] 6.00-7.00 sec 79.0 MBytes 663 Mbits/sec 2 45.2 KBytes
[ 4] 7.00-8.00 sec 84.1 MBytes 705 Mbits/sec 1 82.0 KBytes
[ 4] 8.00-9.00 sec 94.3 MBytes 791 Mbits/sec 1 90.5 KBytes
[ 4] 9.00-10.00 sec 108 MBytes 904 Mbits/sec 0 129 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 991 MBytes 831 Mbits/sec 10 sender
[ 4] 0.00-10.00 sec 991 MBytes 831 Mbits/sec receiver
iperf Done.
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.0.3, port 54338
[ 5] local 192.168.0.9 port 5201 connected to 192.168.0.3 port 54339
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 991 KBytes 8.12 Mbits/sec
[ 5] 1.00-2.00 sec 978 KBytes 8.01 Mbits/sec
[ 5] 2.00-3.00 sec 1.13 MBytes 9.52 Mbits/sec
[ 5] 3.00-4.00 sec 1.24 MBytes 10.4 Mbits/sec
[ 5] 4.00-5.00 sec 873 KBytes 7.15 Mbits/sec
[ 5] 5.00-6.00 sec 75.6 KBytes 619 Kbits/sec
[ 5] 6.00-7.00 sec 579 KBytes 4.74 Mbits/sec
[ 5] 7.00-8.00 sec 931 KBytes 7.62 Mbits/sec
[ 5] 8.00-9.00 sec 295 KBytes 2.42 Mbits/sec
[ 5] 9.00-10.00 sec 1004 KBytes 8.22 Mbits/sec
[ 5] 10.00-10.04 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.04 sec 8.12 MBytes 6.79 Mbits/sec sender
[ 5] 0.00-10.04 sec 7.97 MBytes 6.66 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
#############################################################################################
BOARD "A" test output running Ubuntu
#############################################################################################
Linux localhost.localdomain 3.10.65-7-pine64-longsleep #28 SMP PREEMPT Sat Apr 23 20:13:25 CEST 2016 aarch64 aarch64 aarch64 GNU/Linux
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: external
Auto-negotiation: on
Link detected: yes
Connecting to host 192.168.0.3, port 5201
[ 4] local 192.168.0.167 port 56591 connected to 192.168.0.3 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 108 MBytes 902 Mbits/sec 0 273 KBytes
[ 4] 1.00-2.00 sec 109 MBytes 914 Mbits/sec 0 273 KBytes
[ 4] 2.00-3.00 sec 109 MBytes 918 Mbits/sec 0 273 KBytes
[ 4] 3.00-4.00 sec 109 MBytes 913 Mbits/sec 1 124 KBytes
[ 4] 4.00-5.00 sec 109 MBytes 912 Mbits/sec 0 156 KBytes
[ 4] 5.00-6.00 sec 109 MBytes 916 Mbits/sec 0 181 KBytes
[ 4] 6.00-7.00 sec 109 MBytes 912 Mbits/sec 0 201 KBytes
[ 4] 7.00-8.00 sec 109 MBytes 915 Mbits/sec 0 212 KBytes
[ 4] 8.00-9.00 sec 109 MBytes 912 Mbits/sec 0 218 KBytes
[ 4] 9.00-10.00 sec 109 MBytes 914 Mbits/sec 0 222 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.06 GBytes 913 Mbits/sec 1 sender
[ 4] 0.00-10.00 sec 1.06 GBytes 912 Mbits/sec receiver
iperf Done.
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.0.3, port 54166
[ 5] local 192.168.0.167 port 5201 connected to 192.168.0.3 port 54167
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 105 MBytes 878 Mbits/sec
[ 5] 1.00-2.00 sec 110 MBytes 920 Mbits/sec
[ 5] 2.00-3.00 sec 111 MBytes 931 Mbits/sec
[ 5] 3.00-4.00 sec 110 MBytes 919 Mbits/sec
[ 5] 4.00-5.00 sec 62.8 MBytes 527 Mbits/sec
[ 5] 5.00-6.00 sec 91.9 MBytes 771 Mbits/sec
[ 5] 6.00-7.00 sec 101 MBytes 849 Mbits/sec
[ 5] 7.00-8.00 sec 91.7 MBytes 769 Mbits/sec
[ 5] 8.00-9.00 sec 110 MBytes 927 Mbits/sec
[ 5] 9.00-10.00 sec 111 MBytes 934 Mbits/sec
[ 5] 10.00-10.04 sec 4.65 MBytes 940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.04 sec 1009 MBytes 843 Mbits/sec sender
[ 5] 0.00-10.04 sec 1009 MBytes 843 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
#############################################################################################
BOARD "B" test output running Ubuntu
#############################################################################################
Linux localhost.localdomain 3.10.65-7-pine64-longsleep #28 SMP PREEMPT Sat Apr 23 20:13:25 CEST 2016 aarch64 aarch64 aarch64 GNU/Linux
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: external
Auto-negotiation: on
Link detected: yes
Connecting to host 192.168.0.3, port 5201
[ 4] local 192.168.0.167 port 56141 connected to 192.168.0.3 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.01 sec 83.2 MBytes 690 Mbits/sec 1 113 KBytes
[ 4] 1.01-2.01 sec 82.1 MBytes 693 Mbits/sec 1 99.0 KBytes
[ 4] 2.01-3.01 sec 80.8 MBytes 673 Mbits/sec 1 130 KBytes
[ 4] 3.01-4.01 sec 81.2 MBytes 684 Mbits/sec 1 130 KBytes
[ 4] 4.01-5.00 sec 80.0 MBytes 675 Mbits/sec 2 109 KBytes
[ 4] 5.00-6.01 sec 81.2 MBytes 675 Mbits/sec 1 127 KBytes
[ 4] 6.01-7.00 sec 80.0 MBytes 679 Mbits/sec 2 103 KBytes
[ 4] 7.00-8.00 sec 80.0 MBytes 670 Mbits/sec 2 119 KBytes
[ 4] 8.00-9.01 sec 82.5 MBytes 686 Mbits/sec 0 168 KBytes
[ 4] 9.01-10.01 sec 81.2 MBytes 687 Mbits/sec 3 60.8 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.01 sec 812 MBytes 681 Mbits/sec 14 sender
[ 4] 0.00-10.01 sec 812 MBytes 681 Mbits/sec receiver
iperf Done.
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.0.3, port 54305
[ 5] local 192.168.0.167 port 5201 connected to 192.168.0.3 port 54306
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 1.76 MBytes 14.8 Mbits/sec
[ 5] 1.00-2.00 sec 1.35 MBytes 11.3 Mbits/sec
[ 5] 2.00-3.00 sec 1.41 MBytes 11.8 Mbits/sec
[ 5] 3.00-4.00 sec 210 KBytes 1.72 Mbits/sec
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 10.00-11.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 11.00-12.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 12.00-13.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 13.00-14.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 14.00-15.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 15.00-16.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 16.00-17.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 17.00-18.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 18.00-19.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 19.00-20.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 20.00-21.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 21.00-22.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 22.00-23.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 23.00-24.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 24.00-25.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 25.00-26.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 26.00-27.00 sec 0.00 Bytes 0.00 bits/sec
[ 5] 27.00-28.00 sec 0.00 Bytes 0.00 bits/sec
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------