pine64 512MB network unusable

Manilow · 04-29-2016, 08:05 AM

Hello all,

I received my pine64 (512MB) and my pine64+ (1GB) and realize, that the pine64 is unusable!

While the 64+ has only a slow network (there are other threads about that), the pain64 network is not usable. I installed the debian image (longsleep 20160421), as that seems to be the only viable option for the 512MB version.
Network connectivity is somewhat close to not existent. Downloading files from internet is below 25 kB/sec, while the same file loads from other PCs with 30MB/sec.
Local ssh connections are hanging after pressing enter several times, while entering commands you have to wait sometimes for the single letters to appear on the console.
Ssh sessions are disconnecting after some minutes.
Watching the connection with wireshark , I see many duplicates and out-of-order packets coming from the pain64.
I tried almost every meaningful interface setting with ethtool, setting it to 10MB 100MB full and half duplex, but that does not help.

I thought, that the the pines could be a very good alternative to raspberry pi, because the network interface has direct connectivity to the SoC, but that seems to be wrong.

Any Ideas?

tkaiser · 04-29-2016, 08:32 AM

(04-29-2016, 08:05 AM)Manilow Wrote: I installed the debian image (longsleep 20160421), as that seems to be the only viable option for the 512MB version.

All OS images here are based on longsleep's work and he prepared a simple method to update all these images so they can benefit from latest fixes. I don't understand why this important information is not featured in the download section in the wiki.

Code:
bash <(curl -s https://raw.githubusercontent.com/longsleep/build-pine64-image/master/simpleimage/platform-scripts/pine64_update_uboot.sh)

bash <(curl -s https://raw.githubusercontent.com/longsleep/build-pine64-image/master/simpleimage/platform-scripts/pine64_update_kernel.sh)

bash <(curl -s https://raw.githubusercontent.com/longsleep/build-pine64-image/master/simpleimage/platform-scripts/pine64_fix_whatever.sh)

Regarding the Ethernet problems. None of the devs that do the real work here experienced these problems so far (only the integer overflow issue affecting the 2GB boards that has been fixed by longsleep within a few days 4 weeks ago). Since you're the first here not only whining but using the right equipment what about putting a pcap trace online so that others are able to look through?

And just a thought: When you install on both machines iperf or iperf3 and then connect Pine64 <--> Pine64+ directly. Do you also get duplicate packets and how does performance look like (+85 Mbits/sec is what should be expected)

longsleep · 04-29-2016, 11:23 AM

pain64 made my day. Please push the network dump - might be interesting. Also try to change the MAC address to something else than the auto generated local administrated one. Eg. use the one printed on the backside of the board (add it to /boot/uEnv.txt like ethaddr=xx:xx...) and reboot.

androsch · 04-29-2016, 04:56 PM

(04-29-2016, 11:23 AM)longsleep Wrote: pain64 made my day. Please push the network dump - might be interesting.

While i'm also one of those guys with a Pine64+ Rev. B board and slow ethernet speed i can also offer a tcpdump or similar. While i'm only a user some code/instruction for the dump may help to provide a sufficient file...
I already sent some figures with iperf3 towards Lenny Raposo, but got no response yet, maybe he also is not exactly the right person to adress this to.

Gesendet von meinem K00L mit Tapatalk

Manilow · 05-01-2016, 11:33 AM

Hi,

here my iperf and iperf3 stats.

First the tests with the PINE64+, which works, but performs not as expected:

iperf as client running on the PINE64+ against Server on the same switch

Code:
------------------------------------------------------------

Client connecting to 192.168.168.140, TCP port 5001

TCP window size: 22.5 KByte (default)

------------------------------------------------------------

[  3] local 192.168.168.138 port 35172 connected with 192.168.168.140 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec   452 MBytes   379 Mbits/sec

iperf3 as client running on the PINE64+ against Server on the same switch

Code:
Connecting to host 192.168.168.140, port 5201

[  4] local 192.168.168.138 port 45720 connected to 192.168.168.140 port 5201

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

[  4]   0.00-1.06   sec  21.3 MBytes   169 Mbits/sec    0    205 KBytes      

[  4]   1.06-2.03   sec  21.2 MBytes   183 Mbits/sec    0    208 KBytes      

[  4]   2.03-3.01   sec  31.2 MBytes   268 Mbits/sec    0    219 KBytes      

[  4]   3.01-4.00   sec  46.2 MBytes   390 Mbits/sec    0    223 KBytes      

[  4]   4.00-5.02   sec  47.5 MBytes   391 Mbits/sec    0    228 KBytes      

[  4]   5.02-6.01   sec  47.5 MBytes   406 Mbits/sec    0    255 KBytes      

[  4]   6.01-7.01   sec  51.7 MBytes   431 Mbits/sec    0    270 KBytes      

[  4]   7.01-8.02   sec  51.2 MBytes   428 Mbits/sec    0    286 KBytes      

[  4]   8.02-9.01   sec  51.2 MBytes   433 Mbits/sec    0    291 KBytes      

[  4]   9.01-10.02  sec  51.2 MBytes   428 Mbits/sec    0    294 KBytes      

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bandwidth       Retr

[  4]   0.00-10.02  sec   420 MBytes   352 Mbits/sec    0             sender

[  4]   0.00-10.02  sec   420 MBytes   352 Mbits/sec                  receiver

iperf Done.

iperf running as server on the PINE64+ connected from client on the same switch

Code:
Client connecting to 192.168.168.138, TCP port 5001

TCP window size: 85.0 KByte (default)

------------------------------------------------------------

[  3] local 192.168.168.140 port 44308 connected with 192.168.168.138 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec   993 MBytes   833 Mbits/sec

iperf3 running as server on the PINE64+ connected from client on the same switch

Code:
Connecting to host 192.168.168.138, port 5201

[  4] local 192.168.168.140 port 58996 connected to 192.168.168.138 port 5201

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

[  4]   0.00-1.00   sec  84.5 MBytes   709 Mbits/sec  631   655 KBytes      

[  4]   1.00-2.00   sec  88.8 MBytes   744 Mbits/sec    0    748 KBytes      

[  4]   2.00-3.00   sec   102 MBytes   861 Mbits/sec    0    843 KBytes      

[  4]   3.00-4.00   sec   110 MBytes   923 Mbits/sec    0    936 KBytes      

[  4]   4.00-5.00   sec   105 MBytes   881 Mbits/sec    0   1018 KBytes      

[  4]   5.00-6.00   sec   105 MBytes   881 Mbits/sec    0   1.07 MBytes      

[  4]   6.00-7.00   sec   104 MBytes   870 Mbits/sec    0   1.14 MBytes      

[  4]   7.00-8.00   sec   106 MBytes   891 Mbits/sec    0   1.20 MBytes      

[  4]   8.00-9.00   sec   106 MBytes   891 Mbits/sec    0   1.27 MBytes      

[  4]   9.00-10.00  sec   104 MBytes   870 Mbits/sec    0   1.33 MBytes      

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bandwidth       Retr

[  4]   0.00-10.00  sec  1016 MBytes   852 Mbits/sec  631             sender

[  4]   0.00-10.00  sec  1013 MBytes   850 Mbits/sec                  receiver

iperf Done.

Now the tests with the 512MB pine64 (w/o +)

iperf as client running on the PINE64+ against Server on the same switch

Code:
Client connecting to 192.168.168.140, TCP port 5001

TCP window size: 22.5 KByte (default)

------------------------------------------------------------

[  3] local 192.168.168.138 port 42403 connected with 192.168.168.140 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec   113 MBytes  94.4 Mbits/sec

iperf3 as client running on the PINE64+ against Server on the same switch

Code:
Connecting to host 192.168.168.140, port 5201

[  4] local 192.168.168.138 port 51087 connected to 192.168.168.140 port 5201

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

[  4]   0.00-1.00   sec  11.5 MBytes  96.8 Mbits/sec    0    154 KBytes       

[  4]   1.00-2.00   sec  11.3 MBytes  94.6 Mbits/sec    0    158 KBytes       

[  4]   2.00-3.00   sec  11.2 MBytes  94.1 Mbits/sec    0    160 KBytes       

[  4]   3.00-4.00   sec  11.3 MBytes  94.6 Mbits/sec    0    161 KBytes       

[  4]   4.00-5.00   sec  11.3 MBytes  94.7 Mbits/sec    0    161 KBytes       

[  4]   5.00-6.00   sec  11.2 MBytes  94.1 Mbits/sec    0    163 KBytes       

[  4]   6.00-7.00   sec  11.2 MBytes  94.0 Mbits/sec    0    164 KBytes       

[  4]   7.00-8.00   sec  11.2 MBytes  93.6 Mbits/sec    0    164 KBytes       

[  4]   8.00-9.00   sec  11.3 MBytes  94.4 Mbits/sec    0    165 KBytes       

[  4]   9.00-10.00  sec  11.3 MBytes  94.5 Mbits/sec    0    167 KBytes       

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bandwidth       Retr

[  4]   0.00-10.00  sec   113 MBytes  94.5 Mbits/sec    0             sender

[  4]   0.00-10.00  sec   112 MBytes  94.3 Mbits/sec                  receiver

iperf Done.

iperf running as server on the PINE64+ connected from client on the same switch

Code:
Client connecting to 192.168.168.138, TCP port 5001

TCP window size: 85.0 KByte (default)

------------------------------------------------------------

[  3] local 192.168.168.140 port 44314 connected with 192.168.168.138 port 5001

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec   109 MBytes  91.1 Mbits/sec

iperf3 running as server on the PINE64+ connected from client on the same switch

Code:
Connecting to host 192.168.168.138, port 5201

[  4] local 192.168.168.140 port 59032 connected to 192.168.168.138 port 5201

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

[  4]   0.00-1.00   sec  11.0 MBytes  91.9 Mbits/sec   42   17.0 KBytes       

[  4]   1.00-2.00   sec  10.8 MBytes  90.7 Mbits/sec   32   36.8 KBytes       

[  4]   2.00-3.00   sec  11.2 MBytes  93.8 Mbits/sec   27   41.0 KBytes       

[  4]   3.00-4.00   sec  11.0 MBytes  92.2 Mbits/sec   45   46.7 KBytes       

[  4]   4.00-5.00   sec  10.8 MBytes  90.7 Mbits/sec   47   14.1 KBytes       

[  4]   5.00-6.00   sec  8.20 MBytes  68.8 Mbits/sec   84   5.66 KBytes       

[  4]   6.00-7.00   sec  7.08 MBytes  59.4 Mbits/sec   66   14.1 KBytes       

[  4]   7.00-8.00   sec  9.69 MBytes  81.3 Mbits/sec   48   48.1 KBytes       

[  4]   8.00-9.00   sec  10.6 MBytes  89.1 Mbits/sec   85   14.1 KBytes       

[  4]   9.00-10.00  sec  4.29 MBytes  36.0 Mbits/sec   91   5.66 KBytes       

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bandwidth       Retr

[  4]   0.00-10.00  sec  94.7 MBytes  79.4 Mbits/sec  567             sender

[  4]   0.00-10.00  sec  94.3 MBytes  79.1 Mbits/sec                  receiver

iperf Done.

---------------------

Connecting to host 192.168.168.138, port 5201

[  4] local 192.168.168.140 port 59056 connected to 192.168.168.138 port 5201

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd

[  4]   0.00-1.00   sec  2.74 MBytes  23.0 Mbits/sec   54   2.83 KBytes       

[  4]   1.00-2.00   sec  10.7 MBytes  89.7 Mbits/sec   74   18.4 KBytes       

[  4]   2.00-3.00   sec  5.34 MBytes  44.8 Mbits/sec  110   5.66 KBytes       

[  4]   3.00-4.00   sec   127 KBytes  1.04 Mbits/sec   14   2.83 KBytes       

[  4]   4.00-5.00   sec   127 KBytes  1.04 Mbits/sec   12   2.83 KBytes       

[  4]   5.00-6.00   sec  0.00 Bytes   0.00 bits/sec    12   2.83 KBytes       

[  4]   6.00-7.00   sec   127 KBytes  1.04 Mbits/sec   12   2.83 KBytes       

[  4]   7.00-8.00   sec   127 KBytes  1.04 Mbits/sec   20   1.41 KBytes       

[  4]   8.00-9.00   sec  0.00 Bytes   0.00 bits/sec    11   2.83 KBytes       

[  4]   9.00-10.00  sec  6.71 MBytes  56.3 Mbits/sec  104   5.66 KBytes       

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bandwidth       Retr

[  4]   0.00-10.00  sec  26.0 MBytes  21.8 Mbits/sec  423             sender

[  4]   0.00-10.00  sec  25.7 MBytes  21.6 Mbits/sec                  receiver

iperf Done.

The Duplicates are in most cases SACK (selective acknowledgements).

I will try tomorrow to get some stripped tcpdumps, I can post here

tkaiser · 05-01-2016, 11:43 AM

(05-01-2016, 11:33 AM)Manilow Wrote: First the tests with the PINE64+, which works, but performs not as expected:

Well, quoting you: 'Downloading files from internet is below 25 kB/sec'. And now you get +90 Mbits/sec with Pine64. I wouldn't call this 'Ethernet problem' but 'Internet problem' instead (for whatever reasons, as already said, BSP Ethernet drivers are known to be somewhat crappy).

Might be interesting to see traces of Internet downloads instead.

Manilow · 05-01-2016, 11:57 AM

Its for sure not the internet!

One more fact:

Starting
wget http://cdn.kernel.org/pub/linux/kernel/v...5.2.tar.xz
on the PINE (512MB)
results in a download rate starting with 70 KB/s, dropping to 22 KB/s after 30sec.

on the PINE+ (1GB)
the download starts with 4MB/s and rises to 5.5MB/s after 10 sec

When I download a similar sized file from a server in the same network, the connection rate on the PINE64+ rises to 35MB/s
the download rate of the 512MB version starts with 1.5MB/s and drops to 100KB/s after some seconds

And watch the rates for the PINE+ compared to the PINE without "+"

tkaiser · 05-01-2016, 12:07 PM

(05-01-2016, 11:57 AM)Manilow Wrote: Its for sure not the internet!

Sorry, misunderstanding. I do network debugging for a living and should've been more precise Smile

Obviously we're NOT talking about Ethernet or general network problems (layer 2) but stuff that 'only' affects connections to the outside behind a router. You're the first in the forums who's not just whining but is able to provide at least pieces of information regarding these performance drops so we might be able to resolve the problem soon.

Please keep in mind that Pine64 only has a Fast Ethernet PHY (you won't exceed 95 MBits/sec with Fast Ethernet) and that the Ethernet driver is crappy and might also need further 'TX delay' adjustments --> that might explain the mismatch between TX/RX speed you experience on Pine64+ -- please don't make the mistake to expect +940 Mbit/sec in both directions with 'tablet grade' ARM SoCs, the A64 is not known to be fast but cheap instead!)

Are you able to check for MTU mismatches between Pine64 and the router? And/or tracing at the wire between board and router or at the router itself (might give a different picture compared to a host only trace)?

Manilow · (This post was last modified: 05-02-2016, 03:41 AM by Manilow.)

I know abot all that!
I do my tests with exact the same ethernet cable on the same switch and I change the SD-card as well between both Pines, to make sure, that I have no hidden problem.
I know, that the PINE 512 has just a 100Mbit device, but that should not be the reason fort the BIG performance issues I see!

Again: I am not able to work on the ssh command line (the pine is running headless), command line is via ssh. There is no immediate feedback, characters appearing on the console in chunks. You see the high retry-rate in iperf3 statistics!

To debug that problem (and make it as well independent from crypto stuff), i have now created a small shell-script using netcat talking to the core debian xinetd based echo service. (Port 7).

Code:
#!/bin/bash

DESTIP=$1

LASTRUN=`date +%s%N`

while true; do 

  echo "########" | nc $DESTIP 7

   CURRTIME=`date +%s%N`

   echo Elapsed: $(( ($CURRTIME - $LASTRUN) / 1000000 ))

   LASTRUN=$CURRTIME

done

I am now able to run that script from the pine to my local server and vice versa.

Results for the Pine (512MB):

Running on Pine connecting to echo port on local server

Code:
########

Elapsed: 28

########

Elapsed: 1019

########

Elapsed: 28

########

Elapsed: 29

########

Elapsed: 1028

########

Elapsed: 28

########

Elapsed: 28

########

Elapsed: 1025

########

Elapsed: 28

########

Elapsed: 28

########

Elapsed: 2023

In that direction I have every 5 to 12 loops a delay of 1 to 3 seconds

Running on the Server connecting to echo port of the Pine

Code:
Elapsed: 14

########

Elapsed: 1017

########

Elapsed: 13

########

Elapsed: 20

########

Elapsed: 5055

########

Elapsed: 245

########

Elapsed: 14

########

Elapsed: 236

########

Elapsed: 15

########

Elapsed: 13

########

Elapsed: 5050

########

Elapsed: 5055

########

########

Elapsed: 10024

########

Elapsed: 209

########

Elapsed: 1013

In this direction it is the hell: Delays up to 10 seconds and more!!

And again the same test with th PINE64+
Results for the PINE64+ (1GB):

Running on Pine connecting to echo port on local server

Code:
########

Elapsed: 20

########

Elapsed: 21

########

Elapsed: 21

########

Elapsed: 21

########

Elapsed: 21

########

Elapsed: 21

########

Elapsed: 31

########

Elapsed: 25

########

Elapsed: 24

########

Elapsed: 24

########

Elapsed: 24

########

Elapsed: 24

########

Elapsed: 24

########

Elapsed: 26

########

Elapsed: 25

########

Elapsed: 24

########

Elapsed: 23

########

Elapsed: 23

########

Elapsed: 24

########

Elapsed: 24

########

Elapsed: 23

########

Elapsed: 24

########

Elapsed: 23

########

Elapsed: 24

Here we are in the range between 20ms up to 25 ms, as it should be!

Running on the Server connecting to echo port of the Pine

Code:
########

Elapsed: 15

########

Elapsed: 15

########

Elapsed: 15

########

Elapsed: 14

########

Elapsed: 16

########

Elapsed: 13

########

Elapsed: 16

########

Elapsed: 12

########

Elapsed: 12

########

Elapsed: 13

########

Elapsed: 13

########

Elapsed: 12

########

Elapsed: 12

########

Elapsed: 16

########

Elapsed: 13

########

Elapsed: 12

########

Elapsed: 15

########

Elapsed: 16

########

Elapsed: 12

########

Elapsed: 14

########

Elapsed: 13

########

Elapsed: 13

########

Elapsed: 13

########

Elapsed: 13

########

Elapsed: 12

########

Elapsed: 13

########

Elapsed: 17

########

Elapsed: 12

########

Elapsed: 13

########

Elapsed: 18

########

Elapsed: 11

########

Elapsed: 13

########

Elapsed: 12

########

Elapsed: 12

########

Elapsed: 13

########

Elapsed: 12

########

Elapsed: 13

########

Elapsed: 13

########

Elapsed: 13

########

Elapsed: 12

########

Elapsed: 14

########

Elapsed: 12

And even faster! No problem.

Using echo I have removed a lot of overhead from the tcpdump, comparing to ssh. It is now very easy to debug single sessions out of an larger tcpdump.

I see now, that the roundtrips of single packets are quiet OK. The data and the FIN packet of the echo-request is ACKed immediately, however the response packet is just delayed 5 to 10 seconds.

In the attached tcpdump the second echo request is delayed by 2 sec, the forth delayed by 10sec and the 9th is delayed by 5sec!

For me it looks like, that something must be wrong deep in the tcp-stack. Probably it has to do with the NIC-driver, as that is the only difference between PINE64 and PINE64+
I am clueless right now.
If there are some more tools, how i can produce other logs , especially about the dataflow through the TCP-stack, please provide me with hints.

Manilow · 05-02-2016, 10:45 AM

I have now updated to the newest available debian (Debian Linux with Mate GUI Image [20160501] by lenny.raposo), which behaves just the same.
The pain64 (512MB) is still unresponsive, while the PINE64+ is ok.

I have now STRACEd the xinetd-echo process on the PINE, while I run my echo script against. (strace -r -e trace=network -p PID-OF-XINETD -ff -o ./strace.log)
I have written some scripts, which have searched for TCP-retries in tcpdump, extracted those sessions, and matched them against the corresponding strace-file, which could be identified by the port number.

It looks like, that all sessions with retries are not longer than 1 sec delayed. The sessions running running longer have no retries.

Possibly Related Threads…
Thread		Author	Replies	Views	Last Post
	Programming languages support under PINE64	baryluk	6	13,452	09-23-2020, 11:46 PM Last Post: Phillip Bell
	Manjaro arm on pine64+	roel	2	6,544	10-19-2019, 05:13 AM Last Post: roel
	Volumio for PINE64 Released	mikelangeloz	11	26,621	03-22-2019, 09:25 AM Last Post: llungster
	I2S Audio patches for Pine64+ uploaded to GitHub	ramstadt	1	4,452	01-19-2019, 11:31 AM Last Post: ramstadt
	Pine64: Minimal SDL config	Max11	3	6,665	01-04-2019, 03:47 PM Last Post: Max11
	Fedora 27 on Pine64	gregjo	18	38,602	01-02-2019, 10:42 PM Last Post: heatfanjohn
	Official build procedure of ayufan pine64	r.tanaka	0	3,374	08-21-2018, 10:54 PM Last Post: r.tanaka
	DKMS on kernel 3.10.107-pine64	obrienmd	5	9,658	06-21-2018, 05:45 PM Last Post: evilbunny
	Gentoo for pine64 - longsleep kernel	incoherent	0	3,901	12-09-2017, 08:14 AM Last Post: incoherent
	Fedora Running On Pine64!	cztian	26	41,849	10-26-2017, 05:51 PM Last Post: gregjo

Login




Remember me Lost Password?

About Us