I know abot all that!
I do my tests with exact the same ethernet cable on the same switch and I change the SD-card as well between both Pines, to make sure, that I have no hidden problem.
I know, that the PINE 512 has just a 100Mbit device, but that should not be the reason fort the BIG performance issues I see!
Again: I am not able to work on the ssh command line (the pine is running headless), command line is via ssh. There is no immediate feedback, characters appearing on the console in chunks. You see the high retry-rate in iperf3 statistics!
To debug that problem (and make it as well independent from crypto stuff), i have now created a small shell-script using netcat talking to the core debian xinetd based echo service. (Port 7).
I am now able to run that script from the pine to my local server and vice versa.
Results for the Pine (512MB):
Running on Pine connecting to echo port on local server
In that direction I have every 5 to 12 loops a delay of 1 to 3 seconds
Running on the Server connecting to echo port of the Pine
In this direction it is the hell: Delays up to 10 seconds and more!!
And again the same test with th PINE64+
Results for the PINE64+ (1GB):
Running on Pine connecting to echo port on local server
Here we are in the range between 20ms up to 25 ms, as it should be!
Running on the Server connecting to echo port of the Pine
And even faster! No problem.
Using echo I have removed a lot of overhead from the tcpdump, comparing to ssh. It is now very easy to debug single sessions out of an larger tcpdump.
I see now, that the roundtrips of single packets are quiet OK. The data and the FIN packet of the echo-request is ACKed immediately, however the response packet is just delayed 5 to 10 seconds.
In the attached tcpdump the second echo request is delayed by 2 sec, the forth delayed by 10sec and the 9th is delayed by 5sec!
For me it looks like, that something must be wrong deep in the tcp-stack. Probably it has to do with the NIC-driver, as that is the only difference between PINE64 and PINE64+
I am clueless right now.
If there are some more tools, how i can produce other logs , especially about the dataflow through the TCP-stack, please provide me with hints.
I do my tests with exact the same ethernet cable on the same switch and I change the SD-card as well between both Pines, to make sure, that I have no hidden problem.
I know, that the PINE 512 has just a 100Mbit device, but that should not be the reason fort the BIG performance issues I see!
Again: I am not able to work on the ssh command line (the pine is running headless), command line is via ssh. There is no immediate feedback, characters appearing on the console in chunks. You see the high retry-rate in iperf3 statistics!
To debug that problem (and make it as well independent from crypto stuff), i have now created a small shell-script using netcat talking to the core debian xinetd based echo service. (Port 7).
Code:
#!/bin/bash
DESTIP=$1
LASTRUN=`date +%s%N`
while true; do
echo "########" | nc $DESTIP 7
CURRTIME=`date +%s%N`
echo Elapsed: $(( ($CURRTIME - $LASTRUN) / 1000000 ))
LASTRUN=$CURRTIME
done
I am now able to run that script from the pine to my local server and vice versa.
Results for the Pine (512MB):
Running on Pine connecting to echo port on local server
Code:
########
Elapsed: 28
########
Elapsed: 1019
########
Elapsed: 28
########
Elapsed: 29
########
Elapsed: 1028
########
Elapsed: 28
########
Elapsed: 28
########
Elapsed: 1025
########
Elapsed: 28
########
Elapsed: 28
########
Elapsed: 2023
Running on the Server connecting to echo port of the Pine
Code:
Elapsed: 14
########
Elapsed: 1017
########
Elapsed: 13
########
Elapsed: 20
########
Elapsed: 5055
########
Elapsed: 245
########
Elapsed: 14
########
Elapsed: 236
########
Elapsed: 15
########
Elapsed: 13
########
Elapsed: 5050
########
Elapsed: 5055
########
########
Elapsed: 10024
########
Elapsed: 209
########
Elapsed: 1013
And again the same test with th PINE64+
Results for the PINE64+ (1GB):
Running on Pine connecting to echo port on local server
Code:
########
Elapsed: 20
########
Elapsed: 21
########
Elapsed: 21
########
Elapsed: 21
########
Elapsed: 21
########
Elapsed: 21
########
Elapsed: 31
########
Elapsed: 25
########
Elapsed: 24
########
Elapsed: 24
########
Elapsed: 24
########
Elapsed: 24
########
Elapsed: 24
########
Elapsed: 26
########
Elapsed: 25
########
Elapsed: 24
########
Elapsed: 23
########
Elapsed: 23
########
Elapsed: 24
########
Elapsed: 24
########
Elapsed: 23
########
Elapsed: 24
########
Elapsed: 23
########
Elapsed: 24
Running on the Server connecting to echo port of the Pine
Code:
########
Elapsed: 15
########
Elapsed: 15
########
Elapsed: 15
########
Elapsed: 14
########
Elapsed: 16
########
Elapsed: 13
########
Elapsed: 16
########
Elapsed: 12
########
Elapsed: 12
########
Elapsed: 13
########
Elapsed: 13
########
Elapsed: 12
########
Elapsed: 12
########
Elapsed: 16
########
Elapsed: 13
########
Elapsed: 12
########
Elapsed: 15
########
Elapsed: 16
########
Elapsed: 12
########
Elapsed: 14
########
Elapsed: 13
########
Elapsed: 13
########
Elapsed: 13
########
Elapsed: 13
########
Elapsed: 12
########
Elapsed: 13
########
Elapsed: 17
########
Elapsed: 12
########
Elapsed: 13
########
Elapsed: 18
########
Elapsed: 11
########
Elapsed: 13
########
Elapsed: 12
########
Elapsed: 12
########
Elapsed: 13
########
Elapsed: 12
########
Elapsed: 13
########
Elapsed: 13
########
Elapsed: 13
########
Elapsed: 12
########
Elapsed: 14
########
Elapsed: 12
Using echo I have removed a lot of overhead from the tcpdump, comparing to ssh. It is now very easy to debug single sessions out of an larger tcpdump.
I see now, that the roundtrips of single packets are quiet OK. The data and the FIN packet of the echo-request is ACKed immediately, however the response packet is just delayed 5 to 10 seconds.
In the attached tcpdump the second echo request is delayed by 2 sec, the forth delayed by 10sec and the 9th is delayed by 5sec!
For me it looks like, that something must be wrong deep in the tcp-stack. Probably it has to do with the NIC-driver, as that is the only difference between PINE64 and PINE64+
I am clueless right now.
If there are some more tools, how i can produce other logs , especially about the dataflow through the TCP-stack, please provide me with hints.