pine64 512MB network unusable

[longsleep]

Play around with https://github.com/longsleep/build-pine6....dts#L2562 and https://github.com/longsleep/build-pine6....dts#L2563

[Manilow]

And one more observation:

The pine64(512) stays at 480MHz and 36°C temperature, when I run my testscript, while the PINE64+ (1GB) jumps up to 912MHz and 53°C, when running the script. This looks like a chicken-or-egg problem. Is the CPU frequency not rising, because the network is to slow, or has the network problems, because the CPU is not running faster?

I still don't get, where the problem could be. It is annoying, that interactive work is not possible, because the terminal stucks up to 10sec, while typing. Same with download rates reminding to old modem times.  However iperf3 shows the right performance, while also a lot of retries.  Which layer of the OS is iperf3 circumventing?

Reminder:  For my tests, I start both pines alternating with the same SD-card, connected to the same network cable and the same power supply.  So there can't be nothing different in in the configuration.

---

Play around with https://github.com/longsleep/build-pine6....dts#L2562 and https://github.com/longsleep/build-pine6....dts#L2563

Hi,

how the decision is made, which of those dtb files is used?  Right now there are both files in the /boot/pine64 folder. 
Furthermore: The files in the /boot/pine64 directory or binary, while your github files are text. How do I convert them to binary?

[tkaiser]

The pine64(512) stays at 480MHz and 36°C temperature, when I run my testscript, while the PINE64+ (1GB) jumps up to 912MHz and 53°C, when running the script. [...]

how the decision is made, which of those dtb files is used?  Right now there are both files in the /boot/pine64 folder.

Regarding .dtb first: This is u-boot's job. By looking at the available DRAM the correct .dtb will be chosen (using the wrong .dtb would most probably result in no network at all but I'm not 100% sure).

Cpufreq does matter (at least with ARMv7 boards this was an observation) so please consider adding priot to "exit 0" to /etc/rc.local:

Code:

echo performance >/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

And then regarding TX/RX delay. We have several sunxi boards where depending on these values you get almost no packets through the wire. Longsleep did many testings (details here) but IIRC he was focused on Pine64+ so maybe they're completely off for Pine64. Some background information: http://linux-sunxi.org/Ethernet#GMAC and http://forum.armbian.com/index.php/topic...h-gigabit/

And also another issue that plagued users of Olimex boards that use a different external PHY than most other boards (RTL8211CL instead of RTL8211E): https://www.mail-archive.com/u-boot@list...07006.html

The links aren't meant as 'here you find a solution' but just to outline in which sorts of problems you might run (and I really mean you since I believe of the few linux-sunxi devs that work on Pine64 support only longsleep got also the 512MB board -- and pays not that much attention to it since not just in his opinion the 2GB version with GbE is the only interesting one)

---

Addendum: Using longsleep's monitoring script you can get the actual values but I find it sometimes more convenient to use a real monitoring solution that also gives access to 'historical' data. I provide a simple installation routine for Ubuntu on Pine64+ with all the adjustments for A64/AXP803: http://kaiser-edv.de/tmp/4U4tkD/install-...for-a64.sh (takes a few minutes to install)

---

EDIT: Just realized that you don't use an original OS image but a 3rd party one. In case you haven't done already please ensure you really use latest u-boot/kernel fixes:

Code:

bash <(curl -s https://raw.githubusercontent.com/longsleep/build-pine64-image/master/simpleimage/platform-scripts/pine64_update_uboot.sh)
bash <(curl -s https://raw.githubusercontent.com/longsleep/build-pine64-image/master/simpleimage/platform-scripts/pine64_update_kernel.sh)
bash <(curl -s https://raw.githubusercontent.com/longsleep/build-pine64-image/master/simpleimage/platform-scripts/pine64_fix_whatever.sh)

Just to ensure u-boot is up-to-date to choose the correct .dtb.

---

Furthermore: The files in the /boot/pine64 directory or binary, while your github files are text. How do I convert them to binary?

Sorry, overread that. You need the dtc tool. No idea whether that's part of your distro but since this is Debian an 'apt-get install devicetree-compiler' should do the job (ditto a web search for 'dtc dtb dts' for the basic syntax)

[longsleep]

through the wire. Longsleep did many testings (details here) but IIRC he was focused on Pine64+ so maybe they're completely off for Pine64.

since I believe of the few linux-sunxi devs that work on Pine64 support only longsleep got also the 512MB board -- and pays not that much attention to it since not just in his opinion the 2GB version with GbE is the only interesting one)

Yes, but i run all tests on all 3 Pine64 models always. So the Pine64 100M NIC is good with delays set to 0. With my switches it works just fine - so similar to the 1000M NIC issue this is not a general issue. Its probably even the same issue and it is likely related to the rest of the network equipment. Also the remember that the driver is crap and thus there might be plenty of issues with various network equipments.

Pine64 512M

Code:

~$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.29 port 5001 connected with 192.168.1.232 port 52744
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   113 MBytes  94.2 Mbits/sec

~$ iperf -c 192.168.1.232
------------------------------------------------------------
Client connecting to 192.168.1.232, TCP port 5001
TCP window size: 22.5 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.29 port 35395 connected with 192.168.1.232 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   113 MBytes  94.4 Mbits/sec

This is just my standard image current version everything - no further tweaks / none required.

[Manilow]

Thanks for all your information, which helped me a lot, to understand some more dependencies.
I can confirm, that setting the cpu-clock to performance is a big step in reducing the seen problems.  

• The local download rate (wget of an local http URL) rises from the range [100KB/sec..1.5MB/sec] to the range [5MB/sec..10MB/sec].  
  

• The retry rate (as seen in iperf3 ) goes down from 400-600 retries per session to below 100.
  

• The maximal delay of an echo roundtrip (my script) goes down from 10 sec. to 2 sec.
  

The only left question here is, why the CPU rate goes only to 1152.00 MHz and not 1200.00 MHz, as I would have expected by the specs.

Furthermore I figured out, that at least my PINE(512) behaves like described in some of the above linked discussions: After cold boot the network performance is ways better, than after a warm restart.  The numbers above are all after cold boot.  Running for some time, the rates go down by est. 20%-30%

Unfortunately I see almost no reaction in setting tx-delay and rx-delay in /boot/pine/sun50i-a64-pine64.dtb.
I have started with the dts file from http://github.com/longsleep/build-pine64...noplus.dts and changed the TX and RX values from <0x00000000> to 0x5, later to 0xF0, 0xF00, up to 0xF0000000 and detected no measurable difference.
To avoid any confusion, I deleted the "sun50i-a64-pine64-plus.dtb" file in /boot/pine64, so that I have only the following files in the directory:

• sun50i-a64-pine64.dtb
  
• pine64noplus.dts
  
• Image
  

After changing the values in the dts file, i created the new dtb file and prepared the restart of the pine with that commands:

Code:

cd /boot/pine64
rm sun50i-a64-pine64.dtb
dtc -I dts -O dtb pine64noplus.dts > sun50i-a64-pine64.dtb
cd /
umount /boot # make sure, filesystem is clean!
shutdown -P now
#unplug power cable
#unplug network cable
#reconnect power
#after 10 sec reconnect network

I would have expected rendering my network unusable, setting RX and TX to such high values. So I wonder, why that has no measurable influence.

Can it be, that the PINE512 needs an slightly different network device description, as it has a different NIC on board?  One thought is, that this section is not used and the interface is just configured with some default values.

I have also observed, that the PINE(512) does not setup random MAC addresses like described in the forums.  I can delete /usr/local/sbin/pine64_eth0-mackeeper.sh and delete the "ethaddr=xxxx" line in the /boot/uEnv.txt file.  The PINE(512) starts always with the same MAC.

[tkaiser]

Thanks for all your information, which helped me a lot, to understand some more dependencies.

Great that you keep up. Normally if one comes up with information and suggestions quite the opposite happens here.

Regarding the issues: Can you please try out 'interactive' cpufreq governor and see if that helps also (if so for whatever reasons another governor must have been used by default that didn't increase clockspeeds when needed)?

Regarding 1200 MHz: If you have an hour please read through the whole story. In short words: If you know what you do you can increase that but we as the few people that looked into decided against since reliability would be concerned. Means: We stay at the default 1152 MHz (I've no idea why Pine64 has been advertised to be able to run at 1200 MHz, same applies to the promoted consumption and thermal values -- everything a bit or way off compared with reality)

Now to .dts modifications. I hope longsleep corrects if I'm (again) wrong: But it seems for these changes to take effect you would've to setup a cross compile environment, adjust .dts settings, create u-boot-with-dtb.bin and overwrite it on SD card as can be seen below:

https://github.com/longsleep/build-pine6...ther-parts
https://github.com/longsleep/build-pine6...ot_only.sh

I automated the whole process back in the days we tried to optimise cpufreq/dvfs settings and I doubt it's possible to deal in another way with this if you really have to test through a couple of different delay settings. But since longsleep reports pretty good throughput numbers (that are in sync with the numbers I get on H3 equipped Orange Pis also that share nearly the same driver code but use a different Fast Ethernet PHY) with the defaults I would suspect the real reason is somewhere else.

[Manilow]

Hello,

I have done some more tests and digged in tcpdumps.  I see, that the PINE(512) loses sometimes a packet while receiving. In combination with a really big TCP receive window of 1MB (while the PINE+ stays around 50kB) that leads to severe breakdowns in transfer rate.  
I see the packet while sniffing on the server and sniffing on the wire (with a small NetGear ProSafe Plus Switch as dedicated sniffing TAP before the PINE)
The PINE is sending then up to 130 SACKs, excluding the missing packet (caused by the big transfer window)

Switching windows scaling off on the Pine(512) [sysctl -w net.ipv4.tcp_window_scaling=0 ] has the effect, that the available peak transfer rates are somewhat slower, but the overall transfer rate is significantly higher and more predictable.  But there is still the situation, that network packets are just delayed (which is seen in my echo-script).  This delay may have the same reason as the lost packages or not.

When I switch off generic-receive-offload on the PINE(512) [ethtool -K eth0 gro off ] , the rate of delayed packages is almost disappearing.

I wonder, if that is a problem of my specific PINE (defect in NIC) or if other users are seeing similar problems?

Right now I can use the PINE(512) for my planned project, as with the described changes the connectivity is now reliable.

With some more time, I may start digging in deeper. As for now I will continue with other projects.

Thanks for all your support!