1Gbps clock sync

stepw · (This post was last modified: 09-17-2016, 12:19 PM by stepw.)

I have a 'bad' PINE64+ board, running armbian with legacy kernel, powered via micro-USB AC adapter. NIC and UART0 serial are the only active peripherals. 100Mbps is perfectly fine, but there's a major packet loss at 1Gbps, no need for iperf to tell there's an issue.

VDD33 voltage is between 3.26V and 3.27V, regardless of
- 1Gbps or 100Mbps or 10Mbps link speed
- CPU idle (/sys/class/thermal/thermal_zone0/temp @ 42 degrees) or busy (/sys/class/thermal/thermal_zone0/temp @ 65 degrees)
- 3 different microUSB AC 5V adapters and cables

That voltage is within tolerance range of 2.97V to 3.63V for PHY VDD33 (per PHY DS). PMIC AC input tolerance range is 3.5V to 7V (per PMIC DS), so PHY should be getting conditioned ~3.3V consistently, even when microUSB AC source is used. Unfortunately I don't have a scope to monitor continuously, but if AC-IN voltage drops were affecting 1Gbps communications, I'd expect them to equally affect 100Mbps communications and that is not the case. So the issue with my board doesn't seem to be caused by PHY input voltage fluctuations.

However, there is a noticeable difference in the amount of packet loss with different power sources:
1. laptop USB, so <500mA, 2 feet USB cable
PC -> SBC: 30% loss
SBC -> PC: 36% loss

2. AC 5V/2A adapter, 6 feet USB cable
PC -> SBC: 5% loss
SBC -> PC: 11% loss

3. AC 5V/3A adapter, 3 feet USB cable
PC -> SBC: 15% loss
SBC -> PC: 31% loss

I'd speculate that there's possibly another element sensitive to AC input inconsistencies that might not be conditioned by the PMIC or that may have a narrow tolerance range. Alternatively, PINE64+ PCB layout might not be strictly following guidelines for RTL8211E IC as set forth in PHY DS, resulting in EMI affecting its operation or VDD10 conditioned input being outside of a narrow 0.95V to 1.06V tolerance range.

References:
PHY DS - http://files.pine64.org/doc/datasheet/pine64/rtl8211e(g)-vb(vl)-cg_datasheet_1.6.pdf
PMIC DS - http://files.pine64.org/doc/datasheet/pi...t_V1.0.pdf

So I've fiddled around with EMAC IC and its RGMII interface to PHY IC.

Some findings:
1. MII RXERC = 0 in mii-tool reg dump, supposedly that indicates PHY is not receiving erroneous frames
#mii-tool -vvv
registers for MII PHY 0:
1140 796d 001c c915 01e1 cde1 000d 2001
6801 0300 7800 0000 0000 0000 0000 3000
016e acc2 9f01 0000 8040 1006 4100 2100
0000 8c00 0040 0106 21fc 8038 0123 0000
2. EMAC reports RGMII speed/duplex and clock is consistent with PHY link type

1Gbps/full:
EMAC BASIC_CTL_0:
#devmem2 0x1c30000 w

0x1 = 1Gbps/full
EMAC RGMII_STA:
#devmem2 0x1c300d0 w

0xD = 0b1101 - 1Gbps, 125MHz clock
MII PHY:
#mii-tool -vvv
registers for MII PHY 0:
1140 796d 001c c915 0001 cde1 000f 2001
6801 0200 7800 0000 0000 0000 0000 3000
016e acc2 9f01 6c52 8040 1006 4100 2100
0000 8c00 0040 0106 21fc 8038 0123 0000
100Mbps/full:
EMAC BASIC_CTL_0:
#devmem2 0x1c30000 w

0xD = 100Mbps/full
EMAC RGMII_STA:
#devmem2 0x1c300d0 w

0xB = 0b1011 - 100Mbps, 25MHz clock
MII PHY:
#mii-tool -vvv
registers for MII PHY 0:
1140 796d 001c c915 01e1 cde1 000d 2001
6801 0300 7800 0000 0000 0000 0000 3000
016e acc2 9f01 0000 8040 1006 4100 2100
0000 8c00 0040 0106 21fc 8038 0123 0000

3. EMAC is using internal clock for 1Gbps/125MHz
A64 EMAC_CLK_REG:
#devmem2 0x1c00030 w

clock 0b01 = GMII/RGMII external
clock 0b10 = GMII/RGMII internal (default)
This said, there could be a problem with 125MHz clock sync, required for proper RGMII communication between EMAC and PHY at 1Gbps rate. Judging by latest PCB diagrams, PHY CLK125 is attached to EMAC, presumably to supply 125MHz clock to EMAC, however EMAC's internal clock is enabled (by the EMAC driver?), so EMAC and PHY clocks are not synchronized. However, switching to external clock (supplied by PHY?) results in even higher packet loss, upwards of 80%

I was trying to dump EMAC DMA TX/RX descriptors and buffers to get an idea of whether frames received by PHY make it to EMAC over RGMII and vice versa, but so far didn't manage to access them from userland. I guess it would take to debug EMAC driver to monitor them and accumulate TX/RX frames and errors statistics.

Above is a re-post from my earlier post at armbian forum.

I don't know if my findings are conclusive, but I'd be interested to know if EMAC internal vs. external clock makes a difference on other boards.

Could someone who has a Pine64+ board with working 1Gbps NIC
1. Confirm if EMAC clock is internal on their board?
2. Try switching EMAC clock to external to confirm if that cripples 1Gbps performance?

Clock source change is volatile, can be reverted manuually or by reloading the board, so there's no permanent effect.

It would take to:
a). install devmem2

Code:
# sudo apt-get install devmem2

b). read EMAC register EMAC_CLK_REG at 0x1c00030

Code:
# devmem2 0x1c00030 w

Sample output:
/dev/mem opened.
Memory mapped at address 0x7f88009000.
Value at address 0x1C00030 (0x7f88009030): 0x10C06

c). test ping to default gateway

Code:
# ping `ip route show 0.0.0.0/0 | awk '{print $3}'`

Sample output:
PING 192.168.137.1 (192.168.137.1) 56(84) bytes of data.
64 bytes from 192.168.137.1: icmp_seq=2 ttl=128 time=0.725 ms
64 bytes from 192.168.137.1: icmp_seq=3 ttl=128 time=0.397 ms
64 bytes from 192.168.137.1: icmp_seq=5 ttl=128 time=0.366 ms
64 bytes from 192.168.137.1: icmp_seq=6 ttl=128 time=0.436 ms
64 bytes from 192.168.137.1: icmp_seq=7 ttl=128 time=0.303 ms
^C
--- 192.168.137.1 ping statistics ---
7 packets transmitted, 5 received, 28% packet loss, time 6004ms

d). write EMAC register EMAC_CLK_REG at 0x1c00030

Code:
# devmem2 0x1c00030 w 0x10C05

Sample output:
/dev/mem opened.
Memory mapped at address 0x7f911c2000.
Value at address 0x1C00030 (0x7f911c2030): 0x10C06
Written 0x10C05; readback 0x10C05

e). test ping to default gateway

Code:
# ping `ip route show 0.0.0.0/0 | awk '{print $3}'`

Sample output:
PING 192.168.137.1 (192.168.137.1) 56(84) bytes of data.
64 bytes from 192.168.137.1: icmp_seq=3 ttl=128 time=0.479 ms
64 bytes from 192.168.137.1: icmp_seq=7 ttl=128 time=0.398 ms
64 bytes from 192.168.137.1: icmp_seq=12 ttl=128 time=0.504 ms
^C
--- 192.168.137.1 ping statistics ---
14 packets transmitted, 3 received, 78% packet loss, time 13004ms

f). revert to internal clock

Code:
# devmem2 0x1c00030 w 0x10C06

Sample output:
/dev/mem opened.
Memory mapped at address 0x7f911c2000.
Value at address 0x1C00030 (0x7f911c2030): 0x10C05
Written 0x10C06; readback 0x10C06

Reference:
page 158 of http://files.pine64.org/doc/datasheet/pi...l_V1.0.pdf

**pfeerick** · 09-17-2016, 06:50 PM

Take 1 (SSH over ethernet... this can't end well!!)

a) self-compiled devmem2 as wasn't in my packges list... wget www.lartmaker.nl/lartware/port/devmem2.c && gcc -o devmem2 devmem2.c

b). read EMAC register EMAC_CLK_REG at 0x1c00030

Code:
pfeerick@pine64:~$ sudo ./devmem2 0x1c00030 w

/dev/mem opened.

Memory mapped at address 0x7f8f8d4000.

Value at address 0x1C00030 (0x7f8f8d4030): 0x10C06

c). test ping to default gateway

Code:
pfeerick@pine64:~$ ping `ip route show 0.0.0.0/0 | awk '{print $3}'`

PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.

64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.512 ms

64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.761 ms

64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=0.476 ms

64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=0.536 ms

64 bytes from 192.168.0.1: icmp_seq=5 ttl=64 time=0.498 ms

64 bytes from 192.168.0.1: icmp_seq=6 ttl=64 time=0.725 ms

64 bytes from 192.168.0.1: icmp_seq=7 ttl=64 time=0.489 ms

64 bytes from 192.168.0.1: icmp_seq=8 ttl=64 time=0.896 ms

64 bytes from 192.168.0.1: icmp_seq=9 ttl=64 time=0.718 ms

64 bytes from 192.168.0.1: icmp_seq=10 ttl=64 time=0.476 ms

^C

--- 192.168.0.1 ping statistics ---

10 packets transmitted, 10 received, 0% packet loss, time 9009ms

rtt min/avg/max/mdev = 0.476/0.608/0.896/0.147 ms

d). write EMAC register EMAC_CLK_REG at 0x1c00030

Code:
sudo ./devmem2 0x1c00030 w 0x10C05

promptly lost SSH connection... and device wouldn't reconnect on cable re-insertion. Good thing I'd started an auto-reboot timer just in case ;-)

---

Take 2 (with serial connection this time)

b). read EMAC register EMAC_CLK_REG at 0x1c00030

Code:
pfeerick@pine64:~$ sudo ./devmem2 0x1c00030 w                                                       

/dev/mem opened.                                                                                    

Memory mapped at address 0x7fae01c000.                                                              

Value at address 0x1C00030 (0x7fae01c030): 0x10C06 

c). test ping to default gateway

Code:
pfeerick@pine64:~$ ping `ip route show 0.0.0.0/0 | awk '{print $3}'`                                

PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.                                                

64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.931 ms                                          

64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.568 ms                                          

64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=0.796 ms                                          

64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=0.618 ms                                          

64 bytes from 192.168.0.1: icmp_seq=5 ttl=64 time=0.847 ms                                          

64 bytes from 192.168.0.1: icmp_seq=6 ttl=64 time=0.674 ms                                          

64 bytes from 192.168.0.1: icmp_seq=7 ttl=64 time=0.892 ms                                          

64 bytes from 192.168.0.1: icmp_seq=8 ttl=64 time=0.714 ms                                          

64 bytes from 192.168.0.1: icmp_seq=9 ttl=64 time=0.740 ms                                          

64 bytes from 192.168.0.1: icmp_seq=10 ttl=64 time=0.563 ms                                         

^C                                                                                                  

--- 192.168.0.1 ping statistics ---                                                                 

10 packets transmitted, 10 received, 0% packet loss, time 8993ms                                    

rtt min/avg/max/mdev = 0.563/0.734/0.931/0.126 ms

d). write EMAC register EMAC_CLK_REG at 0x1c00030

Code:
pfeerick@pine64:~$ sudo ./devmem2 0x1c00030 w 0x10C05

/dev/mem opened.                                                                                    

Memory mapped at address 0x7f908b4000.                                                              

Value at address 0x1C00030 (0x7f908b4030): 0x10C06                                                  

Written 0x10C05; readback 0x10C05 

e). test ping to default gateway

Code:
pfeerick@pine64:~$ ping `ip route show 0.0.0.0/0 | awk '{print $3}'`                                

PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.                                                

64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.580 ms                                          

64 bytes from 192.168.0.1: icmp_seq=7 ttl=64 time=0.744 ms                                          

64 bytes from 192.168.0.1: icmp_seq=8 ttl=64 time=0.767 ms                                          

64 bytes from 192.168.0.1: icmp_seq=10 ttl=64 time=0.786 ms                                         

64 bytes from 192.168.0.1: icmp_seq=11 ttl=64 time=1.20 ms                                          

64 bytes from 192.168.0.1: icmp_seq=14 ttl=64 time=0.691 ms                                         

64 bytes from 192.168.0.1: icmp_seq=15 ttl=64 time=0.717 ms                                         

64 bytes from 192.168.0.1: icmp_seq=16 ttl=64 time=0.746 ms                                         

64 bytes from 192.168.0.1: icmp_seq=18 ttl=64 time=0.802 ms                                         

64 bytes from 192.168.0.1: icmp_seq=19 ttl=64 time=0.831 ms                                         

^C                                                                                                  

--- 192.168.0.1 ping statistics ---                                                                 

20 packets transmitted, 10 received, 50% packet loss, time 19012ms                                  

rtt min/avg/max/mdev = 0.580/0.786/1.201/0.155 ms

Other notes:
After executing d) above, very poor ping response from other device on network. Cable re-insertion made no difference whatsover.

Pinging pine from windows box:

Code:
Pinging 192.168.0.57 with 32 bytes of data:

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Reply from 192.168.0.57: bytes=32 time<1ms TTL=64

Request timed out.

Reply from 192.168.0.57: bytes=32 time<1ms TTL=64

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Ping statistics for 192.168.0.57:

    Packets: Sent = 16, Received = 2, Lost = 14 (87% loss),

Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms

Running e) (revert clock back to internal) resulted in immediate return to normal response.

Code:
pfeerick@pine64:~$ ping `ip route show 0.0.0.0/0 | awk '{print $3}'`                                

PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.                                                

64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.904 ms                                          

64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.579 ms                                          

64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=0.606 ms                                          

64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=0.427 ms                                          

64 bytes from 192.168.0.1: icmp_seq=5 ttl=64 time=0.647 ms                                          

64 bytes from 192.168.0.1: icmp_seq=6 ttl=64 time=0.468 ms                                          

64 bytes from 192.168.0.1: icmp_seq=7 ttl=64 time=0.687 ms                                          

64 bytes from 192.168.0.1: icmp_seq=8 ttl=64 time=0.506 ms                                          

64 bytes from 192.168.0.1: icmp_seq=9 ttl=64 time=0.726 ms                                          

64 bytes from 192.168.0.1: icmp_seq=10 ttl=64 time=0.549 ms                                         

^C                                                                                                  

--- 192.168.0.1 ping statistics ---                                                                 

10 packets transmitted, 10 received, 0% packet loss, time 8993ms                                    

rtt min/avg/max/mdev = 0.427/0.609/0.904/0.136 ms

Pinging pine64 from windows box again:

Code:
C:\Users\Peter>ping -t 192.168.0.57

Pinging 192.168.0.57 with 32 bytes of data:

Reply from 192.168.0.57: bytes=32 time<1ms TTL=64

Reply from 192.168.0.57: bytes=32 time<1ms TTL=64

Reply from 192.168.0.57: bytes=32 time<1ms TTL=64

Reply from 192.168.0.57: bytes=32 time=1ms TTL=64

Reply from 192.168.0.57: bytes=32 time=1ms TTL=64

Reply from 192.168.0.57: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.0.57:

    Packets: Sent = 6, Received = 6, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 1ms, Average = 0ms

MarkHaysHarris777 · 09-17-2016, 08:10 PM

@stepw, if you have a scope it might be helpful to probe the PHY_VDD33 voltage while in GbE mode and be looking for a ripple in the voltage (signal on the VDD) ?

stepw · 09-18-2016, 08:57 PM

(09-17-2016, 06:50 PM)pfeerick Wrote: Other notes:
After executing d) above, very poor ping response from other device on network. Cable re-insertion made no difference whatsover.

Thank you so much pfeerick!

So that confirms:
1. Internal EMAC transmit clock (on A64 SoC) is used both on boards working well at 1Gbps and those that don't
2. External EMAC transmit clock (from RTL PHY or a separate source) is resulting in severe packet loss on both too

On "good" boards EMAC and PHY appear to be passing data reliably while using their respective internal clocks, resulting in no data loss and reasonable 1Gbps performance, which could reach link capacity with some software tweaking. On "bad" boards, like mine, looks like there could be a clock/timing issue between EMAC and PHY in Pine64's configuration and board layout, resulting in major loss of data as it transitions between layer1 and layer2 or vice-versa. Perhaps clock skew is outside of acceptable range or internal 125MHz clock is not stable either on EMAC or on PHY. It may or may not be related to voltage anomalies, I don't have the skills or tools to confirm voltage as the root cause.

Per RTL8211E DS, a 125MHz clock reference can be supplied by PHY to EMAC for clock to be in sync:
1. Features:
Supports 25/50MHz external crystal or OSC
Provides 125MHz clock source for MAC
2. RGMII mode
In 1000Base-T mode (RGMII interface is selected), TXC and RXC sources are 125 MHz.
TXC will always be generated by the MAC and RXC will always be generated by the PHY.

If I'm reading Rev.C PCB layout diagrams correctly, RTL8211E CLK125 is attached to A64 GCLKIN, which, once external TX clock is enabled for EMAC (with devmem2 or in driver), should have sync'ed the clock between PHY and EMAC. However, enabling it makes things even worse, kind of counter-logical to me.

It's not clear from A64 user guide if its EMAC uses internal clock for 100Mbps communication as well, but associated 25MHz clock is external on PHY and that appears to work well/stable on all boards, regardless of power source and base input voltage.

(09-17-2016, 08:10 PM)MarkHaysHarris777 Wrote: @stepw, if you have a scope it might be helpful to probe the PHY_VDD33 voltage while in GbE mode and be looking for a ripple in the voltage (signal on the VDD) ?

Unfortunately I don't have a scope to monitor continuously, but if PMIC AC-IN voltage drops were affecting 1Gbps communications, I'd expect them to equally affect 100Mbps communications and that is not the case. So the issue with my board doesn't seem to be caused by PHY VDD33/AVDD33 input voltage fluctuations. It could be related to 1V fluctuations off of PHY's internal regulator on PHY VDD10/AVVD10 pins, I have not measured that.

I wonder if the inductor tllim suggested could be shorted to fix 1Gbps is part of 1V voltage regulator scheme. Unfortunately he hasn't commented when I asked him in the other thread.

Login




Remember me Lost Password?

About Us