+- PINE64 (https://forum.pine64.org)
+-- Forum: ROCKPRO64 (https://forum.pine64.org/forumdisplay.php?fid=98)
+--- Forum: Linux on RockPro64 (https://forum.pine64.org/forumdisplay.php?fid=101)
+--- Thread: Rockpro64 Headless Server Reliably Becomes Unresponsive (/showthread.php?tid=7561)
Rockpro64 Headless Server Reliably Becomes Unresponsive - Doneganai7 - 06-05-2019
Hello, All.
I am attempting to use a Rockpro64 with bionic-minimal-rockpro64-0.7.9-1067-arm64.img as a headless server.
When initially powered on, everything works as expected, but after an amount of time that seems to vary between six hours and two days, the Rockpro64 stops responding to ssh, pings, and web requests. I tried connecting a monitor and keyboard to troubleshoot the problem. From the Rockpro64, the following things are true:
1. I can successfully ping other computers on the network by IP address, but pinging domains fails.
2. I can ssh into localhost, but cannot ssh into any other servers, even by referencing their IP addresses.
3. "arp" runs extremely slowly, but "arp -n" completes near-instantaneously.
4. Restarting the Rockpro64 fixes the problem temporarily, until it inevitably comes back.
It looks like a DNS issue to me, but I do not know where to look from there. Do any of you wonderful people have suggestions?
RE: Rockpro64 Headless Server Reliably Becomes Unresponsive - Nikolay_Po - 06-05-2019
Try to analyse dmesg output. May be there will be a complains for some interface or service. Also check free memory and process list. May be the cause is a memory leak or similar. Also, if possible, try different distributive.
RE: Rockpro64 Headless Server Reliably Becomes Unresponsive - mknawabi - 06-06-2019
It sounds exactly like a DNS outage. What are the contents of /etc/resolv.conf (which specifies your DNS nameservers) ?
RE: Rockpro64 Headless Server Reliably Becomes Unresponsive - Doneganai7 - 06-07-2019
(06-05-2019, 10:51 PM)Nikolay_Po Wrote: Try to analyse dmesg output. May be there will be a complains for some interface or service. Also check free memory and process list. May be the cause is a memory leak or similar. Also, if possible, try different distributive.
Thanks, I will take a look at dmesg and see if I can learn anything.
I set cron to record the output of "free" every hour, and ram usage never got over 2GB usage. I am on the 4GB board, and have plenty of swap available, so I doubt that this is a memory leak issue.
I may end up needing to switch distros.
RE: Rockpro64 Headless Server Reliably Becomes Unresponsive - Doneganai7 - 06-07-2019
(06-06-2019, 07:05 PM)mknawabi Wrote: It sounds exactly like a DNS outage. What are the contents of /etc/resolv.conf (which specifies your DNS nameservers) ?
Thanks for your response. Command outputs are below.
cat /etc/resolv.conf:
Quote:# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.
nameserver 127.0.0.53
systemd-resolve --status:
Quote:Global
DNSSEC NTA: 10.in-addr.arpa
16.172.in-addr.arpa
168.192.in-addr.arpa
17.172.in-addr.arpa
18.172.in-addr.arpa
19.172.in-addr.arpa
20.172.in-addr.arpa
21.172.in-addr.arpa
22.172.in-addr.arpa
23.172.in-addr.arpa
24.172.in-addr.arpa
25.172.in-addr.arpa
26.172.in-addr.arpa
27.172.in-addr.arpa
28.172.in-addr.arpa
29.172.in-addr.arpa
30.172.in-addr.arpa
31.172.in-addr.arpa
corp
d.f.ip6.arpa
home
internal
intranet
lan
local
private
test
Link 2 (eth0)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Servers: 1.1.1.1
192.168.0.1
The DNS is coming from my router's DHCP server, which was previously relying on 8.8.8.8, but I changed it in hopes that it would solve the issue. It did not.
RE: Rockpro64 Headless Server Reliably Becomes Unresponsive - mknawabi - 06-07-2019
(06-07-2019, 04:06 PM)Doneganai7 Wrote:(06-06-2019, 07:05 PM)mknawabi Wrote: It sounds exactly like a DNS outage. What are the contents of /etc/resolv.conf (which specifies your DNS nameservers) ?
Thanks for your response. Command outputs are below.
cat /etc/resolv.conf:
Quote:# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.
nameserver 127.0.0.53
systemd-resolve --status:
Quote:Global
DNSSEC NTA: 10.in-addr.arpa
16.172.in-addr.arpa
168.192.in-addr.arpa
17.172.in-addr.arpa
18.172.in-addr.arpa
19.172.in-addr.arpa
20.172.in-addr.arpa
21.172.in-addr.arpa
22.172.in-addr.arpa
23.172.in-addr.arpa
24.172.in-addr.arpa
25.172.in-addr.arpa
26.172.in-addr.arpa
27.172.in-addr.arpa
28.172.in-addr.arpa
29.172.in-addr.arpa
30.172.in-addr.arpa
31.172.in-addr.arpa
corp
d.f.ip6.arpa
home
internal
intranet
lan
local
private
test
Link 2 (eth0)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Servers: 1.1.1.1
192.168.0.1
The DNS is coming from my router's DHCP server, which was previously relying on 8.8.8.8, but I changed it in hopes that it would solve the issue. It did not.
If you have dig installed, you can use dig @nameserver google.com to specify the name server to use for a lookup. That way, you can run a DNS lookup against 127.0.0.53 (the systemd local resolver), and the IP of your router. You can also use dig @8.8.8.8 google.com to use google's DNS servers to look up google.com.
This way you check your local system, the home network router, and then you finally can test internet connectivity + name server lookups against a well-known working DNS server
RE: Rockpro64 Headless Server Reliably Becomes Unresponsive - Doneganai7 - 06-08-2019
(06-07-2019, 05:04 PM)mknawabi Wrote: If you have dig installed, you can use dig @nameserver google.com to specify the name server to use for a lookup. That way, you can run a DNS lookup against 127.0.0.53 (the systemd local resolver), and the IP of your router. You can also use dig @8.8.8.8 google.com to use google's DNS servers to look up google.com.
This way you check your local system, the home network router, and then you finally can test internet connectivity + name server lookups against a well-known working DNS server
Great. Yes, I have dig installed. I will wait until the issue comes up again and report back with more information about what dig reports.
I appreciate the guidance.
RE: Rockpro64 Headless Server Reliably Becomes Unresponsive - Doneganai7 - 06-09-2019
I have collected more data, but I am not sure what to make of it.
ping -c 4 192.168.0.1 # This is my router
Quote:PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=3.15 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.669 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=0.656 ms
64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=0.808 ms
--- 192.168.0.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 0.656/1.322/3.156/1.060 ms
ping -c 4 1.1.1.1
Quote:PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=11.1 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=56 time=23.4 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=56 time=13.5 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=56 time=14.1 ms
--- 1.1.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 11.142/15.587/23.474/4.692 ms
ping -c 4 google.com
Quote:ping: google.com: Name or service not known
dig google.com
Quote:; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> google.com
;; global options: +cmd
;; connection timed out; no servers could be reached
dig @1.1.1.1 google.com
Quote:; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> @1.1.1.1 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
dig @8.8.8.8 google.com
Quote:; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> @8.8.8.8 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
I was also monitoring "dmesg | tail" and nothing new came up between when everything was working, and when it went back to broken.
I am not sure why I would be able to successfully ping a well known DNS server, but not use it to resolve IPs. Does this give any clues as to where to look next?
RE: Rockpro64 Headless Server Reliably Becomes Unresponsive - Doneganai7 - 06-15-2019
For anyone who has this problem in the future, I could not find a solution using bionic-minimal-rockpro64-0.7.9-1067-arm64.img.
I just hit 48 hours of continuous uptime with Armbian_5.88_Rockpro64_Ubuntu_bionic_default_4.4.180_desktop.img though. Everything seems to be running correctly now (including dm-crypt, mdadm, and ats, some of which had issues on other images).