Rockpro64 Headless Server Reliably Becomes Unresponsive
#1
Hello, All.

I am attempting to use a Rockpro64 with bionic-minimal-rockpro64-0.7.9-1067-arm64.img as a headless server.

When initially powered on, everything works as expected, but after an amount of time that seems to vary between six hours and two days, the Rockpro64 stops responding to ssh, pings, and web requests. I tried connecting a monitor and keyboard to troubleshoot the problem. From the Rockpro64, the following things are true:

1. I can successfully ping other computers on the network by IP address, but pinging domains fails.
2. I can ssh into localhost, but cannot ssh into any other servers, even by referencing their IP addresses.
3. "arp" runs extremely slowly, but "arp -n" completes near-instantaneously.
4. Restarting the Rockpro64 fixes the problem temporarily, until it inevitably  comes back.

It looks like a DNS issue to me, but I do not know where to look from there. Do any of you wonderful people have suggestions?
  Reply
#2
Try to analyse dmesg output. May be there will be a complains for some interface or service. Also check free memory and process list. May be the cause is a memory leak or similar. Also, if possible, try different distributive.
  Reply
#3
It sounds exactly like a DNS outage. What are the contents of /etc/resolv.conf (which specifies your DNS nameservers) ?
  Reply
#4
(06-05-2019, 10:51 PM)Nikolay_Po Wrote: Try to analyse dmesg output. May be there will be a complains for some interface or service. Also check free memory and process list. May be the cause is a memory leak or similar. Also, if possible, try different distributive.

Thanks, I will take a look at dmesg and see if I can learn anything.

I set cron to record the output of "free" every hour, and ram usage never got over 2GB usage. I am on the 4GB board, and have plenty of swap available, so I doubt that this is a memory leak issue.

I may end up needing to switch distros.
  Reply
#5
(06-06-2019, 07:05 PM)mknawabi Wrote: It sounds exactly like a DNS outage. What are the contents of /etc/resolv.conf (which specifies your DNS nameservers) ?

Thanks for your response. Command outputs are below.

cat /etc/resolv.conf:
Quote:# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53

systemd-resolve --status:
Quote:Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 1.1.1.1
                      192.168.0.1

The DNS is coming from my router's DHCP server, which was previously relying on 8.8.8.8, but I changed it in hopes that it would solve the issue. It did not.
  Reply
#6
(06-07-2019, 04:06 PM)Doneganai7 Wrote:
(06-06-2019, 07:05 PM)mknawabi Wrote: It sounds exactly like a DNS outage. What are the contents of /etc/resolv.conf (which specifies your DNS nameservers) ?

Thanks for your response. Command outputs are below.

cat /etc/resolv.conf:
Quote:# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53

systemd-resolve --status:
Quote:Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 1.1.1.1
                      192.168.0.1

The DNS is coming from my router's DHCP server, which was previously relying on 8.8.8.8, but I changed it in hopes that it would solve the issue. It did not.

If you have dig installed, you can use dig @nameserver google.com to specify the name server to use for a lookup. That way, you can run a DNS lookup against 127.0.0.53 (the systemd local resolver), and the IP of your router. You can also use dig @8.8.8.8 google.com to use google's DNS servers to look up google.com. 

This way you check your local system, the home network router, and then you finally can test internet connectivity + name server lookups against a well-known working DNS server
  Reply
#7
(06-07-2019, 05:04 PM)mknawabi Wrote: If you have dig installed, you can use dig @nameserver google.com to specify the name server to use for a lookup. That way, you can run a DNS lookup against 127.0.0.53 (the systemd local resolver), and the IP of your router. You can also use dig @8.8.8.8 google.com to use google's DNS servers to look up google.com. 

This way you check your local system, the home network router, and then you finally can test internet connectivity + name server lookups against a well-known working DNS server

Great. Yes, I have dig installed. I will wait until the issue comes up again and report back with more information about what dig reports.

I appreciate the guidance.
  Reply
#8
I have collected more data, but I am not sure what to make of it.

ping -c 4 192.168.0.1 # This is my router

Quote:PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=3.15 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.669 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=0.656 ms
64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=0.808 ms

--- 192.168.0.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 0.656/1.322/3.156/1.060 ms


ping -c 4 1.1.1.1
Quote:PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=11.1 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=56 time=23.4 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=56 time=13.5 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=56 time=14.1 ms

--- 1.1.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 11.142/15.587/23.474/4.692 ms

ping -c 4 google.com
Quote:ping: google.com: Name or service not known

dig google.com

Quote:; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> google.com
;; global options: +cmd
;; connection timed out; no servers could be reached


dig @1.1.1.1 google.com
Quote:; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> @1.1.1.1 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

dig @8.8.8.8 google.com
Quote:; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> @8.8.8.8 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

I was also monitoring "dmesg | tail" and nothing new came up between when everything was working, and when it went back to broken.

I am not sure why I would be able to successfully ping a well known DNS server, but not use it to resolve IPs. Does this give any clues as to where to look next?
  Reply
#9
For anyone who has this problem in the future, I could not find a solution using bionic-minimal-rockpro64-0.7.9-1067-arm64.img.

I just hit 48 hours of continuous uptime with Armbian_5.88_Rockpro64_Ubuntu_bionic_default_4.4.180_desktop.img though. Everything seems to be running correctly now (including dm-crypt, mdadm, and ats, some of which had issues on other images).
  Reply


Possibly Related Threads...
Thread Author Replies Views Last Post
  Rockpro64 not stable... crashes now and then. Pineapple 0 22 9 hours ago
Last Post: Pineapple
Thumbs Up Batocera for RockPro64 Luke 3 1,800 11-06-2019, 02:12 PM
Last Post: Wizzard
  DLNA and/or Plex problem with RockPro64 aldrick 10 256 10-20-2019, 10:34 PM
Last Post: aldrick
  slarm64 (unofficial slackware) ROCKPro64 RK3399 (aarch64) mara 24 4,042 10-20-2019, 06:51 AM
Last Post: mara
  Arch Linux on RockPro64 mmatyas 27 8,753 09-28-2019, 04:30 AM
Last Post: TaborFife
  Rockpro64 and stock arch linux TaborFife 0 154 09-21-2019, 04:29 PM
Last Post: TaborFife
  RockPro64 - Armbian no sound from jack Pineapple 0 129 09-16-2019, 08:27 AM
Last Post: Pineapple
  PCIe ath10k on RockPro64 dasfranky 5 198 08-31-2019, 11:36 AM
Last Post: tuxd3v
  RockPro64 Official Kernel Support ASIC 23 3,840 08-29-2019, 05:39 PM
Last Post: tuxd3v
  DietPi for the RockPro64 Luke 12 4,175 08-29-2019, 04:07 AM
Last Post: hexxx

Forum Jump:


Users browsing this thread: 1 Guest(s)