Rockpro64 Headless Server Reliably Becomes Unresponsive
#1
Hello, All.

I am attempting to use a Rockpro64 with bionic-minimal-rockpro64-0.7.9-1067-arm64.img as a headless server.

When initially powered on, everything works as expected, but after an amount of time that seems to vary between six hours and two days, the Rockpro64 stops responding to ssh, pings, and web requests. I tried connecting a monitor and keyboard to troubleshoot the problem. From the Rockpro64, the following things are true:

1. I can successfully ping other computers on the network by IP address, but pinging domains fails.
2. I can ssh into localhost, but cannot ssh into any other servers, even by referencing their IP addresses.
3. "arp" runs extremely slowly, but "arp -n" completes near-instantaneously.
4. Restarting the Rockpro64 fixes the problem temporarily, until it inevitably  comes back.

It looks like a DNS issue to me, but I do not know where to look from there. Do any of you wonderful people have suggestions?
#2
Try to analyse dmesg output. May be there will be a complains for some interface or service. Also check free memory and process list. May be the cause is a memory leak or similar. Also, if possible, try different distributive.
#3
It sounds exactly like a DNS outage. What are the contents of /etc/resolv.conf (which specifies your DNS nameservers) ?
#4
(06-05-2019, 10:51 PM)Nikolay_Po Wrote: Try to analyse dmesg output. May be there will be a complains for some interface or service. Also check free memory and process list. May be the cause is a memory leak or similar. Also, if possible, try different distributive.

Thanks, I will take a look at dmesg and see if I can learn anything.

I set cron to record the output of "free" every hour, and ram usage never got over 2GB usage. I am on the 4GB board, and have plenty of swap available, so I doubt that this is a memory leak issue.

I may end up needing to switch distros.
#5
(06-06-2019, 07:05 PM)mknawabi Wrote: It sounds exactly like a DNS outage. What are the contents of /etc/resolv.conf (which specifies your DNS nameservers) ?

Thanks for your response. Command outputs are below.

cat /etc/resolv.conf:
Quote:# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53

systemd-resolve --status:
Quote:Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 1.1.1.1
                      192.168.0.1

The DNS is coming from my router's DHCP server, which was previously relying on 8.8.8.8, but I changed it in hopes that it would solve the issue. It did not.
#6
(06-07-2019, 04:06 PM)Doneganai7 Wrote:
(06-06-2019, 07:05 PM)mknawabi Wrote: It sounds exactly like a DNS outage. What are the contents of /etc/resolv.conf (which specifies your DNS nameservers) ?

Thanks for your response. Command outputs are below.

cat /etc/resolv.conf:
Quote:# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53

systemd-resolve --status:
Quote:Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 1.1.1.1
                      192.168.0.1

The DNS is coming from my router's DHCP server, which was previously relying on 8.8.8.8, but I changed it in hopes that it would solve the issue. It did not.

If you have dig installed, you can use dig @nameserver google.com to specify the name server to use for a lookup. That way, you can run a DNS lookup against 127.0.0.53 (the systemd local resolver), and the IP of your router. You can also use dig @8.8.8.8 google.com to use google's DNS servers to look up google.com. 

This way you check your local system, the home network router, and then you finally can test internet connectivity + name server lookups against a well-known working DNS server
#7
(06-07-2019, 05:04 PM)mknawabi Wrote: If you have dig installed, you can use dig @nameserver google.com to specify the name server to use for a lookup. That way, you can run a DNS lookup against 127.0.0.53 (the systemd local resolver), and the IP of your router. You can also use dig @8.8.8.8 google.com to use google's DNS servers to look up google.com. 

This way you check your local system, the home network router, and then you finally can test internet connectivity + name server lookups against a well-known working DNS server

Great. Yes, I have dig installed. I will wait until the issue comes up again and report back with more information about what dig reports.

I appreciate the guidance.
#8
I have collected more data, but I am not sure what to make of it.

ping -c 4 192.168.0.1 # This is my router

Quote:PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=3.15 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.669 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=0.656 ms
64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=0.808 ms

--- 192.168.0.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 0.656/1.322/3.156/1.060 ms


ping -c 4 1.1.1.1
Quote:PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=11.1 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=56 time=23.4 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=56 time=13.5 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=56 time=14.1 ms

--- 1.1.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 11.142/15.587/23.474/4.692 ms

ping -c 4 google.com
Quote:ping: google.com: Name or service not known

dig google.com

Quote:; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> google.com
;; global options: +cmd
;; connection timed out; no servers could be reached


dig @1.1.1.1 google.com
Quote:; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> @1.1.1.1 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

dig @8.8.8.8 google.com
Quote:; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> @8.8.8.8 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

I was also monitoring "dmesg | tail" and nothing new came up between when everything was working, and when it went back to broken.

I am not sure why I would be able to successfully ping a well known DNS server, but not use it to resolve IPs. Does this give any clues as to where to look next?
#9
For anyone who has this problem in the future, I could not find a solution using bionic-minimal-rockpro64-0.7.9-1067-arm64.img.

I just hit 48 hours of continuous uptime with Armbian_5.88_Rockpro64_Ubuntu_bionic_default_4.4.180_desktop.img though. Everything seems to be running correctly now (including dm-crypt, mdadm, and ats, some of which had issues on other images).


Possibly Related Threads…
Thread Author Replies Views Last Post
  irradium (based on crux linux) RockPro64 riscv64, aarch64 mara 1 342 03-25-2024, 12:12 PM
Last Post: mara
  yocto for RockPro64 Fide 1 648 01-16-2024, 10:01 AM
Last Post: Fide
  Installing Ubuntu Server on RockPro64 deutschlmao 2 2,517 10-29-2023, 04:43 PM
Last Post: brotherj4mes
  Vanilla mainline Debian 11 (Bullseye) on the RockPro64 Pete Tandy 22 16,821 08-16-2023, 01:34 AM
Last Post: varac
  slarm64 (unofficial slackware) ROCKPro64 RK3399 (aarch64) mara 54 82,196 08-11-2023, 11:13 AM
Last Post: mara
  How to enable CoreSight ETM trace on RockPro64 shpark 0 540 05-21-2023, 11:34 PM
Last Post: shpark
  Rockpro64 Dead on arrival? quixoticgeek 1 879 03-12-2023, 06:55 PM
Last Post: quixoticgeek
  RockPro64 boot questions misterc 3 1,488 01-13-2023, 06:21 PM
Last Post: misterc
  A fix for Bluetooth audio stuttering on the RockPro64 raph 2 1,580 01-03-2023, 06:53 PM
Last Post: raph
  hello i want to drive an edp screen with my rockpro64 hannescam 0 707 10-20-2022, 01:22 PM
Last Post: hannescam

Forum Jump:


Users browsing this thread: 1 Guest(s)