reboot / crash

[Der Geist der Maschine]

My Pinebook Pro was running for several hours. I left for an hour or so and when I came back, it had rebooted.

The systemd journal is not in persistent memory and so I don't know what happened right before the reboot or crash. All other logfiles in /var/log/ don't show anything in the 30 min before the reboot / crash.

Anyone has seen a reboot or crash, as well?

[sinistermrcream]

I had one unexpected reboot. Screen froze for around 4-5 seconds then machine rebooted. I had Chromium up with one tab and a terminal.

[dawncold]

I got some crashes, no logs

[Nappael]

I had one reboot earlier. Next time it happens, I'll see if I can collect logs.

[Der Geist der Maschine]

My Pinebook Pro was running for several hours. I left for an hour or so and when I came back, it had rebooted.

A few minutes ago, the Pinebook Pro crashed while I was watching Youtube videos on chromium. Besides Mate itself, a few terminals have been running. All in a sudden the screen became black and the laptop rebooted in no time.

As X was running, I could not see potential messages on the console. So, it's not clear if the kernel crashed (with some last messages on the console) or if the hardware reset the system.

Anyone has a suggestion how to debug this? If not, I will set up kdump over the weekend and see what that buys me.

I was also making the systemd journal persistent, but the reboot happened so quickly that I'm not sure if something will be noted down by systemd.

BTW: the journal is a mess. A lot of warnings from the kernel and some systemd scripts. That needs to be cleaned up!

[Der Geist der Maschine]

[quote pid='51946' dateline='1573105992']
I was also making the systemd journal persistent, but the reboot happened so quickly that I'm not sure if something will be noted down by systemd.
[/quote]

I was running the system over night. No load, just Mate, a few terminals and chromium. After an uptime of 2 hours, the system crash-rebooted again. The systemd journal just shows the typical noise at the end

Nov 07 00:42:08 pinebook-pro kernel: wl_run_escan: LEGACY_SCAN sync ID: 65, bssidx: 0
Nov 07 00:42:14 pinebook-pro upowerd[1118]: energy_full (42.698600) is greater than energy_full_design (0.000000)


Can someone help me with serial consoles. Pine offers a serial console cable for the earphone jack. In lack of foresight, I did not order it. Is that the only way I can get access to the console (and see what happens right before the crash-reboot)?

[MoonDog]

Just to chime in that I also had a crash. At that moment the only thing i was doing was browsing through Chromium. The screen went black. I don't remember if it restarted on it's own or if I had to press the power button.

Would be good if we could read the relevant logs without the serial console method...

[Der Geist der Maschine]

1. If the kernel panics while x-window is running, you see a freeze.

2. Our default extlinux.conf option panic=10 makes the crashed kernel reboot after 10 seconds.

So, if you see x-window freeze for 10 seconds and then a reboot, you hit likely a kernel crash.

I wrote earlier that I've seen a reboot "in no time". That would point to a hardware problem, but maybe I just didn't realize that the screen already froze for 10 seconds? That would point to a kernel crash.


@tllim  Pine Microsystems is registered in my extended neighborhood. Would it be possible for you to borrow me a serial console cable so I see the kernel backtrace some of us are getting from time to time.

[lispy]

My Pinebook is barely usable. All browser usage crashes the OS and reboots. It started after updating to the latest updates. Maybe Winedivine had something to do with it?

[Der Geist der Maschine]

I got my 4th crash since Friday.

My Pinebook is barely usable. All browser usage crashes the OS and reboots. It started after updating to the latest updates. Maybe Winedivine had something to do with it?

I suggested already two times to provide debs for all of MrFixIT's updates. With debs we could easily and independently go back and forth new kernel versions, uboots and what else he provides.


I don't recall when I did what, but I *think* I got my first kernel crash before installing Winedivine.

Whoever created the Winedivine installer was sure that this is the greatest gift to mankind and did not provide an uninstaller. If you really want to uninstall Winedivine, perhaps uninstall chromium, delete all files left in /usr/lib/chromium-browser/ (that will be what the installer installed) and reinstall chromium, again.

---

Anyone tried setting up kdump? I did. Just that it does not work.

1. I built a kernel with kdump support.

# zgrep -i KEXEC /proc/config.gz
CONFIG_KEXEC_CORE=y
CONFIG_KEXEC=y

2. It turns out that kdump-tools do not support aarch64:

# systemctl status kdump- kdump-tools.service - Kernel crash dump capture service
Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; vendor preset: enabled)
Active: active (exited) since Thu 2019-11-07 20:34:43 PST; 1min 14s ago
Process: 3108 ExecStop=/etc/init.d/kdump-tools stop (code=exited, status=0/SUCCESS)
Process: 3138 ExecStart=/etc/init.d/kdump-tools start (code=exited, status=0/SUCCESS)
Main PID: 3138 (code=exited, status=0/SUCCESS)

Nov 07 20:34:43 pinebook-pro systemd[1]: Starting Kernel crash dump capture service...
Nov 07 20:34:43 pinebook-pro kdump-tools[3138]: Starting kdump-tools: Unsupported machine type: aarch64
Nov 07 20:34:43 pinebook-pro kdump-tools[3138]: failed to load kdump kernel ... failed!
Nov 07 20:34:43 pinebook-pro systemd[1]: Started Kernel crash dump capture service.


The unsupported message comes from kexec-tools: kexec/phys_arch.c. There is support for arm64 but not aarch64. What the heck is the difference? Just simply overwriting one string with the other is not enough as we then bail out in the syscall to the kernel.

kexec_load(0xebc08000, 3, [{buf=0xf5a0d008, bufsz=25827336, mem=0xebc08000, memsz=25829376}, {buf=0xab36c728, bufsz=102504, mem=0xf3731000, memsz=106496}, {buf=0xab36c320, bufsz=1024, mem=0xf3b00000, memsz=4096}], KEXEC_ARCH_ARM|KEXEC_ON_CRASH) = -1 EINVAL (Invalid argument)

Uh, why is flag to kexec_load KEXEC_ARCH_ARM and not KEXEC_ARCH_ARM64? I fixed that, too.

Now kdump is set up:

# kdump-config status
current state : ready to kdump
root@pinebook-pro:~# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0xebc00000
/boot/Image.bak
kdump initrd:

current state: ready to kdump

kexec command:
/sbin/kexec -p --command-line="rw root=/dev/mmcblk1p2 rootwait rootfstype=ext4 init=/sbin/init coherent_pool=1M ethaddr=5e:1d:04:11:80:3f eth1addr=5e:1d:04:11:80:5f serial=8ac3cf935afdad1 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1 zswap.enabled=1 video=HDMI-A-1:1920x1080@60 video=eDP-1:1920x1080@60 loglevel=3 vga=current earlycon=uart8250,mmio32,0xff1a0000 swiotlb=1 usbcore.autosuspend=-1 net.ifnames=0 irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service ata_piix.prefer_ms_hyperv=0" /boot/Image.bak


When I manually crash the kernel with echo c > /proc/sysrq*, then kernel just freezes as if kdump is not set up. Damn. Let's see when I have time to look into that.

---

In the meantime, when I don't use my laptop, I switch with CTL-ALT-F1 to the console. I also disabled the panic=10 auto reboot. So, if the kernel crashes while I'm on the console, I expect to see its backtrace.