Can not get board to boot up reliably

[dkryder]

so i was wondering while looking at your work if someone wanted to implement your work if it is a direct replacement of stock directories with your directories and during the unpacking some files were flagged as being the same with one set of those files iwith lower case filenames and the other with upper case filenames. i chose to not replace the uppercase with the lowercase. was that the correct choice? also the screen resolution listed in your Uenv.txt, is that the only resolution available or can i provide another like 1080p60? thanks

Sorry i just found out, that some files (in the netfilter kernel modules directory) are duplicated in the tar.gz You should be safe if you
select the latest version. I will make a new .tar.gz tomorrow. I did not disable any device driver option, so you should be able to select other resolutions. Although i only tested with the one in the config (i think 720p50).

Hopefully other people who are also having bootup problems can test with this kernel (settings) and let us know if it fixed the bootup problem.
This is still a workaround, but once confirmed working helps in finding a real solution.

If i have some time left i will also do some power measurements with a scope to see how the 5Volt line is behavng at startup. Just to rule out power issues that people are reporting. I don't know if anybody has done this already?

well, once i have a good understanding about how to install your work on/over the current install i'm very interested in testing particularly one board i rarely use now since there were too numerous boot failures which i could never get a finger on the reason/reasons for the failures. now, if that board starts behaving like it should with your work then i will be convinced beyond a doubt that you isolated and fixed whatever it was. so take you time to do what you feel is needed. one thing i ask of you to include is a basic set of instructions as i'm not sure if i swap out your work with what is currently there or if i just add your work to what is there and replace present files with your files if the filenames match. thanks.

[pkfpeters]

so i was wondering while looking at your work if someone wanted to implement your work if it is a direct replacement of stock directories with your directories and during the unpacking some files were flagged as being the same with one set of those files iwith lower case filenames and the other with upper case filenames. i chose to not replace the uppercase with the lowercase. was that the correct choice? also the screen resolution listed in your Uenv.txt, is that the only resolution available or can i provide another like 1080p60? thanks

Sorry i just found out, that some files (in the netfilter kernel modules directory) are duplicated in the tar.gz You should be safe if you
select the latest version. I will make a new .tar.gz tomorrow. I did not disable any device driver option, so you should be able to select other resolutions. Although i only tested with the one in the config (i think 720p50).

Hopefully other people who are also having bootup problems can test with this kernel (settings) and let us know if it fixed the bootup problem.
This is still a workaround, but once confirmed working helps in finding a real solution.

If i have some time left i will also do some power measurements with a scope to see how the 5Volt line is behavng at startup. Just to rule out power issues that people are reporting. I don't know if anybody has done this already?

well, once i have a good understanding about how to install your work on/over the current install i'm very interested in testing particularly one board i rarely use now since there were too numerous boot failures which i could never get a finger on the reason/reasons for the failures. now, if that board starts behaving like it should with your work then i will be convinced beyond a doubt that you isolated and fixed whatever it was. so take you time to do what you feel is needed. one thing i ask of you to include is a basic set of instructions as i'm not sure if i swap out your work with what is currently there or if i just add your work to what is there and replace present files with your files if the filenames match. thanks.

I double checked the .tar.gz but i think there is nothing wrong with it. You have probably tried to extract it under Windows?
There are actually several kernel modules with the same names but only some characters are in captial. The actual kernel
modules are different from each other so both of them should be installed. I dont have this problem when extracting the
.tar.gz under Linux.

Please follow the steps below to get my changes added to your MicroSD card.

-Boot your Pine64 with the Debian version provided on the forum.
-Make sure it's up to date by running: apt-get update  followed by apt-get upgrade
-Make sure you are using the latest kernel and uboot from longsleep by using the following commands: pine64_update_kernel.sh  followed by pine64_update_uboot.sh
-Copy the .tar.gz file you downloaded from my stack space to a USB stick and put this usb stick in the Pine64.
-Check with the 'dmesg' command the enumeration for your usb stick partition containing the .tar.gz file.
-Create a directory for mounting the usb stick for example: 'mkdir /mnt/usbstick'
-Mount the usbstick (using enumerated device id found in dmesg or cat /proc/partitions) by using someting like: mount /dev/sda1 /mnt/usbstick  <<==replace sda1 with correct device id.
-Make a backup of your current /boot directory, using the following commands:
    cd /
    tar -cpzf /mnt/usbstick/backup-of-bootdir.tar.gz /boot
-Now make sure you are in root path (top level), using command: cd /
-And now let tar extract my .tar.gz over current installation (only /boot files will be overwritten) by using the command:
    tar -xpzf /mnt/usbstick/pine64-working-kernel-longsleep-with-ppe-workaround.tar.gz
-Unmount usbstick with the command: umount /mnt/usbstick
-Make sure all items are really in sync, bit paranoia but it never hurts by using command: sync

-Reboot and you should now be using the new kernel and modules.

[pkfpeters]

update: I just did some quick measurements on the euler power pins while powering the Pine with the standard micro usb power supply delivered through Pine online shop.
I did measurements with a scope and did not find any power issues. Keep in mind it was only on the euler power pins so depending on the board layout some ic's could still have power problems due to wrong capacitance or inductance on traces. I saw a max. voltage ripple of 200mV. I also did a sysbench to max. out the cpu usage and saw a max. voltage drop of 300mV. Voltage measured on the euler 5Volt pins at idle : 5.2Volt average.

If needed i can do extra tests. Like voltage ramp up patterns to see if the unit has problems with it. For now i am happy with it.

[dkryder]

@pkfpeters, yes i was on a windows machine when i unzipped. just a force of habit i guess as i usually unzip images and write to sd card via a windows laptop. thanks for these instructions. i'll wire up my older pine64 board and get started with the installation & testing later this morning and see what i find out. i still have a copy of debian from back in june of last year i want to try out since that is the o/s install that gave me the most bootup issues with the original pine64. thanks.

[Kaarme42]

Hi

My Pine type is Pine A64-DB-2G-Rev B 2016-03-21 and boot problem is allmost similar than Leo's
"Sometimes boots up or fails due to 'soft lockup' CPU#0 stuck for 22 seconds
at a point when calling cfg80211."

small pause and after that continue until
  717.319035] rt_rq[3]:
[  717.328809]   .rt_nr_running                 : 0
[  717.336718]   .rt_throttled                  : 0
[  717.344600]   .rt_time                       : 0.021607
[  717.353173]   .rt_runtime                    : 950.000000
[  717.361934]
[  717.361934] runnable tasks:
[  717.361934]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[  717.361934] ----------------------------------------------------------------------------------------------------------
[  717.401782]
and then small pause. After that bootin starts again and result is tha same lines as upper. (power supply is official Pine international )

Tested operatim systems: Wifi board is installed

Ubuntu Base Image [20161218-1] by longsleep
Ubuntu Minimal Image
Ubuntu Mate [20161215] built by PINE64
Debian Jessie with Mate DE [3.10.102 BSP 2] by Lenny Raposo

**********
WORKS!!!
Android 7.1 Community Image [v0.3.5-r61]
Tryed severl times reboot and poweroff
This is interesting. I think no total HW problem.
**********

Then I found this thread and tryed pkfpeters's links
https://pkfpeters.stackstorage.com/s/mL1bQL0Dlj8Xw04


ubuntu@pine64:~$ uname -a
Linux pine64 3.10.105+ #1 SMP Thu Jun 8 10:34:39 CEST 2017 aarch64 aarch64 aarch64 GNU/Linux

Ten times sudo reboot and result is OK. This is new record :-)
After that I installed "official" WIFI 802.11BGN/BLUETOOTH 4.0 MODULE:
Booting several times  OK, but ifconfig do not show anything about wifi.

Main reason for this post:
- Please keep going this excelent hacking
- Please take a minute and think what is so different with Android 7.1. Official version works
- Maybe someday official Ubuntu and Debian booting is OK.

BR: Kaarme42

[pfeerick]

I did not have the time to really do some in depth research, so I first
made a workaround by letting the kernel panic on soft lockups and auto reboot. At least this will result
in an auto retry without having to toggle power. 

I think the fix is triggered by compiling the kernel without preemptive support; see also my
kernel config.

I think that is somewhat similar to what the Armbian guys have done... I'm pretty sure I've seen their kernel panic and reboot, so at least their setup actually tries to get past the problem. It is weird though that removing preemptive support would b0rk thinks though, isn't it? Isn't that exactly what preemptive support was supposed to prevent (i.e. zombie drivers)? I'll have to get another plain boot image up and running, and then I'll give it a try... as I can reproduce this 'gremlin' fairly regularly when I don't want to! :-D

"Sometimes boots up or fails due to 'soft lockup' CPU#0 stuck for 22 seconds
at a point when calling cfg80211."

This is interesting. I think no total HW problem.

I think we're pretty much put a finger on the fact that it is the WiFi driver module that is screwing up... (the fact that it b0rks when doing something related to 80211 is a pretty big hint that it's network related) basically because it is such a unreliable POS of a driver. Problem is it's still hanging around when the WiFi module isn't installed. But I suspect pkfpeters's may still be onto something re: it having some connection to register initialisation sequence... since a few reboots can sometimes 'fix' it, or often simply a full power cycle and then a reboot.

[pkfpeters]

Hi

My Pine type is Pine A64-DB-2G-Rev B 2016-03-21 and boot problem is allmost similar than Leo's
"Sometimes boots up or fails due to 'soft lockup' CPU#0 stuck for 22 seconds
at a point when calling cfg80211."

small pause and after that continue until
  717.319035] rt_rq[3]:
[  717.328809]   .rt_nr_running                 : 0
[  717.336718]   .rt_throttled                  : 0
[  717.344600]   .rt_time                       : 0.021607
[  717.353173]   .rt_runtime                    : 950.000000
[  717.361934]
[  717.361934] runnable tasks:
[  717.361934]             task   PID         tree-key  switches  prio     exec-runtime         sum-exec        sum-sleep
[  717.361934] ----------------------------------------------------------------------------------------------------------
[  717.401782]
and then small pause. After that bootin starts again and result is tha same lines as upper. (power supply is official Pine international )

Tested operatim systems: Wifi board is installed

Ubuntu Base Image [20161218-1] by longsleep
Ubuntu Minimal Image
Ubuntu Mate [20161215] built by PINE64
Debian Jessie with Mate DE [3.10.102 BSP 2] by Lenny Raposo

**********
WORKS!!!
Android 7.1 Community Image [v0.3.5-r61]
Tryed severl times reboot and poweroff
This is interesting. I think no total HW problem.
**********

Then I found this thread and tryed pkfpeters's links
https://pkfpeters.stackstorage.com/s/mL1bQL0Dlj8Xw04


ubuntu@pine64:~$ uname -a
Linux pine64 3.10.105+ #1 SMP Thu Jun 8 10:34:39 CEST 2017 aarch64 aarch64 aarch64 GNU/Linux

Ten times sudo reboot and result is OK. This is new record :-)
After that I installed "official" WIFI 802.11BGN/BLUETOOTH 4.0 MODULE:
Booting several times  OK, but ifconfig do not show anything about wifi.

Main reason for this post:
- Please keep going this excelent hacking
- Please take a minute and think what is so different with Android 7.1. Official version works
- Maybe someday official Ubuntu and Debian booting is OK.

BR: Kaarme42

Hi Kaarme42,

Thank you for testing. Do you mean by "official" wifi 802.11bgn module, the add on wifi/bt board you can
order in the online Pine shop? I also tested with this board and it should work fine. Have you tried ifconfig -a
or iwconfig ? You should see a wlan0, wlan1 or something similar. You could also try and look if a device
name exists in /sys/class/net by using the command: ls -al /sys/class/net
Once you have a valid device name for your wifi module you can attach your unit to an access point with
the help of wpa_supplicant -c /etc/wpa_supplicant.conf -B -i wlan0   <<-- replace wlan0 with your device name.
If your access point is providing IP addresses (DHCP based) use the following command to aquire a valid IP address
    dhclient -i wlan0   <<-- replace wlan0 with your device name.

You boot log indicates abnormal boot times indeed. I am currently up and running from power on to login prompt in 28seconds. That is
including wifi + lan + usb serial converters + ssh + postgresql database engine. In Debian's case try to make sure udev and/or network interface settings are mostly static in order to prevent network service startup delays.

I hope you know get your wifi module also working. Good luck.

[Kaarme42]

Hi pkfpeters,

" Do you mean by "official" wifi 802.11bgn module, the add on wifi/bt board you can order in the online Pine shop?"
> Yes

"I also tested with this board and it should work fine. Have you tried ifconfig -a
or iwconfig ? You should see a wlan0, wlan1 or something similar"
> I can see booth wlan0 and wlan1

"If your access point is providing IP addresses (DHCP based) use the following command to aquire a valid IP address  dhclient -i wlan0   <<-- replace wlan0 with your device name."
> Access point do not provide IP address

"You boot log indicates abnormal boot times indeed. I am currently up and running from power on to login prompt in 28seconds."
> I cut  and  paste Boot log  to answer, but it was third (if I remember correct) booting loop round.

***
Today's experiments

I make fresh start and flash Samsung 32GB EVO (Etcher Pine version)
pine64-image-debianmate-310102bsp-2.img
Booting was OK, but no network (Eth0 or Wlan)

debian@pine64:~$ sudo ifconfig -a

dummy0    Link encap:Ethernet  HWaddr de:56:8d:bf:dc:41
         BROADCAST NOARP  MTU:1500  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr 36:c9:e3:f1:b8:05
         inet6 addr: fe80::34c9:e3ff:fef1:b805/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:0 (0.0 B)  TX bytes:2339 (2.2 KiB)
         Interrupt:114

lo        Link encap:Local Loopback
         inet addr:127.0.0.1  Mask:255.0.0.0
         inet6 addr: ::1/128 Scope:Host
         UP LOOPBACK RUNNING  MTU:65536  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

sit0      Link encap:IPv6-in-IPv4
         NOARP  MTU:1480  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

wlan0     Link encap:Ethernet  HWaddr 34:c3:d2:b3:65:86
         UP BROADCAST MULTICAST  MTU:1500  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

wlan1     Link encap:Ethernet  HWaddr 36:c3:d2:b3:65:86
         BROADCAST MULTICAST  MTU:1500  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Several times reboot but no IPaddress (with or without wifi card)

So I can not install yours patc, because first instruction is
sudo apt-get update
sudo apt-get upgrade
and after that yours patch.
chiken -egg problem :-)

Next time when trying Debian I install first yours patch and after that update and upgrade.
But before that playing with Ubuntu.
Result is:

Ubuntu Base Image build 20161218-1 7.7GB Pine64 V2.0.0-beta2 (Etcher Pine version)
Boot problem two times (stay in booting loop. Only solutions is power off -> on).
Third boot trying -> OK

ubuntu@pine64:~$ ifconfig -a

dummy0    Link encap:Ethernet  HWaddr 8a:d5:e7:d9:ed:9e
        BROADCAST NOARP  MTU:1500  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr ce:8c:40:1a:43:c1
         inet addr:192.168.0.28  Bcast:192.168.0.255  Mask:255.255.255.0
         inet6 addr: fe80::cc8c:40ff:fe1a:43c1/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:16 errors:0 dropped:0 overruns:0 frame:0
         TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:2326 (2.3 KB)  TX bytes:1684 (1.6 KB)
         Interrupt:114

lo        Link encap:Local Loopback
         inet addr:127.0.0.1  Mask:255.0.0.0
         inet6 addr: ::1/128 Scope:Host
         UP LOOPBACK RUNNING  MTU:65536  Metric:1
         RX packets:80 errors:0 dropped:0 overruns:0 frame:0
         TX packets:80 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:5920 (5.9 KB)  TX bytes:5920 (5.9 KB)

sit0      Link encap:IPv6-in-IPv4
         NOARP  MTU:1480  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

wlan0     Link encap:Ethernet  HWaddr 34:c3:d2:b3:65:86
         BROADCAST MULTICAST  MTU:1500  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

wlan1     Link encap:Ethernet  HWaddr 36:c3:d2:b3:65:86
         BROADCAST MULTICAST  MTU:1500  Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

So no Wlan but Eth0 working and booting is reliable after patch

*****

And again I tried Android 7.1 version because SD card was ready and it is not bad idea check out wifi module health.

This time HDMI is in use. Debian and Ubuntu tryed Putty serial connection.
Booting, eth0 and wlan was OK
Android working OK. Reliable booting, wlan and lan.

Is this booting problem relatetd to Pine64 PCB version?
My version is A64-DB-2G-Rev B 2016-03-21

BR: Kaarme42

[pkfpeters]

Is this booting problem relatetd to Pine64 PCB version?
My version is A64-DB-2G-Rev B 2016-03-21

BR: Kaarme42

I did/do my experiments on a Pine A64-DB-2G-Rev B 2016-02-25
I don't expect this to be a problem. I got the same boot up problems if i don't
use my alternate kernel version.

During your Ubuntu test you did got eth0 (LAN) working without my new kernel?
I saw your ifconfig output saying eth0 got IP address 192.168.0.28. Don't know
if this was automaticly assigned. You still should be able to get the lan and wifi
working without my kernel. It's just a matter of setting the network configuraton.
You can always force an IP address to the eth0 device, by using something like:
   ifconfig eth0 192.168.0.28/24        <<== Please make sure this IP address is not in use

I still have to do some research as why preemptive would give problems during
bootup. I am guessing it has something to do with interrupt configuration of
some device driver which conflicts with a different driver all depending on the
speed of initialisation during startup of these devices. Best changes, it is a conflict
between the cfg80211 driver and HDMI.

I also did a second power usage test with a LAB power supply. When connected
to the EULER power pins and setting the LAB power supply at 5.00 VDC, I
monitored power usage. Power usage is very low. Starting at around 0,1A going
up and down to about 0,45A. Only when doing heavy CPU work as with a
sysbench benchmark i saw current spikes of about 0,9A. Keep in mind that i
only work console based, so i have no X desktop or heavy graphical usage.
Maybe in the future i will do another power test with full desktop usage.

[Kaarme42]

pkfpeters wrote

"During your Ubuntu test you did got eth0 (LAN) working without my new kernel?"
> Yes, but it takes 2 endless boot loop (power off -> ON) before 3 booting was succes.
> Then yours patch and after that booting is OK


"I saw your ifconfig output saying eth0 got IP address 192.168.0.28. Don't know
if this was automaticly assigned."
> Yes it was  automaticly assigned

"You can always force an IP address to the eth0 device, by using something like:
  ifconfig eth0 192.168.0.28/24        <<== Please make sure this IP address is not in use"
> Sorry, but I did not test it.

I did not make any wlan testing because it is not so important to me. I need Nextcloud. I am weary but  it does not work (I am just a beginner)

I changed to Armbian -> Booting and eth0 are realible.

Nextcloud is different story and not belongs to this thread.

Sorry about late answer.

Br: Kaarme42