Pinebook Pro doesn't boot most of the time unless connected via USB-serial!

[nib0]

My Pinebook Pro has evolved into a lazy AI. It sometimes boots, but most of the time it can't be bothered.

It all started yesterday, when after a couple months of non-use, I decided to re-install a fresh manjaro. Since I was trying different versions of the distro, I cloned a couple of times to the eMMC and finally settled on a minimal version which I cloned to the eMMC using dd.

After that, the machine failed to boot, with only the power led turning on, but in orange. I tried rebooting a few times and finally opened up the case, flipped the switch and connected the serial to USB cable to check for any error messages.

The machine turned on!

I tried a couple of reboots, all worked ok, so I disconnected the serial cable.
 
The machine stopped turning on!

Since then, the machine mostly turns when connected via serial but not every time. It also sometimes starts without a problem without the cable connected, but most of the time it doesn't!

I managed to capture some boot logs.

Failed boot 1:

Code:

U-Boot TPL 2020.07-2 (Aug 14 2020 - 07:33:56)
Channel 0: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
Channel 1: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
256B stride
256B stride
lpddr4_set_rate: change freq to 400000000 mhz 0, 1
lpddr4_set_rate: change freq to 800000000 mhz 1, 0
Trying to boot from BOOTROM
Returning to boot ROM...

U-Boot SPL 2020.07-2 (Aug 14 2020 - 07:33:56 +0000)
Trying to boot from MMC2
mmc_load_image_raw_sector: mmc block read error
Trying to boot from MMC2
mmc_load_image_raw_sector: mmc block read error
Trying to boot from MMC1
mmc_load_image_raw_sector: mmc block read error
SPL: failed to boot from all boot devices
### ERROR ### Please RESET the board ###

Failed boot 2:

Code:

U-Boot TPL 2020.07-2 (Aug 14 2020 - 07:33:56)
Channel 0: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
Channel 1: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
256B stride
256B stride
lpddr4_set_rate: change freq to 400000000 mhz 0, 1
lpddr4_set_rate: change freq to 800000000 mhz 1, 0
Trying to boot from BOOTROM
Returning to boot ROM...

U-Boot SPL 2020.07-2 (Aug 14 2020 - 07:33:56 +0000)
Trying to boot from MMC2

and finally a working boot:

Code:

U-Boot TPL 2020.07-2 (Aug 14 2020 - 07:33:56)
Channel 0: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
Channel 1: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=2048MB
256B stride
256B stride
lpddr4_set_rate: change freq to 400000000 mhz 0, 1
lpddr4_set_rate: change freq to 800000000 mhz 1, 0
Trying to boot from BOOTROM
Returning to boot ROM...

U-Boot SPL 2020.07-2 (Aug 14 2020 - 07:33:56 +0000)
Trying to boot from MMC2


U-Boot 2020.07-2 (Aug 14 2020 - 07:33:56 +0000) Manjaro ARM

SoC: Rockchip rk3399
Reset cause: POR
Model: Pine64 Pinebook Pro
DRAM:  3.9 GiB
PMIC:  RK808
MMC:  mmc@fe310000: 2, mmc@fe320000: 1, sdhci@fe330000: 0
In:    serial
Out:  serial
Err:  serial
Model: Pine64 Pinebook Pro
Net:  No ethernet found.
Hit any key to stop autoboot:  0
starting USB...
Bus usb@fe380000: USB EHCI 1.00
Bus usb@fe3c0000: USB EHCI 1.00
Bus dwc3: Register 2000140 NbrPorts 2
Starting the controller
USB XHCI 1.10
Bus dwc3: Register 2000140 NbrPorts 2
Starting the controller
USB XHCI 1.10
scanning bus usb@fe380000 for devices... 1 USB Device(s) found
scanning bus usb@fe3c0000 for devices... 3 USB Device(s) found
scanning bus dwc3 for devices... 1 USB Device(s) found
scanning bus dwc3 for devices... 1 USB Device(s) found
      scanning usb for storage devices... 0 Storage Device(s) found

Device 0: unknown device
Card did not respond to voltage select!
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
236 bytes read in 16 ms (13.7 KiB/s)
1: Manjaro ARM
Retrieving file: /initramfs-linux.img
8462110 bytes read in 384 ms (21 MiB/s)
Retrieving file: /Image
33174016 bytes read in 1458 ms (21.7 MiB/s)
append: initrd=/initramfs-linux.img console=tty1 console=ttyS2,1500000 root=LABEL=ROOT_MNJRO rw rootwait bootsplash.bootfile=bootsplash-themes/manjaro/bootsplash
Retrieving file: /dtbs/rockchip/rk3399-pinebook-pro.dtb
80685 bytes read in 30 ms (2.6 MiB/s)
## Flattened Device Tree blob at 01f00000
  Booting using the fdt blob at 0x1f00000
  Loading Ramdisk to f46f3000, end f4f04f1e ... OK
  Loading Device Tree to 00000000f46dc000, end 00000000f46f2b2c ... OK

Starting kernel ...

:: running early hook [udev]
Starting version 246.5-1-arch
:: running hook [udev]
:: Triggering uevents...
:: performing fsck on '/dev/mmcblk2p2'
ROOT_MNJRO: clean, 60318/3809280 files, 760427/15205211 blocks
:: mounting '/dev/mmcblk2p2' on real root
:: running cleanup hook [udev]

Welcome to Manjaro ARM!

I reinstalled u-boot and followed the instructions to dd the 2 files, didn't make a difference.
I re-seated the eMMC module, it didn't make a difference.

The strangest thing is the fact that, when connected via serial, the machine boots most of the time! I'm afraid that this would that it could be hardware issue, maybe having to do with intermittent power failure. Everything seems ok and plugged in on the board however. The charging light also turns on.

Really sorry for the very long post, especially since it's my very first. Tried to include any relevant info.

Any help will be greatly appreciated.

[wdt]

Ah, uboot.. You realize that uboot also has a dtb, obviously not correct, not finding mmc,, emmc
>Since I was trying different versions of the distro
I assume these were booting?
Experiment time!!!!
(with dd) save emmc mbr, save first 16M of emmc , blank first 16M, restore mbr
This then will be forced to use uboot on SD card, test various uboots, various distros
To make double sure, save SD mbr, delete all partitions with fdisk/cfdisk (on SD)
This then will be forced to use SD uboot and emmc distro
If seems OK, after several boots, copy 1st 16M SD ->emmc, restore emmc mbr
I prefer mrfixits uboot
Note: do not save sd mbr on sd, after a reboot or fdisk write, mbr is re-read,
you will lose access to sd partitions,, dd does NOT re-read mbr (which is in memory)

[nib0]

>Ah, uboot.. You realize that uboot also has a dtb, obviously not correct, not finding mmc,, emmc
The very strange thing is that sometimes it does find it, especially if the USB-serial cable is plugged in. Because of that I thought that maybe it could be a power issue and plugging in the cable powers the board directly bypassing the battery. It does start without the cable sometimes though.

By the way, what is the difference between the mmc and the emmc on the pinebook? I saw in the error logs that  it was trying to locate MMC1 & MMC2 which, as I understand, are different from the eMMC.

>>Since I was trying different versions of the distro
>I assume these were booting?
They were booting from the SD, yes. I think I tried a couple, the swaywm one and mostly variations of a minimal manjaro system I was building using the manjaro-arm-installer (https://gitlab.manjaro.org/manjaro-arm/a...-installer) with and without encryption.

>Experiment time!!!!
>(with dd) save emmc mbr, first 16M of emmc , blank first 16M, restore mbr
>This then will be forced to use uboot on SD card, test various uboots, various distros
>To make double sure, save SD mbr, delete all partitions with fdisk/cfdisk (on SD)
>This then will be forced to use SD uboot and emmc distro
>If seems OK, after several boots, copy 1st 16M SD ->emmc, restore emmc mbr
The SD card also doesn't boot every time now! I just tried it and it didn't, so I'm not sure this could solve this problem.

>I prefer mrfixits uboot.
Could you explain what's the difference? I'm a newbie when it comes to u-boot, its the first time I messed with it. Do you think that with these symptoms, it could be a bootloader issue?

Thanks!

[wdt]

Yes, uboot (and idbloader) is your bootloader
I think mrfixit is bsp uboot (older), manjaro is mainline
No power flows thru serial connection
emmc is just a slightly different mmc, timing and bus width
uboot enumerates in order found often emmc is MMC 0
(since bootrom search order is SPI, emmc, SD, usb for program write)
Do be aware that with no uboot (blank emmc, no SD card) computer will appear dead
Also, with bad uboot, computer will appear dead, remove "bad" card,
insert good card, still appears dead, do "extra long press",, 20+seconds(pwr button)
To blank emmc, any boot will do, so emmc boot is OK
(1st 16M is outside of any partition area, so mounting is irrelevant)
Do all 4 things in one go, don't reboot until mbr is restored
That it sometimes works means that it is only a little bit wrong(for emmc)
more wrong for SD cards which is why it never finds them
The error msg about 'mmc block read error' ,, that is the problem
--edit--
I have gone thru a problem like this, but different with TV boxes
a different rockchip cpu, 3328 and non-standard wiring to regulators
uboot dtb was ok, both SD and emmc readable (boot order for this ,, SD, emmc, usb for rkdeveloptool)
the kernel would load, then kernel dtb and lose power to SD card, not find rootfs, cause card was dead
Needed different kernel dtb, as I type running manjaro with pinned 4.4 kernel (cause I don't have 5.x dtb)
So, the dtb (device tree blob) sets various hardware parameters, memory timing, regulators, all peripherals
kernel has one, so does uboot (simpler) internally

[nib0]

Great! Thanks for the info @wdt

I'll try it out during the weekend and let you know how it went.

[slyecho]

I've also had trouble booting from eMMC.

The main OS is installed on NVMe. uboot is on the eMMC but it seems to have trouble finding it.

Currently the /boot partition is duplicated on both drives, so eventually it boots.

Haven't had time to diagnose properly. Probably a better uboot or one on the SPI so I wouldn't need the eMMC module.

[poVoq]

Please let us know if there are any improvements on this. My may batch PBP seems to be getting worse in regards to booting from emmc

[nib0]

So, I had some time to experiment on Saturday.

Following @wdt 's suggestion that it was a bootloader issue, I decided to just nuke the bootloader on the eMMC since I wasn't interested in preserving my installation. I booted from the SD card using the serial cable and zero-ed out a GB in the mmcblk2 device. I afterwards created a new GPT disklabel using cfdisk.

Code:

dd if=/dev/zero of=/dev/mmcblk2 bs=1M count=1000
cfdisk /dev/mmcblk2

That solved the booting problem. The laptop was now booting every time from the SD card.

Happy with my progress, I flipped the debugging switch, screwed back the backplate and dd'ed my minimal manjaro from the SD to the eMMC, sure that it would work.

Wrong idea! Boot failed again and after trying a few times, did the whole thing again (took off the backplate, flipped the switch, ..., erased eMMC). The laptop can once again boot from the SD card. It can't reboot however, every time I try I get an orange power led and a blank screen.

So, that's where I am now. A few questions I have:

• When there is a problem with the bootloader installed on the eMMC, why does the laptop boot with the debugging cable plugged in and doesn't without?
  
• What is the correct way to install the manjaro version of u-boot to the eMMC?
  
• Why does reboot fail even though normal booting from SD works?
  

Thanks!

[wdt]

>Why does reboot fail even though normal booting from SD works?
With that PARTICULAR distro, on SD (what is it BTW), since its using its uboot
try a different one (the 1st uboot to "get the baton" is the one, all others ignored)
To blank uboot, bs=1M count=16 is enough (do note: this also blanks mbr)
manjaro does have an alternate bsp uboot
maybe serial cable alters timing slightly??
I thought it was a bit random?
I still think mrfixits uboot is the best, updated of course

[nib0]

>>Why does reboot fail even though normal booting from SD works?
>With that PARTICULAR distro, on SD (what is it BTW), since its using its uboot
I'm using a minimal manjaro I've built a few days ago with u-boot 2020.07

>try a different one (the 1st uboot to "get the baton" is the one, all others ignored)
Do you mean I should a try different distro, or a different u-boot? If you're talking about different u-boot versions, where do I find them and how do I install them?
I thought that since manjaro is the distro that the current pinebook pros are shipped with, it would have the best support for the hardware.

>maybe serial cable alters timing slightly??
>I thought it was a bit random?
It is, you're right, but it boots most of the times using the serial cable, and it fails most of the times without.

>I still think mrfixits uboot is the best, updated of course
How can I try it out? Can I mix and match, mrfixits uboot with manjaro?

By the way, if you know of any good absolute beginners tutorial for u-boot, please pass it on.

Thanks once again!