eMMC frequently not found by Linux kernel (especially on low battery)

[v-man]

In order to be able to boot from SD card I used "the U-Boot update script from the mrfixit2001 Debian" mentioned in the Pinebook Pro wiki, so I guess I have that U-Boot loader on my eMMC now. The strange thing now is: in at least a dozen attempts I could always boot the kernel, but often it does not find the eMMC anymore, and drops me to a shell:

Code:

[    2.127663] debugfs: File ':interrupt-controller@fee00000' in directory 'doma
ins' already present!
[    2.xxxxxx] debugfs: File ':interrupt-controller@fee00000' in directory 'doma
ins' already present!
[    2.xxxxxx] debugfs: File ':interrupt-controller@fee00000' in directory 'doma
ins' already present!
[    2.xxxxxx] debugfs: File ':interrupt-controller@fee00000' in directory 'doma
ins' already present!
[    2.xxxxxx] debugfs: File ':interrupt-controller@fee00000' in directory 'doma
ins' already present!
[    2.127790] debugfs: File ':interrupt-controller@fee00000' in directory 'doma
ins' already present!
[    2.720138] rockchip-pcie f8000000.pcie: PCIe link training gen1 timeout!
[    3.023522] OF: graph: no port node found in /i2c@ff3d0000/fusb30x@22
[    3.185664] vcc_sdio: unsupportable voltage range: 3300000-3000000uV
[    3.317248] mmc2: mmc_select_hs200 failed, error -110
[    3.317730] mmc2: error -110 whilst initialising MMC card
[    3.4xxxxx] mmc2: mmc_select_hs200 failed, error -110
[    3.4xxxxx] mmc2: error -110 whilst initialising MMC card
[    3.5xxxxx] mmc2: mmc_select_hs200 failed, error -110
[    3.5xxxxx] mmc2: error -110 whilst initialising MMC card
[    3.7xxxxx] mmc2: mmc_select_hs200 failed, error -110
[    3.7xxxxx] mmc2: error -110 whilst initialising MMC card
[    4.196145] rockchip-dp ff970000.edp: no DP phy configured

BusyBox v1.30.1 (Debian 1:1.30.1-4) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) ...

I've seen different patterns, but to the same effect:

Code:

[      7.519011] cdn-dp fec00000.dp: Direct firmware load for rockchip/dptx.bin failed with error -2
Begin: running / scripts/local-block ... done
Begin: running / scripts/local-block ... done
Begin: running / scripts/local-block ... done
[    11.59xxxx] cdn-dp fec00000.dp: Direct firmware load for rockchip/dptx.bin failed with error -2
Begin: running / scripts/local-block ... done
Begin: running / scripts/local-block ... done
Begin: running / scripts/local-block ... done
...
Gave up waiting for root file system device.  Common problems:
- Boot args (cat /proc/cmdline)
  - Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT!  PARTLABEL=emmc-RootFS does not exist.  Dropping to a shell!


BusyBox v1.30.1 (Debian 1:1.30.1-4) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) ...

Looking for others with the same problem, and for possible solutions, I found this post in the Manjaro Linux Forum:
manjaro won't boot, eMMC, pinebookpro (Post #56)

I can confirm that my PBP has the same behaviour. I had the impression that the PBP boot succeeds more often when attached to the power supply.

Indeed after connecting my Pinebook Pro to the power cord I could boot from eMMC again, after several failed attempts (and having opened the case twice, to check the eMMC being attached properly).

Some speculation: The battery was at only about 30% - maybe it is a power issue?

Obviously it is not the Debian kernel, that is responsible. At least once I booted Manjaro (with its 5.6 kernel) from SD card and could see that /dev/mmcblk2 was not there, only /dev/mmcblk1 (the sd card).

Once, when on battery, the system even "lost" the eMMS it was running from. See the attached photograph for the kernel messages from that crash:
  emmc_loss_pbp.jpg (Size: 399.74 KB / Downloads: 593)

Is anyone else here having similar issues? Any suggestions as to what I could do (apart from keeping the power adapter with me, or a backup system on sd card)? Maybe getting an NVMe and running a system from that would help?

[Arwen]

Hmmm, I thought it was just me. I made a mistake with my eMMC, so I booted Manjaro off SD card. Except while booted to Manjaro on SD card, my eMMC was not found. Could not even get the normal eMMC commands to re-enable the device to work some of the time.

Now I have noticed that booting Manjaro off eMMC was not reliable, and tracked it to U-Boot. Based on another user's forum post, I used the latest default Debian U-Boot files on my eMMC. That has made booting Manjaro off eMMC quite reliable. Perhaps I need to do the same for my Manjaro SD card.

It almost looks like a race issue.

Maybe the eMMC takes longer to stabilize than the SD card?
And maybe the U-Boot files don't wait long enough after initialization?

[v-man]

Now I have noticed that booting Manjaro off eMMC was not reliable, and tracked it to U-Boot. Based on another user's forum post, I used the latest default Debian U-Boot files on my eMMC. That has made booting Manjaro off eMMC quite reliable. Perhaps I need to do the same for my Manjaro SD card.

Thanks for the suggestion! A couple of days ago I did:

Code:

cd /usr/lib/u-boot/pinebook-pro-rk3399/
dd if=idbloader.img of=/dev/mmcblk2 seek=64 conv=notrunc
dd if=u-boot.itb of=/dev/mmcblk2 seek=16384 conv=notrunc

Since then I booted the Debian from eMMC two or three times, with decreasing battery levels, and without any problem. So yes, it does look like a U-Boot issue.

But now again I cannot boot from SD card anymore, which is annoying for at least two reasons:

1. For some reason I think video performance was somewhat better in Manjaro, so being able to use it occasionally would be nice.
   
2. Should something go really wrong with my Debian system I cannot boot a rescue system.
   

Overall I'm quite disappointed with the state of U-Boot support for this device. I have two other ARM systems (RaspberryPi 2 and Cubox-i4) where U-Boot produces display output right from the start. This would probably even allow for a boot menu (which I only tested on yet another ARM, a CB5-311). I'm looking forward to improvements!

[Arwen]

Now I have noticed that booting Manjaro off eMMC was not reliable, and tracked it to U-Boot. Based on another user's forum post, I used the latest default Debian U-Boot files on my eMMC. That has made booting Manjaro off eMMC quite reliable. Perhaps I need to do the same for my Manjaro SD card.

Thanks for the suggestion! A couple of days ago I did:

Code:

cd /usr/lib/u-boot/pinebook-pro-rk3399/
dd if=idbloader.img of=/dev/mmcblk2 seek=64 conv=notrunc
dd if=u-boot.itb of=/dev/mmcblk2 seek=16384 conv=notrunc

Since then I booted the Debian from eMMC two or three times, with decreasing battery levels, and without any problem. So yes, it does look like a U-Boot issue.

But now again I cannot boot from SD card anymore, which is annoying for at least two reasons:

1. For some reason I think video performance was somewhat better in Manjaro, so being able to use it occasionally would be nice.
   
2. Should something go really wrong with my Debian system I cannot boot a rescue system.
   

Overall I'm quite disappointed with the state of U-Boot support for this device. I have two other ARM systems (RaspberryPi 2 and Cubox-i4) where U-Boot produces display output right from the start. This would probably even allow for a boot menu (which I only tested on yet another ARM, a CB5-311). I'm looking forward to improvements!

For the problem on booting a SD card, you might see about a later version of U-Boot. Early ones did not have additional checks for other devices, before booting their own device.

And yes, I too have joined the "Disappointed in Pinebook Pro's U-Boot state" club. (We hold on-line meetings every week or so, to complain :-).

Ideally I'd like to see U-Boot support a simple device menu. So if it loaded off eMMC, and by default it wanted to boot off any installed SD card, I could force select it back to eMMC. Or visa-versa.

Then, I'd like Grub menu for kernel selection and other details related to Linux.

And last of course, I'd like U-Boot in SPI. But, I am in no hurry for that phase, as recovery is more difficult.

[v-man]

It happened again:

Code:

ALERT!  PARTLABEL=emmc-RootFS does not exist.  Dropping to a shell!

That was with U-Boot from Debian sid (unstable), U-Boot 2020.04. Now I updated Das U-Boot on /dev/mmcblk2 to the package in experimental, 2020.07~rc4 - still it does not boot Manjaro from the SD card. It remains to see if it makes any difference in booting from the eMMC when the battery is lower...

[Arwen]

@v-man when they meant restore the U-Boot files, the rest of the "/boot" file system was left alone. Looks like you changed the configuration file. This is what my Manjaro has for it's configuration file;

Code:

/boot/extlinux/extlinux.conf
LABEL Manjaro ARM
KERNEL ../Image
FDT ../dtbs/rockchip/rk3399-pinebook-pro.dtb
APPEND initrd=../initramfs-linux.img console=tty1 root=LABEL=ROOT_MNJRO2 rw rootwait video=eDP-1:1920x1080@60 video=HDMI-A-1:1920x1080@60 bootsplash.bootfile=bootsplash-themes/manjaro/bootsplash

Note that if you changed the kernel, and other files in "/boot" to ones in Debian, that can prevent Manjaro from booting.

What I did was replaced the U-Boot binaries on disk with "dd" using the default Debian ones, not the files in "/boot".

[v-man]

Note that if you changed the kernel, and other files in "/boot" to ones in Debian, that can prevent Manjaro from booting.

What I did was replaced the U-Boot binaries on disk with "dd" using the default Debian ones, not the files in "/boot".

Sure, that's also what I did. I had not changed Manjaro on the SD card at all. As I understood it from the docs, U-Boot is loaded from the internal eMMC, if available, and only from the SD card if the eMMC is disabled (or probably also if it does not contain U-Boot). But even U-Boot from the eMMC, if configured appropriately, can boot a kernel from the SD card. mrfixit2001's U-Boot, the one I mentioned in this thread's first post, is doing exactly that. Apparently Debian's U-Boot is not configured to do so.

For some reason I had used an older version of mrfixit2001's branch initially: my link goes to 1.1. Now going back to his U-Boot to have the flexibility of sometimes booting Manjaro from SD card, sometimes Debian from eMMC, I opted for version 2.0. I modified his mrfixit_update.sh:

Code:

diff --git a/pinebook/filesystem/mrfixit_update.sh b/pinebook/filesystem/mrfixit_update.sh
--- a/pinebook/filesystem/mrfixit_update.sh
+++ b/pinebook/filesystem/mrfixit_update.sh
@@ -1,7 +1,7 @@
#!/bin/bash

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-myver="0.0"
+myver="1.99"
if [ -f /usr/share/myver ]; then
        myver=$(cat /usr/share/myver)
fi
@@ -65,10 +65,10 @@ fi

if [[ $myver < 2.0 ]]; then
        # Leave the current kernel version's modules folder in place so that it can be used as the backup kernel
-      echo "Updating Kernel Modules to 4.4.213..."
-      KER="$(uname -r)"
-      find /lib/modules -mindepth 1 ! -regex '^/lib/modules/'$KER'\(/.*\)?' -delete
-      rm /lib/modules/4.4.213 -r
+#      echo "Updating Kernel Modules to 4.4.213..."
+#      KER="$(uname -r)"
+#      find /lib/modules -mindepth 1 ! -regex '^/lib/modules/'$KER'\(/.*\)?' -delete
+#      rm /lib/modules/4.4.213 -r
        mv $DIR/4.4.213 /lib/modules

        echo "Updating U-Boot..."

Actually it seems like DigiKam in Manjaro is now supporting external MySQL databases! So my main reason for having switched to Debian is gone.
And again Manjaro provides much better video playback support. Even full-hd is now working on Youtube (with Chromium).

Still, this u-boot situation is odd. Maybe I should install the version from Manjaro? The version as shown by pacman is: uboot-pinebookpro 2020.07rc4-1.1

I don't even know if I'll continue to have these eMMC not found issues, with the different u-boot I'm using again. I'll report here as soon as I know more.

[Arwen]

@v-man If you have a U-Boot that works for both eMMC & SD cards, I'd leave it alone for the moment. Wait a month or 2.

My opinion is that U-Boot for the Pinebook needs lots of misc. work. Some trivial, (like settable boot order), some more
complicated, like displaying a menu on the LCD for boot device selection.

[v-man]

@v-man If you have a U-Boot that works for both eMMC & SD cards, I'd leave it alone for the moment. Wait a month or 2.

Well, I did wait, although I did not really have a working U-Boot... But now I think I got one - even though the issue with the eMMC frequently not being available persists. Currently I'm just running a system from SD card, a Manjaro that I installed with the manjaro-arm-installer running on an AMD64 notebook (on Debian). I had to remove one check from the script, and cancel a stuck process with ctrl+c at some point, but it gave me a bootable system.

It seems strange to me, how many images available on the PineBook Wiki, i.e. the Manjaro ARM with KDE (Plasma) desktop, do not actually boot my PineBook.

I did quite some experiments, going as far as trying graphics for U-Boot on the SPI, and using a bootable Nix Linux (and the effect of early graphics and having a boot menu is great!)... but nothing helped with the eMMC issue. At some point I even had a system that would run from SD card, but for some reason had super slow access to the card only (around 1 MB/sec, if I remember correctly). Now "hdparm -t" shows 58 MB/sec for the SD card, and 179 MB/sec for the eMMC. So really I'd still like to use the latter. I'm also considering complaining about a broken product (maybe a replacement part could fix the problem?), or buying the NVMe adapter...

[Arwen]

@v-man You might modify your "/boot" on the SD card to use the root file system on the eMMC. This might not be too bad as a work around.

From what you said about the eMMC not being available, it's almost as if it's too slow during initialization such that the boot code thinks it does not exist. So, skips over the eMMC on many boots. If you are using the stock 64GB eMMC you might try buying another or even the 128GB eMMC.

If you get a NVMe adapter, make sure you get a lower power NVMe M.2 PCIe drive. Many people have had trouble because they bought high performance ones, that required lots of power. More than the CPU even.