eMMC can't be recognize after 6 months idle

[seanlee]

I had Ayufan Ubuntu 0.79 built burnt in a 64GB eMMC for RockPro64. It was running fine until it was powered down and removed and set aside for more than 6 months. 

When recently attempt to boot from eMMC, my RockPro64 failed to recognize the eMMC. Then I tried resetting the eMMC using Ayufan's script, still nothing. Then powered down, move eMMC to windows, it failed to recognize. No media was found.

What could have happened to this eMMC? Nothing was done to it, and it was set aside, untouched. How to determine the condition of this eMMC? Is there any tool I can use to test or revive it?

I read AmeriDroid was able to resolve similar issue and sent back the eMMC to their user with image burnt. The user had the same issue as mine. Is there possibly any specific tools they use to do so?

[abdel]

Hello

Are you sure the issue is your eMMC and not your rockpro64 ?
I have a new rockpro64 bought few days ago and the eMMC module never worked on it. In my case my eMMC was working fine on my pinebook pro and is still working fine when used with USB dongle.

with serial console I could see in uboot this error:

U-Boot 2019.10-1 (Oct 13 2019 - 15:52:24 +0000) Manjaro ARM

Model: Pine64 RockPro64
DRAM: 3.9 GiB
MMC: dwmmc@fe320000: 1, sdhci@fe330000: 0
Loading Environment from MMC... *** Warning - bad CRC, using default environment

In: serial@ff1a0000
Out: serial@ff1a0000
Err: serial@ff1a0000
Model: Pine64 RockPro64
rockchip_dnl_key_pressed: adc_channel_single_shot fail!
Net: eth0: ethernet@fe300000

it will load the kernel anyway, but will fail later when mounting eMMC root partition (to run /init):


16.956461] Run /init as init process
[ 16.972014] mmc1: mmc_select_hs400es failed, error -110
[ 16.972506] mmc1: error -110 whilst initialising MMC card
:: running early hook [udev]
Starting version 243.51-1-arch
:: running hook [udev]
:: Triggering uevents...
[ 17.095321] mmc1: mmc_select_hs400es failed, error -110
[ 17.095866] mmc1: error -110 whilst initialising MMC card
[ 17.219695] usb 4-1: New USB device found, idVendor=0dc6, idProduct=3401, bcdDevice= 1.03
[ 17.220420] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 17.221052] usb 4-1: Product: SCISSORS Keyboard
[ 17.221451] usb 4-1: Manufacturer:
[ 17.230717] input: SCISSORS Keyboard as /devices/platform/fe3e0000.usb/usb4/4-1/4-1:1.0/0003:0DC6:3401.0001/input/input0
[ 17.262923] mmc1: mmc_select_hs400es failed, error -110
[ 17.263409] mmc1: error -110 whilst initialising MMC card
[ 17.296559] hid-generic 0003:0DC6:3401.0001: input,hidraw0: USB HID v1.00 Keyboard [ SCISSORS Keyboard] on usb-fe3e0000.usb-1/input0
Waiting 10 seconds for device /dev/disk/by-partuuid/5b716fc0-01 ...
[ 22.955960] phy phy-ff770000.syscon:usb2-phy@e450.2: charger = USB_DCP_CHARGER
ERROR: device 'PARTUUID=5b716fc0-01' not found. Skipping fsck.
:: mounting 'PARTUUID=5b716fc0-01' on real root
mount: /new_root: can't find PARTUUID=5b716fc0-01.
You are now being dropped into an emergency shell.
sh: can't access tty; job control turned off
[rootfs ]# [ 107.365828] random: crng init done

So to me it seems more an issue on the board.

Note: if you dont have serial console, then you can insert both sd card and eMMC module, you will be able to boot via SD card and you can see in dmesg the same error I got when the mmc block driver is trying to use eMMC module.
in bash you will also see no /dev/mmc
# ls /dev/mmc*
/dev/mmcblk0 /dev/mmcblk0p1

On linux driver the issue happens in mmc.c:
err = mmc_switch_status(card);
if (err)
goto out_err;

Inside mmc_switch_status the error is inside this function

int __mmc_send_status(struct mmc_card *card, u32 *status, unsigned int retries)
{
int err;
struct mmc_command cmd = {};

cmd.opcode = MMC_SEND_STATUS;
if (!mmc_host_is_spi(card->host))
cmd.arg = card->rca << 16;
cmd.flags = MMC_RSP_SPI_R2 | MMC_RSP_R1 | MMC_CMD_AC;

err = mmc_wait_for_cmd(card->host, &cmd, retries);
if (err)
return err;

/* NOTE: callers are required to understand the difference
* between "native" and SPI format status words!
*/
if (status)
*status = cmd.resp[0];

return 0;
}

We basically send a command to eMMC module bu get no response back (that's why we got error -110 which is a timeout error).

So that's why I would be very interested to know if you have the same error.
If you can check it will be great.

Maybe I will buy the cheapest eMMC module from pine store, and I will try that to be 100% sure that the issue is not on my eMMC

[abdel]

nevermind.
My issue was majaro. release 19.08 and 19.10 are bad and not working on eMMC maybe because of missing features on kernel config.
19.12 is ok. (they are others issues like oops but I will investigate)

But in your case I would recommend to try to get serial console to get more information

[dukla2000]

My issue was majaro.

Sorry offtopic I know, but can you get your ethernet port to work on Manjaro? I have tried all 3 19.12 downloads (kde,xfce & minimal) and none of them connect to my router. Which is nuts as I have no problems with Ayufan etc, the router is bog standard. But means I cant do anything meaningful in terms of playing/swapping to Manjaro!

[abdel]

My issue was majaro.

Sorry offtopic I know, but can you get your ethernet port to work on Manjaro? I have tried all 3 19.12 downloads (kde,xfce & minimal) and none of them connect to my router. Which is nuts as I have no problems with Ayufan etc, the router is bog standard. But means I cant do anything meaningful in terms of playing/swapping to Manjaro!

The ethernet port is working fine for me.

What do you have if you run "ethtool eth0" ? Is the last line "link detected: yes" ?

do you see you network interface with command "ip link" ?

is your driver loaded ? ("lsmod |grep mac") 
do you have error on dmesg ? (dmesg |grep ma)

[dukla2000]

The ethernet port is working fine for me.

What do you have if you run "ethtool eth0" ? Is the last line "link detected: yes" ?

do you see you network interface with command "ip link" ?

is your driver loaded ? ("lsmod |grep mac") 
do you have error on dmesg ? (dmesg |grep ma)

Yeah the device/driver seems OK, to me it is some kind of routing issue. But my networking skills are very rusty!

Fresh burn of 19.12 XFCE version onto an SDcard, first boot to select username etc, then reboot.No connection, so I create a new connection in NetworkManager that is essentially defaults except IPv6 disabled. {ethtool is not installed in basic package so that fails :-) }

Code:

$ inxi -FGx
System:    Host: RPro64 Kernel: 5.4.2-2-MANJARO-ARM aarch64 bits: 64 compiler: gcc v: 9.1.0 Desktop: Xfce 4.14.1
          Distro: Manjaro ARM
Machine:   Type: ARM Device System: Pine64 RockPro64 details: N/A serial: b3996cb1d2a8cb9d
CPU:       Topology: 6-Core (2-Die) model: N/A variant-1: cortex-a53 variant-2: cortex-a72 bits: 64 type: MCP MCM arch: ARMv8
          rev: 4
          features: Use -f option to see features bogomips: 0
          Speed: 408 MHz min/max: 408/1416:1800 MHz Core speeds (MHz): 1: 408 2: 408 3: 408 4: 408 5: 408 6: 408
Graphics:  Device-1: display-subsystem driver: rockchip_drm v: N/A bus ID: N/A
          Device-2: rk3399-dw-hdmi driver: dwhdmi_rockchip v: N/A bus ID: N/A
          Device-3: rk3399-mali driver: panfrost v: kernel bus ID: N/A
          Display: x11 server: X.org 1.20.6 driver: fbturbo resolution: <xdpyinfo missing>
          Message: Unable to show advanced data. Required tool glxinfo missing.
Audio:     Device-1: rk3399-dw-hdmi driver: dwhdmi_rockchip bus ID: N/A
          Device-2: simple-audio-card driver: asoc_simple_card bus ID: N/A
          Sound Server: ALSA v: k5.4.2-2-MANJARO-ARM
Network:   Device-1: rk3399-gmac driver: rk_gmac_dwmac v: N/A port: N/A bus ID: N/A
          IF-ID-1: eth0 state: up speed: 100 Mbps duplex: full mac: 4e:d9:eb:5f:b2:85
Drives:    Local Storage: total: 491.86 GiB used: 3.93 GiB (0.8%)
          ID-1: /dev/mmcblk0 model: NCard size: 14.92 GiB
          ID-2: /dev/nvme0n1 vendor: A-Data model: SX8200PNP size: 476.94 GiB
Partition: ID-1: / size: 14.59 GiB used: 3.93 GiB (26.9%) fs: ext4 dev: /dev/mmcblk0p1
Sensors:   Message: No sensors data was found. Is sensors configured?
Info:      Processes: 192 Uptime: 4m Memory: 1.93 GiB used: 559.1 MiB (28.2%) Init: systemd Compilers: gcc: N/A Shell: bash
          v: 5.0.11 inxi: 3.0.37
[chris@RPro64 ~]$
[chris@RPro64 ~]$
[chris@RPro64 ~]$ dmesg | grep ethernet
[   33.564119] rk_gmac-dwmac fe300000.ethernet: IRQ eth_wake_irq not found
[   33.564718] rk_gmac-dwmac fe300000.ethernet: IRQ eth_lpi not found
[   33.565448] rk_gmac-dwmac fe300000.ethernet: PTP uses main clock
[   33.566102] rk_gmac-dwmac fe300000.ethernet: clock input or output? (input).
[   33.566723] rk_gmac-dwmac fe300000.ethernet: TX delay(0x28).
[   33.567230] rk_gmac-dwmac fe300000.ethernet: RX delay(0x11).
[   33.567742] rk_gmac-dwmac fe300000.ethernet: integrated PHY? (no).
[   33.568323] rk_gmac-dwmac fe300000.ethernet: cannot get clock clk_mac_speed
[   33.568933] rk_gmac-dwmac fe300000.ethernet: clock input from PHY
[   33.574970] rk_gmac-dwmac fe300000.ethernet: init for RGMII
[   33.575941] rk_gmac-dwmac fe300000.ethernet: User ID: 0x10, Synopsys ID: 0x35
[   33.576586] rk_gmac-dwmac fe300000.ethernet: DWMAC1000
[   33.577052] rk_gmac-dwmac fe300000.ethernet: DMA HW capability register supported
[   33.577708] rk_gmac-dwmac fe300000.ethernet: RX Checksum Offload Engine supported
[   33.578365] rk_gmac-dwmac fe300000.ethernet: COE Type 2
[   33.578835] rk_gmac-dwmac fe300000.ethernet: TX Checksum insertion supported
[   33.579456] rk_gmac-dwmac fe300000.ethernet: Wake-Up On Lan supported
[   33.580055] rk_gmac-dwmac fe300000.ethernet: Normal descriptors
[   33.580577] rk_gmac-dwmac fe300000.ethernet: Ring mode enabled
[   33.581088] rk_gmac-dwmac fe300000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[   42.974613] rk_gmac-dwmac fe300000.ethernet eth0: PHY [stmmac-0:00] driver [RTL8211F Gigabit Ethernet]
[   42.982114] rk_gmac-dwmac fe300000.ethernet eth0: No Safety Features support found
[   42.982795] rk_gmac-dwmac fe300000.ethernet eth0: PTP not supported by HW
[   42.983397] rk_gmac-dwmac fe300000.ethernet eth0: configuring for phy/rgmii link mode
[   46.112565] rk_gmac-dwmac fe300000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[chris@RPro64 ~]$
[chris@RPro64 ~]$
[chris@RPro64 ~]$ dmesg | grep mac
[   33.564119] rk_gmac-dwmac fe300000.ethernet: IRQ eth_wake_irq not found
[   33.564718] rk_gmac-dwmac fe300000.ethernet: IRQ eth_lpi not found
[   33.565448] rk_gmac-dwmac fe300000.ethernet: PTP uses main clock
[   33.566102] rk_gmac-dwmac fe300000.ethernet: clock input or output? (input).
[   33.566723] rk_gmac-dwmac fe300000.ethernet: TX delay(0x28).
[   33.567230] rk_gmac-dwmac fe300000.ethernet: RX delay(0x11).
[   33.567742] rk_gmac-dwmac fe300000.ethernet: integrated PHY? (no).
[   33.568323] rk_gmac-dwmac fe300000.ethernet: cannot get clock clk_mac_speed
[   33.568933] rk_gmac-dwmac fe300000.ethernet: clock input from PHY
[   33.574970] rk_gmac-dwmac fe300000.ethernet: init for RGMII
[   33.575941] rk_gmac-dwmac fe300000.ethernet: User ID: 0x10, Synopsys ID: 0x35
[   33.576586] rk_gmac-dwmac fe300000.ethernet: DWMAC1000
[   33.577052] rk_gmac-dwmac fe300000.ethernet: DMA HW capability register supported
[   33.577708] rk_gmac-dwmac fe300000.ethernet: RX Checksum Offload Engine supported
[   33.578365] rk_gmac-dwmac fe300000.ethernet: COE Type 2
[   33.578835] rk_gmac-dwmac fe300000.ethernet: TX Checksum insertion supported
[   33.579456] rk_gmac-dwmac fe300000.ethernet: Wake-Up On Lan supported
[   33.580055] rk_gmac-dwmac fe300000.ethernet: Normal descriptors
[   33.580577] rk_gmac-dwmac fe300000.ethernet: Ring mode enabled
[   33.581088] rk_gmac-dwmac fe300000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[   33.705891] libphy: stmmac: probed
[   33.706211] RTL8211F Gigabit Ethernet stmmac-0:00: attached PHY driver [RTL8211F Gigabit Ethernet] (mii_bus:phy_addr=stmmac-0:00, irq=POLL)
[   33.707303] RTL8211F Gigabit Ethernet stmmac-0:01: attached PHY driver [RTL8211F Gigabit Ethernet] (mii_bus:phy_addr=stmmac-0:01, irq=POLL)
[   42.974613] rk_gmac-dwmac fe300000.ethernet eth0: PHY [stmmac-0:00] driver [RTL8211F Gigabit Ethernet]
[   42.982114] rk_gmac-dwmac fe300000.ethernet eth0: No Safety Features support found
[   42.982795] rk_gmac-dwmac fe300000.ethernet eth0: PTP not supported by HW
[   42.983397] rk_gmac-dwmac fe300000.ethernet eth0: configuring for phy/rgmii link mode
[   46.112565] rk_gmac-dwmac fe300000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[chris@RPro64 ~]$
[chris@RPro64 ~]$
[chris@RPro64 ~]$ ethtool eth0
bash: ethtool: command not found
[chris@RPro64 ~]$
[chris@RPro64 ~]$
[chris@RPro64 ~]$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
   link/ether 4e:d9:eb:5f:b2:85 brd ff:ff:ff:ff:ff:ff
[chris@RPro64 ~]$
[chris@RPro64 ~]$
[chris@RPro64 ~]$ ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
   inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
   link/ether 4e:d9:eb:5f:b2:85 brd ff:ff:ff:ff:ff:ff
[chris@RPro64 ~]$
[chris@RPro64 ~]$
[chris@RPro64 ~]$ networkctl
IDX LINK TYPE     OPERATIONAL SETUP    
 1 lo   loopback carrier     unmanaged
 2 eth0 ether    carrier     unmanaged

2 links listed.

[abdel]

Try to setup a static IP
# ip link set up dev eth0
# ip addr add 192.168.0.20/24 dev eth0 (depending on your network)

and then try to ping an another ip/computer on your network
is this working ?

[dukla2000]

Try to setup a static IP
# ip link set up dev eth0
# ip addr add 192.168.0.20/24 dev eth0  (depending on your network)

and then try to ping an another ip/computer on your network
is this working ?

To try simplify I reloaded a minimal image as it uses systemd-networkd (and not NetworkManager as per the xfce & kde images).
It works somewhat better, the log is below. 192.168.1.1 is my router. What is horrific is the packet loss. the second bit is from ip address.

Code:

PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=0.289 ms
From 192.168.1.20 icmp_seq=9 Destination Host Unreachable
From 192.168.1.20 icmp_seq=10 Destination Host Unreachable
From 192.168.1.20 icmp_seq=11 Destination Host Unreachable

--- 192.168.1.1 ping statistics ---
19 packets transmitted, 1 received, +3 errors, 94.7368% packet loss, time 18744ms
rtt min/avg/max/mdev = 0.289/0.289/0.289/0.000 ms, pipe 3


1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
   inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
   link/ether 4e:d9:eb:5f:b2:85 brd ff:ff:ff:ff:ff:ff
   inet 192.168.1.20/24 scope global eth0
      valid_lft forever preferred_lft forever
   inet6 fe80::5666:1550:a509:ca34/64 scope link
      valid_lft forever preferred_lft forever

Just rebooting into my Ubuntu setup I get

Code:

$ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.652 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.667 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.657 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=0.672 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=0.679 ms
64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=0.674 ms
^C
--- 192.168.1.1 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5007ms
rtt min/avg/max/mdev = 0.652/0.666/0.679/0.009 ms


$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
   inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
   link/ether 4e:d9:eb:5f:b2:85 brd ff:ff:ff:ff:ff:ff
   inet 192.168.1.154/24 brd 192.168.1.255 scope global dynamic eth0
      valid_lft 43123sec preferred_lft 43123sec
   inet6 fd12:dca2:57f1::923/128 scope global noprefixroute
      valid_lft forever preferred_lft forever
   inet6 fd12:dca2:57f1:0:4cd9:ebff:fe5f:b285/64 scope global mngtmpaddr noprefixroute
      valid_lft forever preferred_lft forever
   inet6 fe80::4cd9:ebff:fe5f:b285/64 scope link
      valid_lft forever preferred_lft forever


$ ip route
default via 192.168.1.1 dev eth0 proto dhcp src 192.168.1.154 metric 5
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.154
192.168.1.1 dev eth0 proto dhcp scope link src 192.168.1.154 metric 5

(ip route under Manjaro gives null response)

I truly appreciate your patience trying to help this dumbo, thank you

[abdel]

So you say the same board is working fine in ubuntu but not in manjaro ? This is weird ....
I don't know where your issue come from.
Maybe you should report the issue on manjaro web site ?

[dukla2000]

So you say the same board is working fine in ubuntu but not in manjaro ? This is weird ....

Yup - to swap between Ubuntu/Manjaro I just place/remove the jumper to decide whether the RockPro boots from eMMC or SDcard. The network is the same 45cm cable etc untouched.

I guess there is something weird between the Manjaro 5.4 kernel and my ancient TPLink/OpenWRT router, that does not manifest with the 4.4 Ayufan kernel used for Ubuntu. At least knowing you are OK with Manjaro on a RockPro64 means I need to play more - the immediate hurdle is the Manjaro releases do not include the rndis bits for a direct mobile tether. I guess I may have to dig out some WiFi bits and see if that helps.