Earlier this month, Debian 12 bookworm updated the Linux kernel to 6.1.0-23-arm64 (Debian 6.1.99-1 from 2024-07-15). Rock64 boards upgraded to this Stable release kernel hang during boot after logging failure of mmc_select_hs200 to switch the eMMC bus width to 8:
Linux version 6.1.0-23-arm64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP Debian 6.1.99-1 (2024-07-15)
...
mmc1: switch to bus width 8 failed
mmc1: mmc_select_hs200 failed, error -110
mmc1: error -110 whilst initialising MMC card
The problem is not present on the previous version of Linux from Debian, version 6.1.0-22-arm64. There was probably a change to the device tree that is incompatible with the Rock64. I tested this on a Rock64 V2, but not on a V3.
There's also a report from July 22 on Arch Linux ARM: kernel 6.9.8-2: rock64 bootfail due to the emmc init error https://archlinuxarm.org/forum/viewtopic...65&t=16981 that appears to be the same issue. I suspect that these both contained the same breaking change based on the same Rock64 boot failures and log entries after a kernel update the same week.
Boot log with the failure:
U-Boot 2024.04+kaslr.env.emmc-00001-g146d1d4515 (Jun 03 2024 - 09:18:00 -0700)
Model: Pine64 Rock64
DRAM: 4 GiB
PMIC: RK8050 (on=0x40, off=0x01)
Core: 236 devices, 28 uclasses, devicetree: separate
MMC: mmc@ff500000: 1, mmc@ff520000: 0
Loading Environment from MMC... OK
In: serial@ff130000
Out: serial@ff130000
Err: serial@ff130000
Model: Pine64 Rock64
Net: eth0: ethernet@ff540000
Hit any key to stop autoboot: 0
** Booting bootflow 'mmc@ff520000.bootdev.part_4' with script
Target is mmc 0:4 /vmlinuz-6.1.0-23-arm64
Loading Linux kernel to 0x02080000 from /vmlinuz-6.1.0-23-arm64
32688064 bytes read in 731 ms (42.6 MiB/s)
Loading Device Tree to 0x01f00000 from /dtbs/6.1.0-23-arm64/rockchip/rk3328-rock64.dtb
36095 bytes read in 10 ms (3.4 MiB/s)
Loading Initial RAM disk to 0x06000000 from /initrd.img-6.1.0-23-arm64
30464587 bytes read in 691 ms (42 MiB/s)
Generate a KASLR seed to chosen/kaslr-seed from RNG
Working FDT set to 1f00000
Booting Linux 6.1.0-23-arm64 on /dtbs/6.1.0-23-arm64/rockchip/rk3328-rock64.dtb via /initrd.img-6.1.0-23-arm64
booti 0x02080000 0x06000000:1d0da4b 0x01f00000
Moving Image from 0x2080000 to 0x2200000, end=41d0000
## Flattened Device Tree blob at 01f00000
Booting using the fdt blob at 0x1f00000
Working FDT set to 1f00000
Loading Ramdisk to fa1fc000, end fbf09a4b ... OK
Loading Device Tree to 00000000fa1f0000, end 00000000fa1fbfff ... OK
Working FDT set to fa1f0000
Starting kernel ...
[ 0.000000] Linux version 6.1.0-23-arm64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP Debian 6.1.99-1 (2024-07-15)
[ 0.000000] Kernel command line: console=ttyS2,1500000n8 console=tty0 plymouth.ignore-serial-consoles loglevel=6 consoleblank=0 systemd.show_status=1 ipv6.disable=1 rootwait dw_wdt.nowayout=1
[ 0.174232] Registered cp15_barrier emulation handler
[ 0.174276] Registered setend emulation handler
[ 0.185940] audit: type=2000 audit(0.184:1): state=initialized audit_enabled=0 res=1
[ 0.249816] VFS: Disk quotas dquot_6.6.0
[ 0.366262] Initialise system trusted keyrings
[ 0.366374] Key type blacklist registered
[ 0.380770] integrity: Platform Keyring initialized
[ 0.380864] integrity: Machine keyring initialized
[ 0.380887] Key type asymmetric registered
[ 0.380905] Asymmetric key parser 'x509' registered
[ 3.301068] Loading compiled-in X.509 certificates
[ 3.454864] Loaded X.509 cert 'Debian Secure Boot CA: 6ccece7e4c6c0d1f6149f3dd27dfcc5cbb419ea1'
[ 3.455789] Loaded X.509 cert 'Debian Secure Boot Signer 2022 - linux: 14011249c2675ea8e5148542202005810584b25f'
[ 3.460030] Key type .fscrypt registered
[ 3.460414] Key type fscrypt-provisioning registered
[ 3.502399] Key type encrypted registered
[ 4.390877] rockchip-pm-domain ff100000.syscon:power-controller: failed to get ack on domain 'hevc', val=0x88220
[ 6.341543] dwc2 ff580000.usb: supply vusb_d not found, using dummy regulator
[ 6.343547] dwc2 ff580000.usb: supply vusb_a not found, using dummy regulator
[ 6.397893] rockchip-pinctrl pinctrl: pin gpio0-2 already requested by vcc-host-5v-regulator;cannot claim for vcc-host1-5v-regulator
[ 6.398999] rockchip-pinctrl pinctrl: pin-2 (vcc-host1-5v-regulator) status -22
[ 6.399677] rockchip-pinctrl pinctrl: could not request pin 2 (gpio0-2) from group usb20-host-drv on device rockchip-pinctrl
[ 6.400711] reg-fixed-voltage vcc-host1-5v-regulator: Error applying setting, reverse things back
[ 6.861027] mmc1: switch to bus width 8 failed
[ 6.869008] mmc1: mmc_select_hs200 failed, error -110
[ 6.869513] mmc1: error -110 whilst initialising MMC card
[ 7.137041] mmc1: switch to bus width 8 failed
[ 7.147010] mmc1: mmc_select_hs200 failed, error -110
[ 7.147495] mmc1: error -110 whilst initialising MMC card
[ 7.433097] mmc1: switch to bus width 8 failed
[ 7.446840] mmc1: mmc_select_hs200 failed, error -110
[ 7.447320] mmc1: error -110 whilst initialising MMC card
[ 7.737104] mmc1: switch to bus width 8 failed
[ 7.764080] mmc1: mmc_select_hs200 failed, error -110
[ 7.764559] mmc1: error -110 whilst initialising MMC card
There was a fairly long set of changes in Debian's changelog on 2024-07-14 and the next day's Debian 6.1.99-1 release is the hanging one: https://metadata.ftp-master.debian.org/c..._changelog
One of those changes is the culprit.
[arm64] dts: rockchip: Fix SD NAND and eMMC init on rk3308-rock-pi-s
might be related? I seem to remember something about eliminating duplicated dts blocks from many Rockchip boards into a seperate file and referencing that in each board's config. Maybe this change propagated to the Rock64?
Hmm. The Rock64 .dtb are the same in 6.1.20-22 and 6.1.20-23:
$ sha256sum /boot/dtbs/6.1.0-22-arm64/rockchip/rk3328-rock64.dtb /boot/dtbs/6.1.0-23-arm64/rockchip/rk3328-rock64.dtb
a8afb8e300296c0377039a011f17b93d7b7c2ac91f29effda1340f0f45d1f171 /boot/dtbs/6.1.0-22-arm64/rockchip/rk3328-rock64.dtb
a8afb8e300296c0377039a011f17b93d7b7c2ac91f29effda1340f0f45d1f171 /boot/dtbs/6.1.0-23-arm64/rockchip/rk3328-rock64.dtb
So maybe not.
Other hints in that changelog?
Linux version 6.1.0-23-arm64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP Debian 6.1.99-1 (2024-07-15)
...
mmc1: switch to bus width 8 failed
mmc1: mmc_select_hs200 failed, error -110
mmc1: error -110 whilst initialising MMC card
The problem is not present on the previous version of Linux from Debian, version 6.1.0-22-arm64. There was probably a change to the device tree that is incompatible with the Rock64. I tested this on a Rock64 V2, but not on a V3.
There's also a report from July 22 on Arch Linux ARM: kernel 6.9.8-2: rock64 bootfail due to the emmc init error https://archlinuxarm.org/forum/viewtopic...65&t=16981 that appears to be the same issue. I suspect that these both contained the same breaking change based on the same Rock64 boot failures and log entries after a kernel update the same week.
Boot log with the failure:
U-Boot 2024.04+kaslr.env.emmc-00001-g146d1d4515 (Jun 03 2024 - 09:18:00 -0700)
Model: Pine64 Rock64
DRAM: 4 GiB
PMIC: RK8050 (on=0x40, off=0x01)
Core: 236 devices, 28 uclasses, devicetree: separate
MMC: mmc@ff500000: 1, mmc@ff520000: 0
Loading Environment from MMC... OK
In: serial@ff130000
Out: serial@ff130000
Err: serial@ff130000
Model: Pine64 Rock64
Net: eth0: ethernet@ff540000
Hit any key to stop autoboot: 0
** Booting bootflow 'mmc@ff520000.bootdev.part_4' with script
Target is mmc 0:4 /vmlinuz-6.1.0-23-arm64
Loading Linux kernel to 0x02080000 from /vmlinuz-6.1.0-23-arm64
32688064 bytes read in 731 ms (42.6 MiB/s)
Loading Device Tree to 0x01f00000 from /dtbs/6.1.0-23-arm64/rockchip/rk3328-rock64.dtb
36095 bytes read in 10 ms (3.4 MiB/s)
Loading Initial RAM disk to 0x06000000 from /initrd.img-6.1.0-23-arm64
30464587 bytes read in 691 ms (42 MiB/s)
Generate a KASLR seed to chosen/kaslr-seed from RNG
Working FDT set to 1f00000
Booting Linux 6.1.0-23-arm64 on /dtbs/6.1.0-23-arm64/rockchip/rk3328-rock64.dtb via /initrd.img-6.1.0-23-arm64
booti 0x02080000 0x06000000:1d0da4b 0x01f00000
Moving Image from 0x2080000 to 0x2200000, end=41d0000
## Flattened Device Tree blob at 01f00000
Booting using the fdt blob at 0x1f00000
Working FDT set to 1f00000
Loading Ramdisk to fa1fc000, end fbf09a4b ... OK
Loading Device Tree to 00000000fa1f0000, end 00000000fa1fbfff ... OK
Working FDT set to fa1f0000
Starting kernel ...
[ 0.000000] Linux version 6.1.0-23-arm64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP Debian 6.1.99-1 (2024-07-15)
[ 0.000000] Kernel command line: console=ttyS2,1500000n8 console=tty0 plymouth.ignore-serial-consoles loglevel=6 consoleblank=0 systemd.show_status=1 ipv6.disable=1 rootwait dw_wdt.nowayout=1
[ 0.174232] Registered cp15_barrier emulation handler
[ 0.174276] Registered setend emulation handler
[ 0.185940] audit: type=2000 audit(0.184:1): state=initialized audit_enabled=0 res=1
[ 0.249816] VFS: Disk quotas dquot_6.6.0
[ 0.366262] Initialise system trusted keyrings
[ 0.366374] Key type blacklist registered
[ 0.380770] integrity: Platform Keyring initialized
[ 0.380864] integrity: Machine keyring initialized
[ 0.380887] Key type asymmetric registered
[ 0.380905] Asymmetric key parser 'x509' registered
[ 3.301068] Loading compiled-in X.509 certificates
[ 3.454864] Loaded X.509 cert 'Debian Secure Boot CA: 6ccece7e4c6c0d1f6149f3dd27dfcc5cbb419ea1'
[ 3.455789] Loaded X.509 cert 'Debian Secure Boot Signer 2022 - linux: 14011249c2675ea8e5148542202005810584b25f'
[ 3.460030] Key type .fscrypt registered
[ 3.460414] Key type fscrypt-provisioning registered
[ 3.502399] Key type encrypted registered
[ 4.390877] rockchip-pm-domain ff100000.syscon:power-controller: failed to get ack on domain 'hevc', val=0x88220
[ 6.341543] dwc2 ff580000.usb: supply vusb_d not found, using dummy regulator
[ 6.343547] dwc2 ff580000.usb: supply vusb_a not found, using dummy regulator
[ 6.397893] rockchip-pinctrl pinctrl: pin gpio0-2 already requested by vcc-host-5v-regulator;cannot claim for vcc-host1-5v-regulator
[ 6.398999] rockchip-pinctrl pinctrl: pin-2 (vcc-host1-5v-regulator) status -22
[ 6.399677] rockchip-pinctrl pinctrl: could not request pin 2 (gpio0-2) from group usb20-host-drv on device rockchip-pinctrl
[ 6.400711] reg-fixed-voltage vcc-host1-5v-regulator: Error applying setting, reverse things back
[ 6.861027] mmc1: switch to bus width 8 failed
[ 6.869008] mmc1: mmc_select_hs200 failed, error -110
[ 6.869513] mmc1: error -110 whilst initialising MMC card
[ 7.137041] mmc1: switch to bus width 8 failed
[ 7.147010] mmc1: mmc_select_hs200 failed, error -110
[ 7.147495] mmc1: error -110 whilst initialising MMC card
[ 7.433097] mmc1: switch to bus width 8 failed
[ 7.446840] mmc1: mmc_select_hs200 failed, error -110
[ 7.447320] mmc1: error -110 whilst initialising MMC card
[ 7.737104] mmc1: switch to bus width 8 failed
[ 7.764080] mmc1: mmc_select_hs200 failed, error -110
[ 7.764559] mmc1: error -110 whilst initialising MMC card
There was a fairly long set of changes in Debian's changelog on 2024-07-14 and the next day's Debian 6.1.99-1 release is the hanging one: https://metadata.ftp-master.debian.org/c..._changelog
One of those changes is the culprit.
[arm64] dts: rockchip: Fix SD NAND and eMMC init on rk3308-rock-pi-s
might be related? I seem to remember something about eliminating duplicated dts blocks from many Rockchip boards into a seperate file and referencing that in each board's config. Maybe this change propagated to the Rock64?
Hmm. The Rock64 .dtb are the same in 6.1.20-22 and 6.1.20-23:
$ sha256sum /boot/dtbs/6.1.0-22-arm64/rockchip/rk3328-rock64.dtb /boot/dtbs/6.1.0-23-arm64/rockchip/rk3328-rock64.dtb
a8afb8e300296c0377039a011f17b93d7b7c2ac91f29effda1340f0f45d1f171 /boot/dtbs/6.1.0-22-arm64/rockchip/rk3328-rock64.dtb
a8afb8e300296c0377039a011f17b93d7b7c2ac91f29effda1340f0f45d1f171 /boot/dtbs/6.1.0-23-arm64/rockchip/rk3328-rock64.dtb
So maybe not.
Other hints in that changelog?