Kernel OOPs triggered by big writes to ext4 FS
Hi,
I've spent more than two months(!) trying to find out what causes Linux Kernel OOPs that seem to be triggered by big writes to mounted ext4 filesystems. My quest began while trying to boot from SPI and run Armbian with rootfs on eMMC. I tried two different eMMC modules and all the Rockpro64 Linux images that I could find. I burnt a lot of midnight oil trawling the Internet for solutions, but found none...
Most of the Linuix images that I've tried work fine booting from SPI using the latest u-boot running Linux (Armbian, Debian, Ayufan) with rootfs on SD, but crash badly when either installing them to eMMC from SD or running them from eMMC and e.g. installing a big package like OMV.
Watching "htop" during installation of a 'large' package, I can see that the OOPs seem to occur when the filesystem buffers hit a high-water mark and this could be mitigated by mounting ext4 filesystems in 'sync' mode. I've done a lot of testing and now I have a stable OMV6 running under an Armbian image I compiled myself from source, and ext4 filesystems mounted in 'sync' mode: I only have occasional OOPs during heavy i/o to an ext4 filesystem mounted on an "md" raid consisting of two 1TB SATA disks connected to a 2-port Adaptec PCI-e controller.
Using 'sync' mode degrades performance, and could wear out SSD's, so it's only useful as a temporary work-around. The real problem is how to prevent the ext4 FS buffers from overflowing available memory. I suspect, but have no proof, that this is caused by the Linux 'optimistic' memory allocator oversubscribing memory and getting caught-out by the actual usage overflowing available physically memory.
I don't have a deep enough knowledge of the Linux kernel to investigate the problem, but maybe someone else reading this message does, but what I do know, is that the problem is NOT a device driver issue, because direct device access (e.g. using "dd") does not trigger an OOPs. This is not a hardware fault on my particular Rockpro64 board or eMMC module because FreeBSD installs and runs fine on SD or eMMC on the same system.
I'd be interested to read other people's experiences of using the Rockpro64 with OMV6 under Armbian.
Bye,
Tony Travis.
I've spent more than two months(!) trying to find out what causes Linux Kernel OOPs that seem to be triggered by big writes to mounted ext4 filesystems. My quest began while trying to boot from SPI and run Armbian with rootfs on eMMC. I tried two different eMMC modules and all the Rockpro64 Linux images that I could find. I burnt a lot of midnight oil trawling the Internet for solutions, but found none...
Most of the Linuix images that I've tried work fine booting from SPI using the latest u-boot running Linux (Armbian, Debian, Ayufan) with rootfs on SD, but crash badly when either installing them to eMMC from SD or running them from eMMC and e.g. installing a big package like OMV.
Watching "htop" during installation of a 'large' package, I can see that the OOPs seem to occur when the filesystem buffers hit a high-water mark and this could be mitigated by mounting ext4 filesystems in 'sync' mode. I've done a lot of testing and now I have a stable OMV6 running under an Armbian image I compiled myself from source, and ext4 filesystems mounted in 'sync' mode: I only have occasional OOPs during heavy i/o to an ext4 filesystem mounted on an "md" raid consisting of two 1TB SATA disks connected to a 2-port Adaptec PCI-e controller.
Using 'sync' mode degrades performance, and could wear out SSD's, so it's only useful as a temporary work-around. The real problem is how to prevent the ext4 FS buffers from overflowing available memory. I suspect, but have no proof, that this is caused by the Linux 'optimistic' memory allocator oversubscribing memory and getting caught-out by the actual usage overflowing available physically memory.
I don't have a deep enough knowledge of the Linux kernel to investigate the problem, but maybe someone else reading this message does, but what I do know, is that the problem is NOT a device driver issue, because direct device access (e.g. using "dd") does not trigger an OOPs. This is not a hardware fault on my particular Rockpro64 board or eMMC module because FreeBSD installs and runs fine on SD or eMMC on the same system.
I'd be interested to read other people's experiences of using the Rockpro64 with OMV6 under Armbian.
Bye,
Tony Travis.
