Effects of adjusting coherent_pool kernel parameter value

[moonwalkers]

I know that coherent_pool is a Linux kernel command line parameter, and it "sets the size of memory pool for coherent, atomic DMA allocations". I know it is specific to ARM ISA. I know it is set to 1M (1MiB) in bootloader config in Danielt's unofficial Debian installer script, and likely in most other Linux images/installers for PBP. I know in-kernel it defaults to 256KiB. I know (both from own experience and from some experienced Linux systems engineers) that various "performance tweaks" often times at best give marginal improvements or have a placebo effect, at worst can compromise system performance or stability or both. But at least for tweaks like vm.swappiness or the likes there are pretty clear consequences of changing their value either way. Even CMA (Contiguous Memory Allocator) has reasonably clear consequences of tweaking the value - too little and your device drivers for e.g. webcam may be unable to allocate enough RAM to work properly, too much is just pointless though shouldn't hurt your system (in practice it appears to be "eating" into memory usable by normal applications - e.g. when I experimented with very high CMA numbers my applications at some point started OOMing).

What I'm still puzzled by though through and through is coherent_pool - so far I have been unable to figure out what exact effect changing this value would have. It appears there is some interplay with CMA - e.g. if I set coherent_pool to 1M-8M and CMA at their default 64M things seem to work fine, if I drop CMA to 0 I easily get OOMs if coherent_pool is still set to 1M-8M, but system seems to work fine if I set coherent_pool or both to 0 (admittedly, haven't tried using webcam or any USB devices in that setup). So from that I conclude that coherent_pool actually needs CMA to work correctly, even if from https://lwn.net/Articles/479297/ it seems that CMA is a more recent development than coherent_pool. Some googling around shows that people run into some issues with certain drivers if coherent_pool is not set large enough (and especially if CMA is not enabled/large enough - one source had system complain about insufficiently big coherent_pool, and the issue was fixed by increasing CMA size instead), and some other sources claim issues if coherent_pool is set too large.

AFAIK there are some people working on kernel code around here, I am hoping that maybe some of them would know enough about kernel (as large of a knowledge area it is though) to provide some clarity on coherent_pool - what exactly is this setting for, how does it interact with CMA, why is it set by default to 256KiB if it (supposedly) frequently needs (does it?) to be tweaked to larger amounts, what is the effect of increasing or lowering its value too much?

[xyzzy]

The CMA is for drivers that want to allocate large chunks of contiguous memory. Usually this is because the driver needs to DMA a lot of data and doesn't support scatter-gather. You can see how much CMA is free by looking at CmaFree in /proc/meminfo.

coherent_pool just sets the initial size of the coherent DMA pools. They will expand if needed if the kernel is new enough to have that feature, which I think is less than a year old. But this happens in the background, so there is some early value needed before the kernel is running processes to do the expanding, which is what this controls. So as long as the device drivers needed for booting have enough it shouldn't need to be increased.

The current size of the coherent pools can been seen in /sys/kernel/debug/dma_pools/pool_size_*. Mine PBP has them only at 512kB, which was the initial value.

The coherent pools are allocated from the CMA area, so if there is no CMA, then they can't be allocated.

It seems a bit surprising that setting both CMA and coherent to zero works. I'd guess that many something ignores the value of zero and treats that as a request to allocate the default. Or that nothing needs coherent on boot and it can be expanded if/when necessary, so there are no errors. Traditionally, ARM systems did not have coherent DMA prior to ARMv8, so there are probably not a lot of users in the code that runs on a PBP.

[moonwalkers]

The CMA is for drivers that want to allocate large chunks of contiguous memory.  Usually this is because the driver needs to DMA a lot of data and doesn't support scatter-gather.  You can see how much CMA is free by looking at CmaFree in /proc/meminfo.

coherent_pool just sets the initial size of the coherent DMA pools.  They will expand if needed if the kernel is new enough to have that feature, which I think is less than a year old. But this happens in the background, so there is some early value needed before the kernel is running processes to do the expanding, which is what this controls.  So as long as the device drivers needed for booting have enough it shouldn't need to be increased.

The current size of the coherent pools can been seen in /sys/kernel/debug/dma_pools/pool_size_*.  Mine PBP has them only at 512kB, which was the initial value.

The coherent pools are allocated from the CMA area, so if there is no CMA, then they can't be allocated.

It seems a bit surprising that setting both CMA and coherent to zero works.  I'd guess that many something ignores the value of zero and treats that as a request to allocate the default.  Or that nothing needs coherent on boot and it can be expanded if/when necessary, so there are no errors.  Traditionally, ARM systems did not have coherent DMA prior to ARMv8, so there are probably not a lot of users in the code that runs on a PBP.

Thank you for the explanation!

From the linked LWN article my understanding is CMA is not strictly necessary for DMA buffer allocation, but rather is intended to help with it - to keep a contiguous memory area where those buffers can be easily allocated even when they need to be quite big. At the moment in /proc/meminfo CmaTotal is listed as 0kB, while the pools are at 128KiB, 256KiB, and 128KiB for dma, dma32, and kernel respectively, despite coherent_pool=0 in command line.

I've seen some posts out there talking about some drivers having issue when coherent_pool is too low, though in at least one case the issue was solved not by increasing coherent_pool value, but by increasing CMA, which I suppose makes sense if the pools prefer to use CMA when present and fail to expand if CMA is not large enough...

I guess my takeaway is that so long as I can boot fine and I don't see any drivers complaining in the kernel log - I can just remove those parameters from the command line and leave the kernel defaults be. Though I still wonder why coherent_pool is explicitly configured in Pinebook Pro... Could it be that without that setting NVMe SSD initialization may fail? I don't have NVMe adapter/drive to verify myself.

[xyzzy]

coherent_pool=0 just sets the initial value to zero. It will be increased as needed afterward. So there is no surprise that the current values are larger than 0.

The setting on the command line probably just old. It probably predates the kernel's ability to increase the atomic coherent dma pool. Maybe someone needed it for the old binary blob GPU driver before panfrost? Or maybe it does nothing, someone read about it helping stability and added it, even though in reality it accomplished nothing. And then it stays there because no one knows why it's there and if it's safe to remove it.