BETA edition rcu errors

[8bit]

Yes, the frequency of the errors depends greatly on the CPU clock rate. Lower CPU clock seems to be better, though there's no obvious pattern. It also varies greatly for me from run to run; I added random delays to the tool to help compensate for that.

I sent a patch to Linux based on your data (thanks again!): https://lore.kernel.org/linux-sunxi/2021...and.org/T/

If you are able to recompile your kernel, please try with that patch and see if your issues are resolved. If so, feel free to respond to that email with:

Reported-and-tested-by: Your Name <your@email>

Though you don't have to by any means (if you prefer to stay pseudonymous).

Applied the patch to mobian-5.10 https://gitlab.com/mobian1/devices/sunxi64-linux

Tested the 3 devices I previously reported the errors on. All passed 30 passes (1 hour) without any errors.

I have 2 more devices to test. I am also running it overnight on two of the worst performing devices.

[8bit]

@smaeul Thanks for this patch and all the other work you've done on the A64/PinePhone.

I have tested the patched kernel on all 5 of the PinePhone mainboards that were having the rcu errors. All passed without any errors, two boards I ran timer_test for 720 passes (about 24hours) with no errors.

How long do you think it will take for this patch to go live in the main PinePhone distributions? I suspect a lot of people's problems with instability will be fixed by this. I have a 40% failure rate without this patch, 5 out of 12 PinePhone mainboards. 3 boards are really bad, 2 from eBay, I suspect those users gave up on the PinePhone because of the constant crashing.

[8bit]

Arch Linux Arm has released linux-pine64-5.12.4-1 kernel which has the patch. Running it now.

-----

Looks like anything based off https://github.com/megous/linux/releases...10519-1214 will have the fix.

[dsimic]

The results are awesome!  Improved reliability will be great for the PinePhone and all A64-based SBCs, and for our beloved SOM, SOPine.  Great work, @smaeul, and good testing, @8bit!  Thank you both very much! 

I'll submit a patch for U-Boot that brings it to the same "patch level" (i.e. mask width) as the Linux kernel, which should resolve random lockups in U-Boot exeperienced on the PinePhone by myself and a few other forum members.

[8bit]

Manjaro has patched U-Boot in testing/unstable branches. https://gitlab.manjaro.org/manjaro-arm/p...c205b71d69

It has fixed the boot hang on the 4 devices that I have that would hang 25% of the time. I have booted two of them more than 100 times without hanging.

[8bit]

Arch Linux Arm has updated U-Boot with the patch.
https://github.com/dreemurrs-embedded/Pi...452f731228

[dsimic]

It has fixed the boot hang on the 4 devices that I have that would hang 25% of the time. I have booted two of them more than 100 times without hanging.

Great results!  Boot hangs were one of the remaining few stability issues with the PinePhone.  Thank you for the testing.

[some_pinephone_user]

Bug reappears in still present. Kernel 5.14.14-2 from Manjaro ARM Phosh.

bug report here: https://gitlab.manjaro.org/manjaro-arm/p.../issues/14

[8bit]

Bug reappears in still present. Kernel 5.14.14-2 from Manjaro ARM Phosh.

bug report here: https://gitlab.manjaro.org/manjaro-arm/p.../issues/14

Ran the timer_test for 1 hour and did not have any errors. This was on 5.14.14-2-MANJARO-ARM #1 SMP Sun Oct 24 12:31:02 UTC 2021 aarch64 GNU/Linux
This phone would produce errors  every few seconds before the patch.

Also ran on another previously problematic phone running ARCH 5.14.14-1-danctnix #1 SMP PREEMPT Tue Oct 26 16:28:12 UTC 2021 aarch64 GNU/Linux without errors.

It is possible that particular phone has a SOC with a seriously buggy timer. Should run the timer_test as described here https://forum.pine64.org/showthread.php?...9#pid94389

[some_pinephone_user]

I ran the test and got fails inmediately: https://pastebin.com/xbTeksJN

The second ran get the fails inmediately too.