[Volunteer needed]Too high DRAM clock speed MAY be causing you random crashes/freezes

[dukla2000]

You probably mentioned this somewhere but which OS is your phone currently running? My aim is to support all of the popular ones within reasonable effort.

Mobian

---

...

Not quite sure what to investigate next.
...

All I can think of is to have access to some thermal data: not sure it would help this specific issue but for sure I have withdrawal symptoms with nothing useful at /sys/devices/virtual/thermal/thermal_zone0/temp . I am surprised this wasn't properly hooked up for the A64 or similar. Or is it just a device tree issue?

[Djhg2000]

OK, I appreciate memtester is a lousy tool (user space application as opposed to system level) but it is the best one I have. Each run on a 1G file takes about 30 minutes: in 4 runs/2 hours I get say 2 fails at 624MHz. For my Brave Heart this really is a transient problem but I am happier and more stable at 552MHz.

Code:

$ sudo memtester 1G 4
[sudo] password for chris:
memtester version 4.3.0 (64-bit)
Copyright (C) 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 1024MB (1073741824 bytes)
got  1024MB (1073741824 bytes), trying mlock ...locked.
Loop 1/4:
 Stuck Address       : ok        
 Random Value        : ok
 Compare XOR         : ok
FAILURE: 0xfba98a3204294710 != 0xfba98a3200294710 at offset 0x0eb7a9f8.
 Compare SUB         : FAILURE: 0x3a6d34484de28c50 != 0xa48db679f9e28c50 at offset 0x0eb7a9f8.
 Compare MUL         :   Compare DIV         : ok
 Compare OR          : ok
 Compare AND         : ok
 Sequential Increment: ok
 Solid Bits          : ok        
 Block Sequential    : ok        
 Checkerboard        : ok        
 Bit Spread          : ok        
 Bit Flip            : ok        
 Walking Ones        : ok        
 Walking Zeroes      : ok        
 8-bit Writes        : ok
 16-bit Writes       : ok

Loop 2/4:
 Stuck Address       : ok        
 Random Value        : ok
 Compare XOR         : ok
 Compare SUB         : ok
 Compare MUL         : ok
 Compare DIV         : ok
 Compare OR          : ok
 Compare AND         : ok
 Sequential Increment: ok
 Solid Bits          : ok        
 Block Sequential    : ok        
 Checkerboard        : ok        
 Bit Spread          : ok        
 Bit Flip            : ok        
 Walking Ones        : ok        
 Walking Zeroes      : ok        
 8-bit Writes        : ok
 16-bit Writes       : ok

Loop 3/4:
 Stuck Address       : ok        
 Random Value        : ok
 Compare XOR         : ok
 Compare SUB         : ok
 Compare MUL         : ok
 Compare DIV         : ok
 Compare OR          : ok
 Compare AND         : ok
 Sequential Increment: ok
 Solid Bits          : ok        
 Block Sequential    : ok        
 Checkerboard        : ok        
 Bit Spread          : ok        
 Bit Flip            : ok        
 Walking Ones        : ok        
 Walking Zeroes      : ok        
 8-bit Writes        : ok
 16-bit Writes       : ok

Loop 4/4:
 Stuck Address       : ok        
 Random Value        : ok
 Compare XOR         : ok
 Compare SUB         : ok
FAILURE: 0xbafcee1d0048f82a != 0xbafcee1d0448f82a at offset 0x08254d80.
 Compare MUL         :   Compare DIV         : ok
 Compare OR          : ok
 Compare AND         : ok
 Sequential Increment: ok
 Solid Bits          : ok        
 Block Sequential    : ok        
 Checkerboard        : ok        
 Bit Spread          : ok        
 Bit Flip            : ok        
 Walking Ones        : ok        
 Walking Zeroes      : ok        
 8-bit Writes        : ok
 16-bit Writes       : ok

Done.

Thanks everyone for your work!

Ok, so I have enabled memtest and booted the 624 version (which made the phosh session crash within 1 or so after booting).

Unfortunately memtest didn't really help. From the .sh script and by looking at dmesg, everything seemed to be in order.

In fact only looking at the user journal (journalctl --user) did reveal anything was going wrong (phosh segfaulting).

I have attached some of the logs.


*{good,bad}mem.log are the results from running mem.sh.
*_dmesg.log are the dmesg logs
*_journalctl_user.log are journalctl --user logs
*_phosh.log are the journalctl --user logs grepped for phosh

Not quite sure what to investigate next.

Edit: Seems like I cant attach the log files. Oh well:

https://fortysixandtwo.eu/upload/576_dmesg.log
https://fortysixandtwo.eu/upload/576_goodmem.log
https://fortysixandtwo.eu/upload/576_jou...l_user.log
https://fortysixandtwo.eu/upload/576_phosh.log
https://fortysixandtwo.eu/upload/624_dmesg.log
https://fortysixandtwo.eu/upload/624_badmem.log
https://fortysixandtwo.eu/upload/624_jou...l_user.log
https://fortysixandtwo.eu/upload/624_phosh.log

Unfortunately this is an inherent limitation with using Linux memtest; it will just run the tests a set number of times and then continue to boot. Memtester has limitations at the other end where it's limited to whatever memory it can allocate from userspace (with all associated problems), however it's easy to use and can theoretically run indefinitely. Whichever one we pick there will always be compromises compared to Memtest86+.

Ideally we'd want an aarch64 port of Memtest86+ but I think that would be biting off a little more than I can chew, it's doing a lot of trickery on x86 to get bare metal memory access and that's just the start of it. Perhaps it would be easier to Frankenstein the memory test code into Linux? Any such experiments would have to wait until I have an actual device anyway, developing hardware specific code without the hardware is a massive pain I'd rather not repeat when given a choice.

However I'd like to stress the slightly off topic value of having memtest enabled in the kernel; if you have bad memory cells in a more permanent pattern (how RAM usually behaves when failing over time) then doing a few rounds of memtest on boot should be able to extend the service life quite substantially. My last 2 Android phones got retired from what was probably RAM corruption and the one I'm currently using is exhibiting some similar issues (random app crashes, refusing to go out of charging mode after the first charge, random corrupt images, etc.). For better or for worse this also means I'll have to use it as a daily driver and I'll be forced to solve issues as they appear. Who knows, maybe I'll end up patching OS bugs.

On a different and completely off topic note I'm trying to make the back cover 3D-printable. First order of business; the camera bump. One perk of being a mechanical engineering student is experience with CAD software.

You probably mentioned this somewhere but which OS is your phone currently running? My aim is to support all of the popular ones within reasonable effort.

Mobian

Thanks. I'm still on the fence about which toolkit to use, do Qt5 apps work nicely under Phosh? I think I'll be moving over to a complete Python3 rewrite for the app since that seems to be supported on every platform.

---

So I came across this http://www.denx.de/wiki/view/DULG/UBootC...on_5.9.2.7.

Apparently U-Boot also has a built in memory testing facility. Does U-Boot have framebuffer support for PinePhone yet?

I'm struggling to find any up to date information on this; according to https://linux-sunxi.org/PinePhone there isn't even a PinePhone specific config for U-Boot and suggests using this upstream config instead https://gitlab.denx.de/u-boot/u-boot/-/b..._defconfig . Is there some downstream branch for PinePhone I'm not aware of?

Edit: Never mind, this is just me being tired again. The downstream U-Boot repository is referenced in the first post and here is the config I was looking for https://gitlab.com/pine64-org/u-boot/-/b..._defconfig

[pfeerick]

nothing useful at /sys/devices/virtual/thermal/thermal_zone0/temp . I am surprised this wasn't properly hooked up for the A64 or similar. Or is it just a device tree issue?

I would think this is a DTS issue, since the pine64 / pine64 LTS and sopine, which are all A64 based also, correctly report their temperature there - e.g. https://github.com/pfeerick/pine64-scrip...temp.py#L4

[Djhg2000]

nothing useful at /sys/devices/virtual/thermal/thermal_zone0/temp . I am surprised this wasn't properly hooked up for the A64 or similar. Or is it just a device tree issue?

I would think this is a DTS issue, since the pine64 / pine64 LTS and sopine, which are all A64 based also, correctly report their temperature there - e.g. https://github.com/pfeerick/pine64-scrip...temp.py#L4

I had a quick investigation at this and at least upstream the thermal zones for A64 are defined in https://github.com/torvalds/linux/blob/m...#L170-L216 . This definition is then included in the PinePhone DTS tree here https://github.com/torvalds/linux/blob/m...ne.dtsi#L6 .

However, the PinePhone DTS tree didn't exist until the v5.7-rc1 tag (this is what happens when you try to browse the file at v5.6 https://github.com/torvalds/linux/blob/v...phone.dtsi ).

My conclusion is that you need to be running at least upstream kernel 5.7-rc1 or a downstream kernel with patched in support in order to get thermal working. I don't have an environment set up to compile working images yet so I can't give you anything ready to test, but I think this should more or less solve itself once kernel 5.7 gets released.

I think I should start making new threads instead of hijacking this one though, sorry @devrtz 

Edit: Added line references to links.

[devrtz]

...

My conclusion is that you need to be running at least upstream kernel 5.7-rc1 or a downstream kernel with patched in support in order to get thermal working. I don't have an environment set up to compile working images yet so I can't give you anything ready to test, but I think this should more or less solve itself once kernel 5.7 gets released.

I think I should start making new threads instead of hijacking this one though, sorry @devrtz 

Edit: Added line references to links.

I think we can live with waiting on 5.7 (I certainly can).
And imho your insights do have a place in this thread, so no need to worry (:

[pfeerick]

However, the PinePhone DTS tree didn't exist until the v5.7-rc1 tag ... My conclusion is that you need to be running at least upstream kernel 5.7-rc1 or a downstream kernel with patched in support in order to get thermal working.

:nods in blind agreement: Yeah, that sounds about right... can't show stuff if it the config stuff don't exist! :-O

I think I should start making new threads instead of hijacking this one though, sorry @devrtz

You could have just blamed me, you know... I'm not a ban-hammer-happy-wielding moderator! And it is pertinent info, as you need to know that you can't access that info, if you want to diagnose what's going on, but should be able to in the future. Looking at the current Manjaro ARM DTS - running 5.5.0.7, it has stuff for the CPU, similar but different, but trying to read the temperature sysfs point just gets an 'invalid argument' error even though the path is present.

[ozzeruk]

Thanks so much for discovering this. For me it has been the difference between a completely unusable PinePhone that I nearly threw into my pile of useless electronics.

... and a wonderful toy that I've been enjoying without a single problem all day.

I'm currently running on the lowest setting. No issues whatsoever. Previously Mobian and PMO were both completely black screening me after a few seconds, and SXMO was resetting every few seconds too.

I wonder how many people this is affecting?

[arturo2bodegas]

how would one go about lowering their ram speed? is this a kernal/boot thing only, or would it be possible from the OS once the phone has booted? I'm getting some instability in PureOS running from a flashed sdcard when I'm opening and working in multiple apps at once.

[devrtz]

As far as I know it's boot only.

So you could try overwriting the first few sectors of your SD card with a uboot binary from the first post.
The uboot used might be a bit dated (haven't updated it since the original post),
if there is some demand I can build a new batch of images.

[Djhg2000]

As far as I know it's boot only.

So you could try overwriting the first few sectors of your SD card with a uboot binary from the first post.
The uboot used might be a bit dated (haven't updated it since the original post),
if there is some demand I can build a new batch of images.

Now that I've started playing around with this on my PinePhone I've tested a few memory speeds, 576 helps a lot on Mobian. There seems to be some kind of incompatibility with UBPorts now because it doesn't boot after flashing that bootloader. It's a shame because UBPorts seems promising if only it would stop crashing so much.

I have yet to find a way to run proper memtest though.