PinePhone modem keeps "disappearing"

PinePhone modem keeps "disappearing" - Printable Version

+- PINE64 (https://forum.pine64.org)
+-- Forum: PinePhone (https://forum.pine64.org/forumdisplay.php?fid=120)
+--- Forum: PinePhone Hardware (https://forum.pine64.org/forumdisplay.php?fid=122)
+--- Thread: PinePhone modem keeps "disappearing" (/showthread.php?tid=17833)

Pages: 1 2

PinePhone modem keeps "disappearing" - Kevin Kofler - 01-17-2023

[UPDATE: TL;DR: The resolution is that the hardware is faulty (in fact, not just the modem, but the entire USB subsystem except charging), and that I just bought a new PinePhone which works.]

Since late December (so for the last 3 weeks or so), I have been struggling with a strange issue on my PinePhone (Beta Edition). The modem keeps going down for some reason, making it just "disappear" from the PinePhone's point of view. eg25-manager tries to restart it, and sometimes it succeeds, but often the restart also fails, or eg25-manager gets stuck waiting for a long time and does not even try a reset even though the modem device is not visible to ModemManager or anything else (I cannot even see the device node). Sometimes, the modem does not even come up on boot, and even rebooting several times does not necessarily fix it.

It did happen in the past (before December) that the modem had to be restarted, but that was around once a day, and the auto-restart (in eg25-manager and ModemManager) had been just working for weeks now. (And I believe that was a known issue, which is also why the auto-restart had been implemented.) But now, the modem often goes down every 5 minutes, and the auto-restart does not reliably work anymore either. (Sometimes it does, often it does not.)

The first thing I would have suspected would be the latest software updates. (I run Manjaro Plasma Mobile, the one that was preinstalled on the Beta Edition, but continuously upgraded from the stable repository using pamac-gtk. FYI, Discover is just broken for system updates on Manjaro, so I gave up on using that one long ago, but that is not the issue here.) But if that were the case (i.e., if it were a software regression), the issue would have hit more users, would it not? Yet, I have not seen any other report of this happening. Or does anybody else see the same thing?

The behavior looks a lot like a wrong contact on the USB connection, but the modem is soldered onto the mainboard, is it not? So how can it possibly make a loose contact?

The remaining suspect then was the battery, also considering that the PinePhone modem is connected directly to the battery on the PinePhone. So I bought a replacement battery (a Samsung EB-BJ700CBE, which is documented as fitting) from a local retailer today. And well, yes, the battery fits, and yes, the battery works in principle. But the modem issue is the same whether I use the replacement battery or the original PinePhone battery (that I have now put back in), it makes no visible difference.

So at this point, I am stuck. I am completely out of ideas.

I am pretty annoyed at this point because I was using the PinePhone as my daily driver just fine, but now this issue makes it extremely unreliable (since without the modem, it is really just a tablet!) and I have no idea how to resolve the issue.

RE: PinePhone modem keeps "disappearing" - zetabeta - 01-18-2023

if the modem goes every 5 minutes then i start to suspect some failure, maybe in modem chip, or modem's software (firmware), or possible connection in mainboard.

modem has its own firmware, which might have become corrupted.

i think you should install firmware again, either stock or half-free sdk. however, i don't rule out that modem chip is faulty.

https://github.com/Biktorgj/quectel_eg25_recovery

https://github.com/the-modem-distro/pinephone_modem_sdk

however, i recommend setting modem's powersaving "on" in both stock rom and half-free sdk. following recommends it only for sdk.

https://github.com/the-modem-distro/pinephone_modem_sdk/blob/kirkstone/docs/SETTINGS.md

i have this following copy-paste to prevent some issues.

Code:
# example file location: /lib/udev/rules.d/98-prevent-bugs.rules

# prevent powersaving flipping frames bug

KERNEL=="1c40000.gpu", SUBSYSTEM=="platform", DRIVER=="lima", ATTR{power/autosuspend_delay_ms}="-1"

KERNEL=="1c40000.gpu", SUBSYSTEM=="platform", DRIVER=="lima", ATTR{power/control}="on"

# prevent frequency change flipping frames bug

KERNEL=="1c40000.gpu", SUBSYSTEM=="devfreq", ATTR{min_freq}="432000000"

KERNEL=="1c40000.gpu", SUBSYSTEM=="devfreq", ATTR{max_freq}="432000000" 

# prevent modem disappearings

ACTION!="add", GOTO="eg25_end-edit"

SUBSYSTEM!="usb", GOTO="eg25_end-edit"

DRIVER!="usb", GOTO="eg25_end-edit"

ENV{DEVTYPE}!="usb_device", GOTO="eg25_end-edit"

# NOT Default attributes values

ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0125", ATTR{power/control}="on"

ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0125", ATTR{power/autosuspend_delay_ms}="3000"

ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0125", ATTR{power/wakeup}="enabled"

ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0125", ATTR{power/persist}="0"

LABEL="eg25_end-edit"

# end of modem disappearings

i have started to think there is some bad quality in cpu-gpu soc. mostly related to powersaving. some hardware related functions work randomly and unreliably. powersaving is issue for both gpu and modem, is that a coincidence.

RE: PinePhone modem keeps "disappearing" - Gumur - 01-18-2023

(01-17-2023, 05:50 PM)Kevin Kofler Wrote: Since late December ...

Try another Os (vg. mobian), to see if the same issue occurs.

RE: PinePhone modem keeps "disappearing" - Kevin Kofler - 01-18-2023

Some more information: When I initially noticed the problem on December 31, it showed up in the journal as being unable to upload the GNSS data to the modem. (Unfortunately, the journal logs are not kept that long, so I cannot paste the exact log messages.) Well, actually, it was more like that the modem kept going down during the upload, getting restarted, and the upload failing again. So disabling GNSS uploads through /etc/eg25-manager/pine64,pinephone-1.2.toml was the first thing I tried. That showed pretty quickly that the GNSS issue was only a symptom rather than the cause. Still, I kept GNSS uploads disabled because I had the impression that that increased the likelihood of the modem resetting successfully.

Then I have had phases with the modem getting restarted very frequently, but being mostly up, and phases with the modem being mostly down.

Now, for 2 days or so, I have not seen the modem up at all anymore. And no, the replacement battery that I tried cannot be the cause, because the modem was already down for hours when I tried that yesterday evening.

The strange thing is, eg25-manager claims to be able to communicate with the modem, and even that it is "fully ready":

Code:
Jan 18 15:55:41 plasma-mobile eg25-manager[4496]: Opening config file: /usr/share/eg25-manager/pine64,pinephone-1.2.toml

Jan 18 15:55:41 plasma-mobile eg25-manager[4496]: Opening config file: /etc/eg25-manager/pine64,pinephone-1.2.toml

Jan 18 15:55:41 plasma-mobile eg25-manager[4496]: GNSS assistance is disabled!

Jan 18 15:55:41 plasma-mobile eg25-manager[4496]: Starting modem...

Jan 18 15:55:43 plasma-mobile eg25-manager[4496]: Executed power-on/off sequence

Jan 18 15:55:43 plasma-mobile eg25-manager[4496]: taking systemd sleep inhibitor

Jan 18 15:55:43 plasma-mobile eg25-manager[4496]: ModemManager appeared on D-Bus

Jan 18 15:55:43 plasma-mobile eg25-manager[4496]: oFono vanished from D-Bus

Jan 18 15:55:43 plasma-mobile eg25-manager[4496]: inhibitor sleep fd is 21

Jan 18 15:55:53 plasma-mobile eg25-manager[4496]: Response: [RDY]

Jan 18 15:55:53 plasma-mobile eg25-manager[4496]: taking systemd sleep inhibitor (blocking)

Jan 18 15:55:53 plasma-mobile eg25-manager[4496]: Response: [+CPIN: READY]

Jan 18 15:55:53 plasma-mobile eg25-manager[4496]: Executed soft sleep sequence

Jan 18 15:55:53 plasma-mobile eg25-manager[4496]: inhibitor block fd is 20

Jan 18 15:55:53 plasma-mobile eg25-manager[4496]: Response: [+QUSIM: 1]

Jan 18 15:55:53 plasma-mobile eg25-manager[4496]: Executed soft sleep sequence

Jan 18 15:55:53 plasma-mobile eg25-manager[4496]: Response: [+CFUN: 1]

Jan 18 15:55:53 plasma-mobile eg25-manager[4496]: Executed soft sleep sequence

Jan 18 15:55:53 plasma-mobile eg25-manager[4496]: Response: [+QIND: SMS DONE]

Jan 18 15:55:53 plasma-mobile eg25-manager[4496]: Executed soft sleep sequence

Jan 18 15:55:54 plasma-mobile eg25-manager[4496]: Response: [+QIND: PB DONE]

Jan 18 15:55:54 plasma-mobile eg25-manager[4496]: Executed soft sleep sequence

Jan 18 15:57:53 plasma-mobile eg25-manager[4496]: Modem is up for 120 seconds and fully ready

Jan 18 15:57:53 plasma-mobile eg25-manager[4496]: dropping systemd sleep block inhibitor

but I do not see the device actually showing up anywhere else, even the device node seems AWOL (at least I cannot find it under /dev).

systemctl restart eg25-manager can be used to trigger a reset attempt. It has sometimes helped, but not anymore, it seems. (It just gets stuck the same way again.)

Sometimes, eg25-manager also tries to reset the modem on its own. Then I can also see errors such as:

Code:
Jan 17 16:17:01 plasma-mobile eg25-manager[3024]: Modem wasn't probed in time, restart it!

Jan 17 16:17:01 plasma-mobile eg25-manager[3024]: Trying to reset modem with USB ID '3-1'

Jan 17 16:17:01 plasma-mobile eg25-manager[3024]: Couldn't unbind modem: wrote -1/3 bytes

Jan 17 16:17:01 plasma-mobile eg25-manager[3024]: USB reset failed, falling back to AT command

Code:
Jan 18 11:09:47 plasma-mobile eg25-manager[15692]: Modem wasn't probed in time, restart it!

Jan 18 11:09:47 plasma-mobile eg25-manager[15692]: Empty modem USB ID

Jan 18 11:09:47 plasma-mobile eg25-manager[15692]: USB reset failed, falling back to AT command

(I guess the latter is when the USB device node has never come up since bootup, the former seems to mean that the USB connection was lost somehow.)

I have now also tried the /lib/udev/rules.d/98-prevent-bugs.rules snippet (and of course I have rebooted the phone after creating the file, so that the rules are actually applied), it does not seem to make any difference either.

So the modem chip seems pretty FUBAR at this point.

Upgrading the firmware is something I have thought of as well, but if the modem goes down while flashing, it is going to be pretty much bricked. Also, I think I need the device node to flash anything to the modem, do I not?

I have not yet tried booting another OS on the main CPU. I will do so ASAP. (I need a suitable MicroSD card, as I do not really wish to overwrite the Jumpdrive I have on the one SD card I have at hand.) But at this point, I am not positive it will help. But of course it is worth trying.

RE: PinePhone modem keeps "disappearing" - Kevin Kofler - 01-18-2023

At least I now know why eg25-manager can talk to the modem while nothing else can: It looks like the modem is responding only over ttyS2 and not over USB. I can talk to it with minicom:

Code:
Welcome to minicom 2.8

OPTIONS: I18n 

Compiled on Jan  9 2021, 16:07:45.

Port /dev/ttyS2, 03:17:36

Press CTRL-A Z for help on special keys

AT+QGMR

EG25GGBR07A08M2G_01.002.01.002

OK

but it does not show up on lsusb at all.

RE: PinePhone modem keeps "disappearing" - Kevin Kofler - 01-19-2023

A few hours ago, after having been completely down for around 3 days, the modem started intermittently appearing on USB again. But it still disappears frequently, as I described in the original post.

All this sounds a lot like a contact issue with the USB connection. Which, as I have written before, is kinda strange because the modem is soldered on the same board as the CPU. But the symptoms sure look like a bad contact.

RE: PinePhone modem keeps "disappearing" - Westslope - 01-23-2023

(01-19-2023, 09:19 PM)Kevin Kofler Wrote: A few hours ago, after having been completely down for around 3 days, the modem started intermittently appearing on USB again. But it still disappears frequently, as I described in the original post.

All this sounds a lot like a contact issue with the USB connection. Which, as I have written before, is kinda strange because the modem is soldered on the same board as the CPU. But the symptoms sure look like a bad contact.

I've been having this same problem and have tried what's described in this post, and it seems to have worked so far.
https://forum.pine64.org/showthread.php?tid=11815&pid=107262#pid107262

RE: PinePhone modem keeps "disappearing" - Kevin Kofler - 01-24-2023

I am already using ATTR{power/control}="on" (since January 18, see zetabeta's snippet). It does not solve the issue. Often, the modem does not even come up on USB to begin with, so the udev rules will definitely have no effect, and even when it does, the udev rules do not prevent it from disappearing from USB. The modem is still running, mind you (as it can be talked to over ttyS2), just not on USB.

RE: PinePhone modem keeps "disappearing" - zetabeta - 01-24-2023

(01-24-2023, 07:30 AM)Kevin Kofler Wrote: I am already using ATTR{power/control}="on" (since January 18, see zetabeta's snippet). It does not solve the issue. Often, the modem does not even come up on USB to begin with, so the udev rules will definitely have no effect, and even when it does, the udev rules do not prevent it from disappearing from USB. The modem is still running, mind you (as it can be talked to over ttyS2), just not on USB.

i forgot to tell that modification is practically needed for funtional state. modem has bad habit to disconnect every few hours and disappear every day with default settings.

can you get to the state where "lsusb" says "Bus ### Device ###: ID 05c6:9008 Qualcomm, Inc. Gobi Wireless Modem (QDL mode)". does it appear and disappear constantly?

RE: PinePhone modem keeps "disappearing" - Kevin Kofler - 01-24-2023

Hmmm, before I start with that, one question: I see I can enter EDL/QDL mode with the instructions from:
* https://github.com/Biktorgj/quectel_eg25_recovery#enter-edl-with-adb
* https://xnux.eu/devices/feature/modem-pp.html#toc-unlock-adb-access

but is there a way to get back out of that mode without flashing a firmware?