Re: bad gsm connections

[matti kaasinen <matti kaasinen gmail com>]

2017-04-07 18:38 GMT+03:00 Dan Williams <dcbw redhat com>:

On Thu, 2017-03-30 at 19:32 +0300, matti kaasinen wrote:
> I tried to analyze log files from one card having problematic
> connection.
> First my set-up briefly:
> 1) Embedded Linux card behind Huawei E3131 modem connection + NM &
> MM.
> 2) Drivers: Option and huawei_cdc_ncm.
> 3) Connection monitor with following functionality:
> - monitoring frequency 10 minutes
> - monitoring task, 1..18 ping operations with 10s timeout (max 3
> minutes
> for one task)
> - if all the 18 ping operations fail = task fails => USB power lines
> are
> sequenced.

How long is power to the card turned off?

2 seconds

> - any of 18 ping operations passes = task passes => wait to next task
> start.
> - if 12 successive tasks fail (2 hours) => boot card.
> 4) Occasionally bandwidth or field limited conditions that get modem
> firmware crash and boot frequently/at any time. It is not quite sure
> what
> crashes, but usb device gets re-numerated over and over.

If it keeps enumerating, that's the firmware crashing.

I forget if we've covered this or not, but are you sure there's
adequate power to the card?  If the modem gets into a situation where
it thinks it needs to transmit at a higher power and then draws too
much power, that could be the issue.  But I've seen enough firmware
changelogs to know that all kinds of firmware crash bugs get fixed all
the time too.

I tested this modem on my desk enclosed in metal enclosure to damp field. Then I fed modem directly through laboratory power supply, which did not stop operation described in 4). So, most likely firmware is crashing.  

> From logs it seems that:
> - Usually connection recovers automatically.
> - If it does not, power sequencing usually helps.
> - However, some times it does not help. If the first sequencing fails
> helping, next 10 power breaks fail too and 12'th task boots the card.
> It
> shows from the logs that USB enumeration does not work anymore when
> power
> sequencing stops working, which suggests that either udev or most
> likely
> some kernel driver has crashed. However, no kernel crash logs are
> printed.

Yes, it could be a kernel driver issue here, but you'd need to enable
kernel driver debugging for the USB Host Controller to figure that out,
most likely.  Which could require a kernel recompile.

I found a way how to reload usb kernel drivers. It was a bit tricky finding module who built sysfs structures for usb power access (/sys/kernel/debug/musb-hdrc.YY.auto/softconnect) - it was not musb_hdrc but its child: musb_dsps.

Correct sequence was switching usb power off, removing all the drivers in correct order and then reloading musb_dsps, which switched power on starting device enumeration. I used this sequence to monitoring tasks as follows:

Instead of executing pure usb power sequencing I made following variations:

Task #1 power sequencing
Task #2 udev restart
Task #3 usb driver reload
Task #4 usb driver reload and power sequencing
Task #5...9 usb driver reload and power sequencing with one more second off time
Task #10...11 usb driver reload and power sequencing with 8 seconds off time
Task #12 reboot

I hope this helps figuring out correct sequence.

If not, then I suppose, it's time kernel recompilation. It's OK, but It's kind of maintenance problem having several versions of kernel hanging around.

Only problem with this test set-up seems now the fact that gsm conditions have been too good for 15 days on that card that usually ends up booting several times a week.

I'll keep you informed on the results when I get any.

Thanks,

Matti