I tried to analyze log files from one card having problematic connection. First my set-up briefly:
1) Embedded Linux card behind Huawei E3131 modem connection + NM & MM.
2) Drivers: Option and huawei_cdc_ncm.
3) Connection monitor with following functionality:
- monitoring frequency 10 minutes
- monitoring task, 1..18 ping operations with 10s timeout (max 3 minutes for one task)
- if all the 18 ping operations fail = task fails => USB power lines are sequenced.
- any of 18 ping operations passes = task passes => wait to next task start.
- if 12 successive tasks fail (2 hours) => boot card.
4) Occasionally bandwidth or field limited conditions that get modem firmware crash and boot frequently/at any time. It is not quite sure what crashes, but usb device gets re-numerated over and over.
From logs it seems that:
- Usually connection recovers automatically.
- If it does not, power sequencing usually helps.
- However, some times it does not help. If the first sequencing fails helping, next 10 power breaks fail too and 12'th task boots the card. It shows from the logs that USB enumeration does not work anymore when power sequencing stops working, which suggests that either udev or most likely some kernel driver has crashed. However, no kernel crash logs are printed.
Connection seems recovering soon after booting.