Re: Extending the mobile-broadband-provider-info database

[Marcel Holtmann <marcel holtmann org>]

Hi Dan,

> > > > >>> As long as the attributes we add aren't required, it should be fine to
> > > > >>> add them.  See the balance check stuff that was added a while ago; that
> > > > >>> was added in a compatible manner and existing tools simply ignore the
> > > > >>> new attributes.
> > > > >>>
> > > > >> About the existing information there. I see it is mobile network 
> > > > >> identifiers, but I haven't seen that Network manager exploit this to 
> > > > >> automatically select or propose the correct network settings. Is this 
> > > > >> not implemented yet?
> > > > > It's done in the client applications like nm-applet and
> > > > > nm-connection-editor and the KDE bits, which create the actual
> > > > > configuration and display stuff like signal and network names.  NM
> > > > > simply passes the mobile broadband specific configuration along to
> > > > > ModemManager, but isn't itself involved at all with the carrier/APN
> > > > > selection.  nm-applet also makes use of the database to pull out a
> > > > > carrier name based on MCC/MNC/SID if the modem doesn't one.
> > > > 
> > > > Okay, I just haven't seen it happen in nm-applet. I always have had to
> > > > select country and then operator/subscription.
> > > 
> > > nm-applet doesn't yet pull the operator MCC/MNC out of the sim (and in
> > > fact, lots of devices don't allow this for some reason, nothing to do
> > > with the SIM, but the device).  But nm-applet *does* use the database
> > > for the selection dialog you're talking about, and it uses the database
> > > for dispalying the operator name in the menu if the device doesn't
> > > provide one.  It also uses it to show the CDMA operator name (based on
> > > SID) if you're using a CDMA device.  So quite a few things at the
> > > moment.
> > 
> > actually all devices allow you to read the MCC/MNC from the SIM card.
> > The MCC is the first 3 bytes of your IMSI. And the MNC follows the MCC.
> > The only tricky part is that the MNC can be 2 or 3 digits. And to know
> > what it actually is, you need to read the MNC length from the SIM card.
> > You do need proper EF reading for that.
> 
> Yeah, but see below, it often doesn't work.  Not to say we can't try to
> get it if it's available, but it's unreliable at best and broken at
> worst.
> 
> > The only problem is that you need to tweak the EF reading for some
> older
> > devices since it seems that it has not been tested enough for the data
> > dongles. Most likely that is a Qualcomm problem since we only saw issue
> > with it on old dongles using Qualcomm chips and thus most likely their
> > AT command implementation.
> 
> Yeah, that's what I've found on a number of devices.  But SIMs (or
> devices) are also pretty weird here.  My standard T-Mobile USIM (from
> 2009) simply doesn't have >= 4 bytes in EF_AD and thus we can't read the
> MNC size at all, and since T-Mobile is one of the largest companies with
> a 3-digit MNC that seems pretty odd:
> 
> (Option 452 TMO SIM):
> AT+CRSM=176,28589,0,0,255
> +CRSM: 144,0,"00FFFF"

some Option cards are funny. There file size reading is all broken. So
you need to force it to the correct size.

> AT+CRSM=176,28589,0,0,4\r
< \r\r\n+CRSM: 144,0,"00000002"\r\n\r\nOK\r\n

> (Sierra 306 TMO SIM):
> AT+CRSM=176,28589,0,0,4
> +CRSM: 103,0,""
> AT+CRSM=176,28589,0,0,3
> +CRSM: 144,0,"00FFFF"

The Sierra card will give you the correct file size.

> AT+CRSM=192,28589,0,0,255\r
< \r\n+CRSM: 144,0,"62178202412183026FAD8A01058B036F060180020004880118"\r\n\r\nOK\r\n
> AT+CRSM=176,28589,0,0,4\r
< \r\n+CRSM: 144,0,"00000003"\r\n\r\nOK\r\n

> (Option 452 3UK SIM):
> AT+CRSM=176,28589,0,0,255
> +CRSM: 144,0,"00000002"

The really important part is that you have to implement EF reading
properly. You can not just go ahead and try to read the file. That might
work, but in most cases you wanna read the directory structure and the
permission information first.

> All three use Qualcomm chipsets but the SIM is what makes the
> difference.  Both a TMO and a 3UK SIM refuse when requesting the SPN
> too:
> 
> AT+CRSM=176,28486,0,0,255
> +CRSM: 111,0
> 
> Or is that the device?

The SPN does not have to be present. It is optional and then the
reported value from AT+COPS is correct.

Some firmwares mangle the SPN and NITZ updates into AT+COPS responses. I
wish they wouldn't, but in general if SPN is present, then you should be
using SPN.

> > Using the database for displaying the operator name is also wrong. At
> > least for GSM. You need to use the SPN value from the SIM card for this.
> > In addition you need to let the firmware handle NITZ updates. Operator
> > names can change on the fly and can be changed by the network.
> 
> Yes, but I've seen some examples where that's also completely wrong.
> For example, a ZTE MF627 from 3UK with a 3UK SIM lists T-Mobile US as
> "Voicestream" which it hasn't been since 2003, though that might be the
> firmware since sticking that SIM in another device (Option 452) lists
> TMO correctly in COPS results.

As I said, the network will update its values via NITZ. And depending on
the firmware it will store it in flash or not. However the network will
update it again for you. So try reading AT+COPS when you get a network
cell change etc. Sometime it takes a few minutes (or longer), but
usually it works pretty well.

> And some AT&T SIMs force all operators to "AT&T" because they don't want
> you to know how bad their network is when you're roaming.  Some devices
> don't report the operator name at all, which means we have to grab it
> out of the database.  I don't particularly trust whatever is in the
> SIM/device for that and number of other reasons. Operators often don't
> update their SIM cards when their roaming partners merge, get bought
> out, or change names, and the easiest way for us to handle that is
> through the mobile provider database.

The SIM and the network in combination. Meaning SPN and NITZ can be
trusted. You just have to do this the right way.

I have tested this on a bunch of networks that have changed owners and
names over time, they all show up correctly. Some crappier firmware
takes longer than others, but usually it sorts itself out.

And if AT&T SIMs want to force the display name via SPN, then so be it.
That is why the SIM cards provision themselves that way. The AT&T SIM
cards also set a flag for the telephony stack to disable manual network
selection.

> > That is another reason why having the SPN values also in the database.
> > That way we can correctly identify MNVO which just share MCC/MNC with
> > the actual network operator.
> 
> That would be good, yes.  Except when the device doesn't allow you to
> access the SPN at all (see above with Option 452 with 3UK SIM) in which
> case you're stuck back with the database or with the COPS output.

Trust me, it is optional, but the MVNO normally do set it. The actual
operator normally does not.

> > Some devices have special AT commands for reading the SPN, but you can
> > access it via EF from the SIM card as well.
> 
> On some devices, apparently.  Clearly not a ton of them... :(

So far, reading directly from the SIM card has always worked out nicely
for us. Some needed some quirks, but that was about it.

Regards

Marcel