GNOME.org
 

Re: primary device changing from boot to boot

On Tue, 2017-09-05 at 12:32 -0700, Tim Harvey wrote:

On Tue, Sep 5, 2017 at 9:53 AM, Dan Williams <dcbw redhat com> wrote:

On Tue, 2017-09-05 at 09:38 -0700, Tim Harvey wrote:

On Fri, Sep 1, 2017 at 1:22 PM, Dan Williams <dcbw redhat com>
wrote:

On Fri, 2017-09-01 at 13:01 -0700, Tim Harvey wrote:

Greetings,

I've got a Sierra Wireless HL7588 modem which exposes three
ttyACM
devs via the cdc_acm driver. The device appears to work fine
with
ModemManager (1.6.8) but occasionally mm detects the primary
device
as
ttyACM2 instead of the usuall ttyACM0. In this case (detected
primary
AT interface of ttyACM2), if I've already configured
NetworkManager
to
use /dev/ttyACM0 I can't bring up the modem.


NM has a "DeviceIdentifier" property on GSM connections partly
for
this
reason.  You cannot always guarantee that even if MM *did*
detect
the
first exposed TTY as primary, that it's going to be called
ttyACM0.
The kernel is free to name these things whatever it wants, and
if
ttyACM0 is already used it'll pick another one.  I've had this
happen
when some program holds the TTY open and the modem reboots.

So moral of the story is, don't depend on device names.

Get the MM device identifier from the modem:

dbus-send --print-reply --system --
dest=org.freedesktop.ModemManager1
/org/freedesktop/ModemManager1/Modem/0
org.freedesktop.DBus.Properties.Get
string:org.freedesktop.ModemManager1.Modem
string:DeviceIdentifier

and then set that string as the DeviceIdentifier in the
NetworkManager
connection for the modem.  That connection will then only ever
apply to
that specific modem.  You can then do things liek "nmcli con up
<connection mame>" and NM will figure out what device it should
use.



Dan,

This makes sense, but how do I configure NetworkManager to use
the
device-id? I don't see it as an available option for type gsm, so
perhaps I need to edit the file in
/etc/NetworkManager/system-connections/mymodem and add it to the
'gsm'


nmcli con mod <connection name> gsm.device-id \
    6909b49a4d44867387a2b09b8095c579e031874c

Or yeah, drop:

device-id=6909b49a4d44867387a2b09b8095c579e031874c

into the [gsm] section of the keyfile in
/etc/NetworkManager/system-
connections.



Using 'device-id' does not appear to not work, at least not in NM
1.2.6:


1.2.6 should have the right support.  Perhaps I didn't quite understand
what you were asking though.

device-id is useful if you have multiple modems on a system, or if you
swap modems between systems and want a connection to apply to a
specific device.

But your original question was about device naming, and ACM0 vs. ACM2. 
For some modems, it shouldn't matter what the interface name is.  So in
your case, you wouldn't do anything to "configured NetworkManager to
use /dev/ttyACM0", you'd simply set the device-id and then use the NM
connection name to start/stop things.  You wouldn't hardcode "nmcli dev
connect ttyACM0" or anything like that.

Instead you'd "nmcli con up Verizon" and the device-id would tell NM to
use the right device, no matter whether ACM0, ACM2, or USB3 or
whatever.

Basically, you don't want to hardcode tty device names ever, because
they can and do change.  So if you're doing that somewhere, I'd
recommend instead using the NM connection name, and device-id to make
sure the connection always applies to that specific modem.

Dan


root@ventana:~# dbus-send --print-reply --system \
   --dest=org.freedesktop.ModemManager1 \
   /org/freedesktop/ModemManager1/Modem/0 \
   org.freedesktop.DBus.Properties.Get \
   string:org.freedesktop.ModemManager1.Modem string:DeviceIdentifier
method return time=1504639000.492891 sender=:1.2 -> destination=:1.11
serial=36 reply_serial=2
   variant       string "6909b49a4d44867387a2b09b8095c579e031874c"


root@ventana:~# ls /etc/NetworkManager/system-connections
mymodem

root@ventana:~# cat /etc/NetworkManager/system-connections/mymodem
[connection]
id=mymodem
uuid=8ab55a09-43b0-430d-807a-1773ba22d605
type=gsm
interface-name=ttyACM0
permissions=
secondaries=

[gsm]
apn=vzwinternet
number=*99#
device-id=6909b49a4d44867387a2b09b8095c579e031874c

[ipv4]
dns-search=
method=auto

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

root@ventana:~# mmcli -m 0

/org/freedesktop/ModemManager1/Modem/0 (device id
'6909b49a4d44867387a2b09b8095c579e031874c')
  -------------------------
  Hardware |   manufacturer: 'Sierra Wireless'
           |          model: 'HL7588'
           |       revision:
'RHL75xx.V.3.7.151600.201702071034.x7160_1'
           |      supported: 'gsm-umts, lte'
           |        current: 'gsm-umts, lte'
           |   equipment id: '014284000070452'
  -------------------------
  System   |         device:
'/sys/devices/soc0/soc/2100000.aips-bus/2184200.usb/ci_hdrc.1/usb2/2-
1'
           |        drivers: 'cdc_acm, cdc_ncm'
           |         plugin: 'Generic'
           |   primary port: 'ttyACM2'
           |          ports: 'ttyACM2 (at), wwx000011121314 (net),
wwx000011121316 (net), wwx00001112131a (net), wwx000011121318 (net),
ttyACM0 (at)'
  -------------------------
  Numbers  |           own : '+19529136397'
  -------------------------
  Status   |           lock: 'none'
           | unlock retries: 'unknown'
           |          state: 'registered'
           |    power state: 'on'
           |    access tech: 'lte'
           | signal quality: '19' (recent)
  -------------------------
  Modes    |      supported: 'allowed: 2g, 3g, 4g; preferred: none'
           |        current: 'allowed: 2g, 3g, 4g; preferred: none'
  -------------------------
  Bands    |      supported: 'unknown'
           |        current: 'unknown'
  -------------------------
  IP       |      supported: 'ipv4, ipv6, ipv4v6'
  -------------------------
  3GPP     |           imei: '014284000070452'
           |  enabled locks: 'none'
           |    operator id: '311480'
           |  operator name: 'Verizon'
           |   subscription: 'unknown'
           |   registration: 'home'
  -------------------------
  SIM      |           path: '/org/freedesktop/ModemManager1/SIM/0'

  -------------------------
  Bearers  |          paths: 'none'

root@ventana:~# nmcli device status
DEVICE   TYPE      STATE         CONNECTION
ttyACM2  gsm       disconnected  --
eth0     ethernet  unavailable   --
can0     can       unmanaged     --
lo       loopback  unmanaged     --
^^^^ not able to connect when ttyACM2 is the primary

Here's NM log in case that sheds light on the issue:

root@ventana:~# journalctl -u NetworkManager
-- Logs begin at Thu 2016-02-11 16:28:01 UTC, end at Tue 2017-09-05
19:17:51 UTC. --
Sep 05 19:12:49 ventana systemd[1]: NetworkManager.service: About to
execute: /usr/sbin/NetworkManager --no-daemon
Sep 05 19:12:49 ventana systemd[1]: NetworkManager.service: Forked
/usr/sbin/NetworkManager as 291
Sep 05 19:12:49 ventana systemd[1]: NetworkManager.service: Changed
dead -> start
Sep 05 19:12:49 ventana systemd[1]: Starting Network Manager...
Sep 05 19:12:49 ventana systemd[291]: NetworkManager.service:
Executing: /usr/sbin/NetworkManager --no-daemon
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.2151]
NetworkManager (version 1.2.6) is starting...
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.2181]
Read config: /etc/NetworkManager/NetworkManager.conf (etc:
default-wifi-powersave-on.conf)
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.2880]
manager[0xb240f8]: monitoring kernel firmware directory
'/lib/firmware'.
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.2883]
monitoring ifupdown state file '/run/network/ifstate'.
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.3103]
dns-mgr[0xb15cb0]: init: dns=dnsmasq, rc-manager=resolvconf,
plugin=dnsmasq
Sep 05 19:12:50 ventana systemd[1]: NetworkManager.service: D-Bus
name
org.freedesktop.NetworkManager changed owner from  to :1.3
Sep 05 19:12:50 ventana systemd[1]: NetworkManager.service: Changed
start -> running
Sep 05 19:12:50 ventana systemd[1]: NetworkManager.service: Job
NetworkManager.service/start finished, result=done
Sep 05 19:12:50 ventana systemd[1]: Started Network Manager.
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.5646] init!
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.5654]
      interface-parser: parsing file /etc/network/interfaces
Sep 05 19:12:50 ventana NetworkManager[291]:
<warn>  [1504638770.5671]
ignoring out-of-block data 'source-directory
/etc/network/interfaces.d'
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.5694]
      interface-parser: finished parsing file /etc/network/interfaces
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.5706]
management mode: unmanaged
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.5738]
devices added (path:
/sys/devices/soc0/soc/2100000.aips-bus/2188000.ethernet/net/eth0,
iface: eth0)
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.5740]
device added (path:
/sys/devices/soc0/soc/2100000.aips-bus/2188000.ethernet/net/eth0,
iface: eth0): no ifupdown configuration found.
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.5755]
devices added (path: /sys/devices/virtual/net/lo, iface: lo)
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.5781]
device added (path: /sys/devices/virtual/net/lo, iface: lo): no
ifupdown configuration found.
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.5803]
end _init.
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.5822]
settings: loaded plugin ifupdown: (C) 2008 Canonical Ltd.  To report
bugs please use the NetworkManager mailing list.
(/usr/lib/arm-linux-gnueabihf/NetworkManager/libnm-settings-plugin-
ifupdown.so)
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.5841]
settings: loaded plugin keyfile: (c) 2007 - 2015 Red Hat, Inc.  To
report bugs please use the NetworkManager mailing list.
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.5993]
SettingsPlugin-Ofono: init!
Sep 05 19:12:50 ventana NetworkManager[291]:
<warn>  [1504638770.6030]
SettingsPlugin-Ofono: file doesn't exist: /var/lib/ofono
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.6078]
SettingsPlugin-Ofono: end _init.
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.6130]
settings: loaded plugin ofono: (C) 2013-2016 Canonical Ltd.  To
report
bugs please use the NetworkManager mailing list.
(/usr/lib/arm-linux-gnueabihf/NetworkManager/libnm-settings-plugin-
ofono.so)
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.6156]
(11767600) ... get_connections.
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.6182]
(11767600) ... get_connections (managed=false): return empty list.
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.7678]
keyfile: new connection /etc/NetworkManager/system-
connections/mymodem
(8ab55a09-43b0-430d-807a-1773ba22d605,"mymodem")
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.7722]
SettingsPlugin-Ofono: (11767720) ... get_connections.
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.7724]
SettingsPlugin-Ofono: (11767720) connections count: 0
Sep 05 19:12:50 ventana NetworkManager[291]:
<info>  [1504638770.7729]
get unmanaged devices count: 0
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2656]
settings: hostname: using hostnamed
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2658]
settings: hostname changed from (none) to "ventana"
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2670]
Using DHCP client 'dhclient'
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2672]
manager: WiFi enabled by radio killswitch; enabled by state file
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2674]
manager: WWAN enabled by radio killswitch; enabled by state file
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2676]
manager: Networking is enabled by state file
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2679]
Loaded device plugin: NMVxlanFactory (internal)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2681]
Loaded device plugin: NMVlanFactory (internal)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2683]
Loaded device plugin: NMVethFactory (internal)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2685]
Loaded device plugin: NMTunFactory (internal)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2687]
Loaded device plugin: NMMacvlanFactory (internal)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2688]
Loaded device plugin: NMIPTunnelFactory (internal)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2690]
Loaded device plugin: NMInfinibandFactory (internal)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2692]
Loaded device plugin: NMEthernetFactory (internal)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2695]
Loaded device plugin: NMBridgeFactory (internal)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.2696]
Loaded device plugin: NMBondFactory (internal)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.3260]
Loaded device plugin: NMBluezManager
(/usr/lib/arm-linux-gnueabihf/NetworkManager/libnm-device-plugin-
bluetooth.so)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.3927]
Loaded device plugin: NMWwanFactory
(/usr/lib/arm-linux-gnueabihf/NetworkManager/libnm-device-plugin-
wwan.so)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.4155]
Loaded device plugin: NMWifiFactory
(/usr/lib/arm-linux-gnueabihf/NetworkManager/libnm-device-plugin-
wifi.so)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.4510]
Loaded device plugin: NMAtmManager
(/usr/lib/arm-linux-gnueabihf/NetworkManager/libnm-device-plugin-
adsl.so)
Sep 05 19:12:51 ventana NetworkManager[291]:
nm_device_get_device_type: assertion 'NM_IS_DEVICE (self)' failed
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.4581]
device (lo): link connected
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.4683]
manager: (lo): new Generic device
(/org/freedesktop/NetworkManager/Devices/0)
Sep 05 19:12:51 ventana NetworkManager[291]:
<warn>  [1504638771.4716]
device (eth0): failed to find device 2 'eth0' with udev
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.4835]
manager: (eth0): new Ethernet device
(/org/freedesktop/NetworkManager/Devices/1)
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.4974]
manager: startup complete
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.5621]
urfkill disappeared from the bus
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.5785]
ModemManager available in the bus
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.5917]
ofono is now available
Sep 05 19:12:51 ventana NetworkManager[291]:
<warn>  [1504638771.5973]
failed to enumerate oFono devices:
GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name
org.ofono was not provided by any .service files
Sep 05 19:12:51 ventana NetworkManager[291]:
nm_device_get_device_type: assertion 'NM_IS_DEVICE (self)' failed
Sep 05 19:12:51 ventana NetworkManager[291]:
<info>  [1504638771.9398]
manager: (can0): new Generic device
(/org/freedesktop/NetworkManager/Devices/2)
Sep 05 19:12:52 ventana NetworkManager[291]:
<info>  [1504638772.7851]
devices added (path:
/sys/devices/soc0/soc/2100000.aips-bus/2188000.ethernet/net/eth0,
iface: eth0)
Sep 05 19:12:52 ventana NetworkManager[291]:
<info>  [1504638772.7853]
device added (path:
/sys/devices/soc0/soc/2100000.aips-bus/2188000.ethernet/net/eth0,
iface: eth0): no ifupdown configuration found.
Sep 05 19:12:52 ventana NetworkManager[291]:
<info>  [1504638772.7865]
device (eth0): state change: unmanaged -> unavailable (reason
'managed') [10 20 2]
Sep 05 19:12:52 ventana NetworkManager[291]:
<info>  [1504638772.8562]
keyfile: add connection in-memory
(31c82493-a694-336f-a7e3-df589c1e43fe,"Wired connection 1")
Sep 05 19:12:52 ventana NetworkManager[291]:
<info>  [1504638772.8650]
settings: (eth0): created default wired connection 'Wired connection
1'
Sep 05 19:12:53 ventana NetworkManager[291]:
<info>  [1504638773.1879]
devices added (path: /sys/devices/virtual/net/lo, iface: lo)
Sep 05 19:12:53 ventana NetworkManager[291]:
<info>  [1504638773.1936]
device added (path: /sys/devices/virtual/net/lo, iface: lo): no
ifupdown configuration found.
Sep 05 19:12:53 ventana NetworkManager[291]:
<info>  [1504638773.8825]
devices added (path:
/sys/devices/soc0/soc/2000000.aips-bus/2090000.flexcan/net/can0,
iface: can0)
Sep 05 19:12:53 ventana NetworkManager[291]:
<info>  [1504638773.8827]
device added (path:
/sys/devices/soc0/soc/2000000.aips-bus/2090000.flexcan/net/can0,
iface: can0): no ifupdown configuration found.
Sep 05 19:12:54 ventana NetworkManager[291]:
<info>  [1504638774.7653]
devices added (path:
/sys/devices/soc0/soc/2100000.aips-bus/2184200.usb/ci_hdrc.1/usb2/2-
1/2-1:1.12/net/wwx00001112131a,
iface: wwx00001112131a)
Sep 05 19:12:54 ventana NetworkManager[291]:
<info>  [1504638774.7655]
device added (path:
/sys/devices/soc0/soc/2100000.aips-bus/2184200.usb/ci_hdrc.1/usb2/2-
1/2-1:1.12/net/wwx00001112131a,
iface: wwx00001112131a): no ifupdown configuration found.
Sep 05 19:12:54 ventana NetworkManager[291]:
<info>  [1504638774.8066]
devices added (path:
/sys/devices/soc0/soc/2100000.aips-bus/2184200.usb/ci_hdrc.1/usb2/2-
1/2-1:1.8/net/wwx000011121316,
iface: wwx000011121316)
Sep 05 19:12:54 ventana NetworkManager[291]:
<info>  [1504638774.8068]
device added (path:
/sys/devices/soc0/soc/2100000.aips-bus/2184200.usb/ci_hdrc.1/usb2/2-
1/2-1:1.8/net/wwx000011121316,
iface: wwx000011121316): no ifupdown configuration found.
Sep 05 19:12:54 ventana NetworkManager[291]:
<info>  [1504638774.8278]
devices added (path:
/sys/devices/soc0/soc/2100000.aips-bus/2184200.usb/ci_hdrc.1/usb2/2-
1/2-1:1.6/net/wwx000011121314,
iface: wwx000011121314)
Sep 05 19:12:54 ventana NetworkManager[291]:
<info>  [1504638774.8279]
device added (path:
/sys/devices/soc0/soc/2100000.aips-bus/2184200.usb/ci_hdrc.1/usb2/2-
1/2-1:1.6/net/wwx000011121314,
iface: wwx000011121314): no ifupdown configuration found.
Sep 05 19:12:54 ventana NetworkManager[291]:
<info>  [1504638774.9195]
devices added (path:
/sys/devices/soc0/soc/2100000.aips-bus/2184200.usb/ci_hdrc.1/usb2/2-
1/2-1:1.10/net/wwx000011121318,
iface: wwx000011121318)
Sep 05 19:12:54 ventana NetworkManager[291]:
<info>  [1504638774.9197]
device added (path:
/sys/devices/soc0/soc/2100000.aips-bus/2184200.usb/ci_hdrc.1/usb2/2-
1/2-1:1.10/net/wwx000011121318,
iface: wwx000011121318): no ifupdown configuration found.
Sep 05 19:13:00 ventana NetworkManager[291]:
<info>  [1504638780.0730]
manager: WiFi hardware radio set enabled
Sep 05 19:13:00 ventana NetworkManager[291]:
<info>  [1504638780.0732]
manager: WWAN hardware radio set enabled
Sep 05 19:13:12 ventana NetworkManager[291]:
<info>  [1504638792.8925]
(ttyACM2): modem state changed, 'disabled' --> 'enabling' (reason:
user preference)
Sep 05 19:13:12 ventana NetworkManager[291]:
<info>  [1504638792.8988]
manager: (ttyACM2): new Broadband device
(/org/freedesktop/NetworkManager/Devices/3)
Sep 05 19:13:12 ventana NetworkManager[291]:
<info>  [1504638792.9006]
device (ttyACM2): state change: unmanaged -> unavailable (reason
'managed') [10 20 2]
Sep 05 19:13:12 ventana NetworkManager[291]:
<info>  [1504638792.9029]
device (ttyACM2): modem state 'enabling'
Sep 05 19:13:14 ventana NetworkManager[291]:
<info>  [1504638794.3136]
(ttyACM2): modem state changed, 'enabling' --> 'registered' (reason:
user-requested)
Sep 05 19:13:14 ventana NetworkManager[291]:
<info>  [1504638794.3144]
device (ttyACM2): state change: unavailable -> disconnected (reason
'modem-available') [20 30 58]
^^^^ looks like its simply still tryig to use ttyACM2


section but maybe I've got larger issues because nm seems to not
even
like ttyACM0 for the interface:



<snip>


root@ventana:~# nmcli device status
DEVICE   TYPE      STATE         CONNECTION
eth0     ethernet  disconnected  --
ttyACM0  gsm       unavailable   --
can0     can       unmanaged     --
lo       loopback  unmanaged     --


So NM thinks the modem is unavailable.  That's usually caused by
rfkill-switch-type stuff, or because the WWAN plugin hasn't been
built
for NM, or it hasn't been built with the right options.

Can you grab NM log output and we can see what's going on?

Also, 'rfkill list' and 'nmcli radio' output.


Let's chalk this up to user error - I think I may have been
experiencing this due to having multiple modem's configured in
/etc/NetworkManager/system-connections/ when I was playing with
'nmcli
connection add type gsm' syntax.

Tim
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]