Re: Network Manager crashes after a configuration is added and a connection is attempted.

[Chris Hessing <chsoftworks gmail com>]

On 7/3/2014 2:39 AM, Thomas Haller wrote:
> On Wed, 2014-07-02 at 12:48 -0600, Chris Hessing wrote:
> > On 7/2/2014 12:00 PM, Thomas Haller wrote:
> > > On Wed, 2014-07-02 at 09:41 -0600, Chris Hessing wrote:
> > > > On 7/2/2014 7:43 AM, Dan Williams wrote:
> > > > > On Wed, 2014-07-02 at 11:26 +0200, Thomas Haller wrote:
> > > > > > On Tue, 2014-07-01 at 16:52 -0600, Chris Hessing wrote:
> > > > > > > Hi all,
> > > > > > > I thought it might be an issue with the variant type I was sending in to
> > > > > > > some configuration setting that was later being parsed when I requested
> > > > > > > the connection become active.   However, after going through my code to
> > > > > > > generate the settings, they all match what is documented here :
> > > > > > > https://developer.gnome.org/NetworkManager/0.9/ref-settings.html
> > > > > > >
> > > > > > > One interesting tidbit is that after Network Manager crashes, and is
> > > > > > > restarted by the system, the connection comes up fine and seems to work
> > > > > > > from then on.
> > > > > > >
> > > > > > > At this point, short of cracking open the Network Manager code, I am
> > > > > > > running out of ideas as to what it could be.   Perhaps someone that
> > > > > > > understands the Network Manager code can shed some light on what might
> > > > > > > be going on?
> > > > > > Hi Chris,
> > > > > >
> > > > > >
> > > > > >
> > > > > > This looks like a bug in NetworkManager.
> > > > > >
> > > > > > The attached patch should fix it.
> > > > > Patch looks right to me.  It's likely unnoticed because these days most
> > > > > certificates use the "path" scheme instead of the blob scheme, and thus
> > > > > this code doesn't get triggered.
> > > > >
> > > > > > Interestingly,
> > > > > > http://dbus.freedesktop.org/doc/dbus-glib/dbus-glib-Specializable-GType-System.html#DBUS-TYPE-G-UCHAR-ARRAY:CAPS
> > > > > >  says that we cannot pass a GByteArray instead of the expected GArray of uchar. But I think the comment is 
> > > > > > just wrong, because a GByteArray *is* actually a GArray.
> > > > > We should look at this a bit more; I ran into this when doing the
> > > > > wwan-ipv6 stuff and it did cause a crash that was fixed by using GArray
> > > > > instead of GByteArray.  But we've been using GByteArray for a really
> > > > > long time, so I'm sure that at some point in the past this worked.
> > > > >
> > > > > Dan
> > > > Hi guys,
> > > >
> > > > Thanks for the patch and the info.   I'll look in to changing my code to
> > > > use the path method instead of the blob method.   I know I used the path
> > > > method in the past, but there was some issue I ran in to that made me
> > > > move to the blob method.    As I move back toward the path method I'll
> > > > update you guys if I happen to trigger or remember why I went that way.
> > > >
> > > >
> > > > Thanks again for the info!
> > > Hey,
> > >
> > > I opened Fedora bug https://bugzilla.redhat.com/show_bug.cgi?id=1115538
> > > for this. The next version NetworkManager-0.9.9.0-41.git20131003.fc20
> > > will bring a fix.
> > >
> > > @Chris, you can also install the new package manually, before it is
> > > stable. Get the packages from
> > > https://admin.fedoraproject.org/updates/NetworkManager-0.9.9.0-41.git20131003.fc20 .
> > >
> > > I was not so much interested in why you happened to change the method
> > > path vs. blob. Instead I would be very much interested whether the
> > > problem is actually fixed. Thank you!! ;-)
> > >
> > > ciao,
> > > Thomas
> > Nope, the patch didn't seem to fix it.  I am seeing the same log
> > output.   I also double checked the Network Manager -V output, and it is
> > "0.9.9.0-41.git20131003.fc20" which appears to be the version you wanted
> > me to test.
> Oh...
>
>
> > What other information can I provide that might help?
> Is it really the same crash? The original stack trace did not have
> NM-debugging symbols. Could you please explicitly install the
> NetworkManager-debuginfo package that you find on
> http://koji.fedoraproject.org/koji/buildinfo?buildID=541645 ?

I didn't run gdb on it when I tested, so I am making the assumption that 
it is the same crash based on the NM logs showing the same errors right 
before it crashes.  (But, yes, I know what happens when you assume. ;)   
It will probably be early next week before I can get a minute to run the 
test again and get a new backtrace.

> Also, the logfile shows some glib warnings before the actual crash.
>    GLib-GObject-WARNING **: gtype.c:4215: type id '115' is invalid
> These warnings are probably related and closer to the actual cause.
> You can make glib dump core on warnings, by setting the environment
> variable:
>
>    G_DEBUG=fatal-warnings

I do have a theory about that after some work yesterday on moving to 
using paths for the cert.   I believe that QT is trying to be too smart 
about how it is building the D-Bus messages.   When I tried to send in 
the paths, I was handing QT a QByteArray, which I figured would be send 
as a blob.   However, what seemed to arrive at NM, based on using 
dbus-monitor looked quite different than what was being sent by the UI 
in Fedora.   As a result, NM was ignoring the paths that I sent in.    I 
suspect the same type of thing is happening when I send the private key 
in PEM format.   Even though I tell QT to use a byte array, it is 
probably converting it to a string under the hood and sending it in.   
So, when I get a minute next week to do some testing, I am going to see 
if the trick I am using to convert the paths to a byte array works with 
the private key data and prevents the crash from happening.   I'll 
update you on the outcome.

> To do this, you have to run NetworkManager not as system service:
>    systemctl disable NetworkManager.service
>    systemctl stop NetworkManager.service
>    export G_DEBUG=fatal-warnings
>    ulimit -c unlimited
>    gdb /sbin/NetworkManager
>    > run --debug
>
> Afterwards, systemctl enable && systemctl start. ... you need to disable
> the service, because otherwise it might be DBUS activated by systemd
> after you stop it.
>
>
> Alternatively, I figure you could inject the environment to the running
> NM, see http://stackoverflow.com/a/211064 (I did not test that though).

Will do.   And I'll go with the method that you have tested.  No reason 
to bring more variables in to this.

> Debug-logging is also useful.
> Add
>
> [logging]
> level=DEBUG
> domains=ALL
>
> to /etc/NetworkManager/NetworkManager.conf
>
>
> And then, reproduce.

Is this any different than using the debug-helper.py app at 
https://wiki.ubuntu.com/DebuggingNetworkManager?  That is what I have 
been using so far to see debug logs.

> > FWIW, I mentioned the stuff about the path version because that is what
> > I used to use, but something made me move away from it.   That something
> > was either a restriction in the OS, or a different bug in Network
> > Manager.  If it was a bug, I figured you would be interested.
> Sure, sorry for being so blunt :)
> We are interested in bugs.
>
>
> Thank you,
> Thomas