Re: tracking source of error message

[Dan Williams <dcbw redhat com>]

On Wed, 2014-01-01 at 08:48 -0800, skendric wrote:
> I'm seeing the following in syslog:
>
>
> Jan  1 06:36:34 guru kernel: [52348.129747] device eth0 left promiscuous
> mode
> Jan  1 06:36:36 guru NetworkManager[1326]: <warn> error monitoring device
> for netlink events: No buffer space available
> Jan  1 06:36:36 guru NetworkManager[1326]: <warn> error monitoring device
> for netlink events: error processing netlink message: Out of memory
> Jan  1 06:36:36 guru ntpd[6432]: i/o error on routing socket No buffer space
> available - disabling
> Jan  1 06:37:31 guru sshd[6503]: Accepted publickey for xxx from x.x.x.x
> port 51090 ssh2
> [...]
> Jan  1 06:44:22 guru kernel: [52816.295018] device eth0 left promiscuous
> mode
> Jan  1 06:45:06 guru NetworkManager[1326]: <warn> error monitoring device
> for netlink events: error processing netlink message: Out of memory
> Jan  1 06:52:01 guru CRON[6633]: pam_unix(cron:session): session opened for
> user root by (uid=0)
>
>
>
> Poking through src, I can see that NetworkManager generated these from
> nm_netlink_monitor.c using log_error_limited()
>
> [...]
> log_error_limited (NMNetlinkMonitor *monitor, guint32 code, const char *fmt,
> ...)
> {
>         [...]
>
>         if ((code != rl_code) || (now > rl_time + 10)) {
>                 va_start (args, fmt);
>                 msg = g_strdup_vprintf (fmt, args);
>                 va_end (args);
>
>                 nm_log_warn (LOGD_HW, "error monitoring device for netlink
> events: %s\n", msg);
> [...]
>
> But NetworkManager is just transporting the actual error messages ('No
> buffer space available' and 'error processing netlink message: out of
> memory')
>
> How might I continue working backward, to figure out what sent these error
> messages to NetworkManager?mailing 
>
> kernel 3.8.0-34-generic #49~precise1 (LTS 12.04.3).  NetworkManager 0.9.4.0
> ... e1000e-2.5.4-NAPI

It appears that your system is actually out of memory?  The second error
("error monitoring device for netlink events: error processing netlink
message: Out of memory") happens directly as a result of a memory
allocation failure.

Also look at the 'ntpd' output:

Jan  1 06:36:36 guru ntpd[6432]: i/o error on routing socket No buffer
space available - disabling

which would also indicate out of memory.  When you are in this
situation, what is the output of:

ps -aux | grep NetworkManager
cat /proc/meminfo  | grep Mem

?

Dan