Re: Very slow DNS lookup with NetworkManager and dnsmasq

[Dan Williams <dcbw redhat com>]

On Wed, 2011-09-14 at 16:26 +0300, Jarmo Hurri wrote:
> Greetings.
> 
> I have run out of tricks to solve the following problem: I am frequently
> spending _long_ times waiting for DNS lookups even though
> 1. all the Windows machines in the same network perform fast DNS lookups
> 2. I have NetworkManager using dnsmasq.
> 
> My setup is the following:
> - Fedora 14, kernel 2.6.35.14-96.fc14.x86_64
> - NetworkManager 0.8.4
> - dnsmasq 2.52.1.fc13
> 
> In /etc/NetworkManager/NetworkManager.conf I have included the line
> 
> dns=dnsmasq
> 
> and, accordingly, my /etc/resolv.conf looks like this:
> 
> # Generated by NetworkManager
> nameserver 127.0.0.1
> 
> dnsmasq seems to work: the last lines of output from command "dig
> fedoraproject.org" are
> 
> ;; Query time: 3498 msec
> ;; SERVER: 127.0.0.1#53(127.0.0.1)
> 
> As you can see from the above, I had to wait quite a while for the
> response. Running the same command immediately again gives
> 
> ;; Query time: 0 msec
> ;; SERVER: 127.0.0.1#53(127.0.0.1)
> 
> Which makes sense, since the address is now in the cache.
> 
> I have two problems:
> 
> 1.  The system seems to "forget" cached addresses, so that address
>     lookup for a frequently used address - say www.google.com - often
>     initiates a new search that again takes several seconds. This
>     happens very often, several times in a day. What is causing this?
>     Can it be related to DHCP lease time? (It is the only idea I have at
>     the moment.)

Does the system forget the cached addresses after a certain period of
time?  dnsmasq may be restarted periodically when events like lease
renewal happen, but that should not be very often.  This could be what
you're seeing, though you should see indications in the logs when this
happens, for example, I see:

Sep 12 15:35:38 dcbw NetworkManager[952]: <info> DNS: starting dnsmasq...
Sep 12 15:35:38 dcbw dnsmasq[1484]: started, version 2.52 cachesize 150
Sep 12 15:35:38 dcbw dnsmasq[1484]: compile time options: IPv6 GNU-getopt DBus no-I18N DHCP TFTP
Sep 12 15:35:38 dcbw dnsmasq[1484]: using nameserver 216.114.192.10#53
Sep 12 15:35:38 dcbw dnsmasq[1484]: using nameserver 4.2.2.2#53
Sep 12 15:35:38 dcbw dnsmasq[1484]: using nameserver 4.2.2.1#53

and it should print out the configuration it's using.  I used to have a
branch of NM that wouldn't change DNS configuration if it hadn't
actually change, but that was long ago, and we'd want to redo that work.
It would likely help with some of this jitter with shorter DHCP leases.

> 2. Earlier, before I found the dns=dnsmasq option of NetworkManager, I
>    had a different configuration that enabled the use of dnsmasq with
>    NetworkManager: adding 127.0.0.1 as one of the servers in
>    /etc/resolv.conf. Then I could see the name servers obtained from
>    DHCP. By testing these I noticed that they could be divided into two
>    classes: really slow servers (no idea why they are so slow), and
>    acceptably fast servers. The current lookup behaviour, having to wait
>    3.5 seconds for a response, suggests that when dnsmasq does not know
>    the address, it queries the slower servers. Is there some way for me
>    to control (filter) the set of servers obtained from DHCP?  How can I
>    see the servers that dnsmasq is currently using?

You should see the servers that dnsmasq is using in /var/log/messages
(for Fedora at least), as I've pasted above.  It's possible to tell
dhclient to 'override' the DNS servers using a custom dhclient config
file.  On Fedora, that's an interface-specific config file and you
essentially put the overrides into /etc/dhclient-eth0.conf.  The
overrides are standard dhclient configuration overrides which you can
google for.

> We have Windows computers in the same network; none of them have the
> same problem. Any ideas of why I have this Linux-specific problem?

One other thing to do is to run wireshark on the machine and capture
outgoing DNS requests and the replies that come back.  There have been
oddities with respect to IPv4 and IPv6 DNS with the glibc resolver in
the past, where broken DNS servers do not respond correctly to the IPv6
requests that may be sent out (even if IPv6 isnt' necessarily enabled).
That could be one source.  But more interesting would be how quickly
dnsmasq sends out the requests and how quickly the replies come back,
and if dnsmasq is round-robining the servers or something else.

Dan