Re: Looping GET to http://gnome.org, high load

[Owen Taylor <otaylor redhat com>]

On Sun, 2006-09-03 at 09:21 -0400, Owen Taylor wrote:

>  * Stracing a few httpd processes, and watching when they block unexpectedly, it
>    seems to be almost always when writing to the httpd log files.  
> 
>    (Mostly the art.gnome.org log, but I think that's just because it's the most
>    heavily hit site. I've seen it on other log files as well.)
> 
>    This is strange to me ... why would appending a few hundred bytes to a log
>    file block? I don't think we're handling more than maybe 10 web requests / sec
>    total.
> 
> Right now not clear to me what's going on or how to improve it.

Some further investigation indicates that the basic problem is that we
are doing a *ton* of IO writes; a typical vmstat line looks like:


 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  5   6468  17484 259548 883088    0    0   241  5102 9407  9938 35 16  2 47
                                                   ^^^^

Maybe I'm misreading that, but it seems to say to me that we are writing about
5000 blocks = 20M a second to disk.

The system has decent disks (2 10k SCSI drives, raid-1), but that's really pushing
the limits.

What I've tried doing as an experiment is moving the CVS temporary directory to
tmpfs.

  mount -ttmpfs -osize=256M none /var/cvs-tmp

And adding -T /var/cvs-tmp to /usr/local/bin/run-cvs-pserver.sh.

Initial indications are good ... there are still about 5 CVS processes running 
started before the switch using /tmp, but 'blocks-out' has dropped to about
500-1500, and the load average is around 4 rather than over 120.

I'll watch it some more over the next few hours, and if it looks good, make these
changes permanent. 

 - Add the mount to fstab

 - Maybe commit the run-cvs-pserver.sh changes to admin-cvs? (What's the situation
   there .. the comment indicates its checked in and identical between window
   and container, but it's different...)

One question is obviously the sizing of the temporary directory. 256M is not going
to be big enough for a checkout of the whole repository, which happens (though
we'd rather prevent it...), so someone doing that might DOS other checkouts. Not 
sure how to deal with that. We probably could go a bit bigger than 256, and count
on swap.

						- Owen