Re: robots.txt for the software map

[Owen Taylor <otaylor redhat com>]

On Tue, 2003-09-09 at 15:40, Owen Taylor wrote:
> On Tue, 2003-09-09 at 13:10, Jeff Waugh wrote:
> > <quote who="Owen Taylor">
> > 
> > > Could someone add an appropriate robots.txt to keep search engines out of
> > > the software map? The load on canvas is 15 or so, and the obvious
> > > candidate is:
> > 
> > Checked in.
> 
> As a further note here, even with this we were still having severe 
> load average problems on canvas, so I've blocked the software map
> entirely for the moment ... I'm not sure it's the real culprit, but
> it seems to be one of them.
> 
> Once the robots.txt has a chance to be picked up by all the
> search engines (AskJeeves was also appearing in the logs from
> today, maybe others) then we should try turning it back on.
> 
> If someone wants to replace the "Deny From" with a redirect to a 
> "temporarily out of service" page, that probably would be useful
> in keeping down mail to webmaster gnome org 

Having found a huge CVS module that was contributing to the load, I
renabled the software map. We'll see how it goes.

Regards,
						Owen