Re: robots.txt for the software map

[Owen Taylor <otaylor redhat com>]

On Tue, 2003-09-09 at 13:10, Jeff Waugh wrote:
> <quote who="Owen Taylor">
> 
> > Could someone add an appropriate robots.txt to keep search engines out of
> > the software map? The load on canvas is 15 or so, and the obvious
> > candidate is:
> 
> Checked in.

As a further note here, even with this we were still having severe 
load average problems on canvas, so I've blocked the software map
entirely for the moment ... I'm not sure it's the real culprit, but
it seems to be one of them.

Once the robots.txt has a chance to be picked up by all the
search engines (AskJeeves was also appearing in the logs from
today, maybe others) then we should try turning it back on.

If someone wants to replace the "Deny From" with a redirect to a 
"temporarily out of service" page, that probably would be useful
in keeping down mail to webmaster gnome org 

Regards,
						Owen