Re: robots.txt for the software map
- From: Owen Taylor <otaylor redhat com>
- To: Jeff Waugh <jdub perkypants org>
- Cc: gnome-web-list gnome org, gnome-sysadmin gnome org
- Subject: Re: robots.txt for the software map
- Date: Tue, 09 Sep 2003 15:40:49 -0400
On Tue, 2003-09-09 at 13:10, Jeff Waugh wrote:
> <quote who="Owen Taylor">
> > Could someone add an appropriate robots.txt to keep search engines out of
> > the software map? The load on canvas is 15 or so, and the obvious
> > candidate is:
> Checked in.
As a further note here, even with this we were still having severe
load average problems on canvas, so I've blocked the software map
entirely for the moment ... I'm not sure it's the real culprit, but
it seems to be one of them.
Once the robots.txt has a chance to be picked up by all the
search engines (AskJeeves was also appearing in the logs from
today, maybe others) then we should try turning it back on.
If someone wants to replace the "Deny From" with a redirect to a
"temporarily out of service" page, that probably would be useful
in keeping down mail to webmaster gnome org
] [Thread Prev