robots.txt for the software map
- From: Owen Taylor <otaylor redhat com>
- To: gnome-web-list gnome org, gnome-sysadmin gnome org
- Subject: robots.txt for the software map
- Date: Tue, 09 Sep 2003 13:04:51 -0400
Could someone add an appropriate robots.txt to keep search
engines out of the software map? The load on canvas is 15
or so, and the obvious candidate is:
64.68.82.176 - - [09/Sep/2003:13:03:11 -0400] "GET
/softwaremap/list/form_cat/164/discrim/164 HTTP/1.0" 200 48848 "-"
"Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
(The applist database is in mysql on canvas)
Thanks,
Owen
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]