blogs.gnome.org blocks indexing by non-Google search engines and archive.org



The blogs.gnome.org uses a strange robots.txt
<http://blogs.gnome.org/robots.txt> permitting only Googlebot to crawl
it. This prevents using archive.org on the subdomain and cripples other
search engines. It should simply use a wildcard for the first User-agent
field with the second case discarded.

Thanks!

Attachment: signature.asc
Description: OpenPGP digital signature



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]