On 01/03/14 11:27 AM, Daniel Micay wrote:
The blogs.gnome.org uses a strange robots.txt <http://blogs.gnome.org/robots.txt> permitting only Googlebot to crawl it. This prevents using archive.org on the subdomain and cripples other search engines. It should simply use a wildcard for the first User-agent field with the second case discarded. Thanks!
It looks like this has already been fixed. That could be the best bug report response time I've ever seen. ;)
Attachment:
signature.asc
Description: OpenPGP digital signature