Re: [Gimp-web] [Gimp-developer] Jenkins infrastructure (Was: Jenkins tutorial)

[Sam Gleske <sam mxracer gmail com>]

On Sat, Mar 1, 2014 at 10:33 PM, Sam Gleske <sam mxracer gmail com> wrote:

> @gimp-dev, @gimp-web
> My crawler has a stable release now.  I'd like to request permission to
> crawl www.gimp.org.  It will be restricted to that domain and have
> request delays implemented to minimize impact.

FYI most of the requests will only be HTTP method HEAD -
http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.4

It only requests headers and not the response body so it would actually be
less of an impact than most crawlers when doing the actual link testing.
For the crawling itself the response body is received so that it can find
other links.

SAM