Re: Use git 1.6.6+ when using http

[Olav Vitters <olav vitters nl>]

On Thu, Feb 17, 2011 at 09:41:47AM -0500, Behdad Esfahbod wrote:
> On 02/17/11 09:33, Olav Vitters wrote:
> > 62.195.108.47 - - [17/Feb/2011:14:24:15 +0000] "POST /browse/gnome-terminal/git-upload-pack HTTP/1.1" 200 12218190 "-" "git/1.7.4"
> 
> What's the POST about?

No idea :) I know the first hit is special, and then git knows the
server supports the efficient format. I assume/guess within the POST the
git client tells git.gnome.org what stuff to send.

> So, if I understand it correctly, the refs? URLs are passed from cgit
> to git now?

More like s/from/instead of/. I'm using the configuration adapted from 
http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html

More or less the following:
| SetEnv GIT_PROJECT_ROOT /git
| 
| AliasMatch ^/browse/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$          /git/$1
| AliasMatch ^/browse/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /git/$1
| ScriptAliasMatch \
|         "(?x)^/git/(.*/(HEAD | \
|                         info/refs | \
|                         objects/info/[^/]+ | \
|                         git-(upload|receive)-pack))$" \
|         /usr/bin/git-http-backend/$1
| ScriptAlias /git/ /var/www/cgit/cgit.cgi

Or in other words:
Apache decides between:
 - Looking directly on the file system (1st 2 aliasematches)
 - git-http-backend (scriptaliasmatch)
 - cgit (scriptalias)

One IP address was generating 70.000 from last Sun to yesterday. Each
hit would start another cgi process (cgit). That'll reduce to just a few
hits now.

There are still other problems: Spammers (bots scaping the website,
noticeable by lack of robots.txt access), directly linking 100+ of
different images to cgit on a popular website (one of our artists),
Google (allowed, but it really really likes git.gnome.org).
I'll have to do a bit more monitoring.

Top hits since last Suni by IP:
66.249.67.242   315917  Google
77.56.172.56    288777  Java bot (blocked it)
195.176.179.209 144780  Java bot (blocked it)
194.228.254.118 72622   git/1.7.2.3
222.248.226.23  70531   Icon lover/bot (unsure)
137.129.13.90   65588   git/1.7.4


Total hits: 2.594.267 (since last Sun)
Artist website: 592.134
Unwanted bots (rough estimate): 433.031
Google: 315.917
Git client: 238.400 (only counted just now, seemed a much bigger problem)


Perhaps everything at once (bots/artist/git) was a bit too much :P

-- 
Regards,
Olav