Re: gar.gnome.mk download location suggestion

[Karsten Bräckelmann <guenther rudersport de>]

On Thu, 2007-10-18 at 12:32 +0000, Peter wrote:
> I just tested a download (from the US) using a different location:

Peter, thanks for caring, and trying to improve GARNOME. Much
appreciated. :-)


> gar.gnome.mk:
> MASTER_SITES += ftp://ftp.gnome.org/mirror/gnome.org/sources/....
> instead of http://ftp.gnome.org/pub/GNOME/sources/...
> 
> to me, it seemed to be much faster. If I'm not mistaken, the mirror will
> resolve to a free server in a circular queue? Performance has overall
> been much faster by a factor of 2.5 to 3x.

Yes, f.g.o does resolve to different IPs. I believe I once was told it
is round robin DNS (circular), though I am not entirely sure. Also,
AFAIK [1] there is quite a large backend farm serving the actual files
from disks, which is hidden to the outside observer.

Anyway, since I was curious, I just did a quick test. Both http and ftp
quickly saturated the line (tested with a pretty fat [2] pipe ;)  after
initially starting off "moderate" (TCP slow start, congestion control,
inherent in the protocol). Granted, just a quick test, with a couple
iterations for a large-ish file (actually part of the GNOME Platform
Suite). I merely got a meager ~5% difference in total time in favor of
ftp, which is not statistically significant.

Trying the same with a small tarball, http outperformed ftp by 55%.
However, even though most tarballs aren't particularly large -- given
the small sample set and the rather small total time, neither one proves
the case.

Since your results seem to be the opposite, I suspect the cause to be
part of your local network and ISP.


That all said, the process of uploading a new tarball and syncing the
mirrors actually ends with the location of the just published tarball,
and these lines: [3]

  It is important to retain the trailing slash for compatibility with
  broken http clients, and to use http as it is less taxing on the server.

The gods of the mirrors have spoken. ;)


Peter, if it really makes a difference for you, you are free to tweak
gar.gnome.mk to your needs. However, I will not change the default for
obvious reasons.

Since you tend to rebuild frequently, you should definitely keep a
garchive. I assume you already do. The underlying root cause, that one
method gets so much more share of the available bandwidth though,
definitely is neither inherent to the protocol, nor the mirrors. I
believe it to be due to limitations your ISP enforces.

As you seem to be interested in speeding up the process, let me tell you
that I am currently hacking on a particular issue [4], that, once fixed,
potentially can speed up the entire build noticeably -- even with a
fully populated garchive.

Again, thanks for investigating and sharing with us, Peter. :)

  guenther


[1] from various discussions with the sysadmins
[2] in comparison to advanced users and small offices, not university
    environment
[3] a fact I found out just recently
[4] won't tell more, so you don't bug me ;-)

-- 
char *t="\10pse\0r\0dtu\0  ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}