Re: Receiving objects: 44% (121381/272167), 124.05 MiB | 78 KiB/s

[Owen Taylor <otaylor redhat com>]

On Mon, 2009-05-04 at 15:59 -0400, Reid Thompson wrote:
> On Mon, 2009-05-04 at 15:46 -0400, Owen Taylor wrote:
> > On Mon, 2009-05-04 at 15:24 -0400, Reid Thompson wrote:
> > > On Mon, 2009-05-04 at 14:56 -0400, Behdad Esfahbod wrote:
> > > > On 05/04/2009 02:52 PM, Reid Thompson wrote:
> > > > > Is this the 'to be expected' transmission rate for git?  Or is there
> > > > > something abnormal about today?
> > > > >
> > > > > Receiving objects:  44% (121381/272167), 124.05 MiB | 78 KiB/s
> > > > 
> > > > Be more specific?  What's wrong with 78 KiB/s exactly?  I've seen that speed 
> > > > from my home network, and I've see as high as 1 MiB/s from other networks 
> > > > (including a coffeeshop network).
> > > > 
> > > > behdad
> > > 
> > > at that rate, downloading everything for evolution (evolution
> > > evolution-exchange evolution-webcal evolution-data-server glib libsoup
> > > libgweather gvfs gtkhtml etc) is going to take more time to download
> > > than that it took to download, compile and install using svn.  hours ??
> > > longer... just trying to find out if that is going to be the norm going
> > > forward or not.
> > > 
> > > evolution took 42 minutes to download, evolution-data-server won't
> > > download...
> > 
> > Obviously connectivity to git.gnome.org and svn.gnome.org may be quite
> > different - they are hosted in different places (svn.gnome.org in
> > Europe, git.gnome.org in the US)
> > 
> > But in every case we've looked at, bottlenecks checking out from
> > git.gnome.org:
> > 
> >  A) On someone's local connection or network provider
> >  B) Somewhere in hand-offs between major networks
> > 
> > With no network constraints, you should see download rates of
> > 1 or 2 MiB/s. (I consistently get that from the Red Hat office in
> > Massachusetts, somewhat slower from home - but still 10x what you are
> > reporting.)
> > 
> > Of course, checking out everything is a one-time activity so almost by
> > definition it won't be the same in the future....
> > 
> > - Owen
> > 
> > 
> 
> location is Raleigh, NC.  Pipe limitation to internet is 10 Mb, 
> 
> svn glib is 1m26s minutes, git is 3m44s...
> 
> git glib
> $ time git clone git://git.gnome.org/glib
> Initialized empty Git repository
> in /media/disk-1/home/evo-src/git/glib/.git/
> remote: Counting objects: 66163, done.
> remote: Compressing objects: 100% (9910/9910), done.
> remote: Total 66163 (delta 56687), reused 65461 (delta 56149)
> Receiving objects: 100% (66163/66163), 18.88 MiB | 80 KiB/s, done.
> Resolving deltas: 100% (56687/56687), done.
> 
> real	3m44.196s
> user	0m8.326s
> sys	0m0.859s

>From a machine in Raleigh:

$ time git clone git://git.gnome.org/glib
Initialized empty Git repository in /home/devel/otaylor/glib/.git/
remote: Counting objects: 66163, done.
remote: Compressing objects: 100% (9910/9910), done.
remote: Total 66163 (delta 56687), reused 65461 (delta 56149)
Receiving objects: 100% (66163/66163), 18.88 MiB | 4194 KiB/s, done.
Resolving deltas: 100% (56687/56687), done.
Checking out files: 100% (1102/1102), done.

real	0m20.404s
user	0m6.062s
sys	0m1.619s
[file.rdu.redhat.com] [04:17:59 PM]

Now, admittedly, that isn't a completely fair comparison since (if I
recall correctly) there is dedicated bandwidth between our RDU offices
and our datacenter in Phoenix (where git.gnome.org is located). But it
does show that git.gnome.org is happy to spit things out that fast.

And I've seen, and seen people report, quite fast speeds from checkouts
not on the Red Hat corporate network.

So, basically I'd look at RoadRunner for your checkout speeds. 1.5
minutes to checkout glib from svn is not fast either. (< 20 seconds
here).

I guess in summary I'd say:

 - Checking out from git does more network traffic than checking
   out from svn. Because it's downloading the complete history for
   all time. But despite that it's a huge win on a slow network link
   because you *don't* have to go to the network to look at history.

   Repeatedly checking out the same modules from scratch is not
   recommended and should not be necessary.

 - We've observed no bottlenecks on git.gnome.org itself at providing
   downloads at very fast speeds (large modules in just a few seconds);
   if your network connectivity to git.gnome.org is sufficiently good,
   downloading history will not be a big component of the initial
   checkout time; it will take more time to "resolve deltas".

 - git.gnome.org is at a collocation facility with lots of bandwidth
   and connectivity. Despite that, some people have reported slow
   download rates even when they have good speeds doing other
   things on the internet.
 
   Not being much of an expert on wide-area networking, I don't
   feel qualified to say whether that is a result of bandwidth
   throttling by their ISP, problems handing off packets between
   networks, or what. 

Does that speak to your concerns?

- Owen