Re: Idea: daily packs

[Colin Walters <walters verbum org>]

On Mon, 2012-08-20 at 10:49 -0400, Owen Taylor wrote:

> My thought was that that information wasn't available upfront - we don't
> know the set of requested objects until we've done a large chunk of the
> fetching.

Right - for *metadata* fetches.  But there are separate pack files for
metadata and content, and the pull process happens in two separate
phases.  I was thinking of the content fetch stage, where we have all
the information.

At the moment the "pack" command has --metadata-only, but we could add
--content-only.

One thing to note here too: using --related increases the number of
metadata objects one needs to pull a *lot*.   I should maybe take that
off the default command list.  You only need it if you want to rebuild
individual components.

[Just did that]

> So on a re-pull you'd start off with one object - the root tree - which
> is in the new pack - but depending on where you go from there, you could
> either end up downloading zillions of objects, or just 
> / /usr /usr/bin /usr/bin/zenity.
>
> Is your thought that to start off fetching loose objects and at some
> point decide to switch over to fetching the pack? It looks like the
> trees are fetched depth-first before objects, so the switchover could be
> pretty quick.

Yeah, for metadata we could do something like that.  It'd actually
dovetail with asynchronous metadata fetches, because we'd get to the
point where we realize which metadata packfiles we should fetch faster.

So...I'll experiment with trying the 50% heuristic for content by
tonight - I suspect doing that and dropping --related from the default
command will give us a lot of the win of just wget on a .tar.gz.