Re: Thoughts on a new network repository mode: subtrees

[Colin Walters <walters verbum org>]

On Thu, Oct 16, 2014, at 12:25 AM, James Antill wrote:

>  Have you retested that recently, and did you use thttpd or
> apache-httpd? 

Whatever webserver dl.fedoraproject.org is running - probably Apache.

> When I did tests with it the latency/overhead of doing one
> connection per. file wasn't really noticeable (when we are doing 8
> parallel connections, and no fsync).

One connection, or one request?  Last time I looked trying to fetch from
webservers without KeepAlive was quite bad.

This is a complex topic - it really depends on the specific packages
that change and their content.  And the latency.

For example:

# rpm -ql kernel|grep \.ko | wc -l
6487

That's a lot of HTTP requests for every kernel change, multiplied by
each client.

>  It's been a month or three now, but I was testing with like 10 100MB
> files in one repo. and 10k-50k files in another repo.

You're saying there wasn't a substantial difference between those? 
That's interesting.  Though I suspect the difference would magnify a lot
with TCP latency.  Your reference to latency above was wall clock time,
right?  Or did you look at something like tricke/traffic shaping?

Maybe in the priority list, the highest here is strong support for local
mirrors.  We have that now, but there are still some gaps in that we
don't support traversing history yet.