Re: Pointing ftp.gnome.org somewhere (Re: master.gnome.org and FTP server restructurement)

[Mattias Wadenstein <maswan acc umu se>]

On 11 Mar 2001, Owen Taylor wrote:

>
> Mattias Wadenstein <maswan acc umu se> writes:
>
> > On 10 Mar 2001, Owen Taylor wrote:
> >
> > >
> > > Tomas Ogren <stric ing umu se> writes:
> > >
> > > > After discussing the matter with ftp.gnome.org == master.gnome.org and
> > > > anonftp being disabled with Martin on irc, we came to the conclusion
> > > > that a possible solution is to point ftp.gnome.org at ftp.acc.umu.se
> > > > (which is ftp.se.debian.org right now too) in Umeå, Sweden.
> > [snip]
> > >
> > > Hmmm, I'm pretty sure a majority of our traffic is currently
> > > from North America.
> >
> > A quick look at some statistics from before anonftp access was cut of
> > shows that about half the traffic (perhaps a little more than half) came
> > from North America. Of course, that is just estimates from the TLD
> > summary, unresloved IPs could be anywhere and com isn't 100% north
> > america. http://www.acc.umu.se/~maswan/util/tmp/xferstat/xferstats.html
> > is where I have the statisics up now, it will probably be gone in a couple
> > of weeks.
>
> Those aren't really reliable statistics. gnome.org was dropped down to
> 2mb at about the end of January, and on Feb 6 I dropped the user limit
> to 15, so you are looking at stats for:
>
>  - A small userlimit
>  - A very crammed connection

Ah, ok. That does give for a pretty different view of things.

> Also, I don't see how you are doing that computation - from your
> stats we have:

The computation was "com + net + edu seems to be about 20 gigs, so we say
about half of it NA". Not much math behind it.

>  com: 21%
>  unresolved: 17%
>  net: 16%
>  edu: 12%
>  ja: 6.0%
>  ca: 3.5
>
> Beore we get to any European domains. Making the assumption that
> com/net are all North American (not true, but not that far off,
> really) and that unresolved domains are equally distributed
> (my experience, they are tend to be heavily NA cable modems.)

At least swedish cable modems are just as bad when it comes to having
reverse resolving. There are just fewer of them.

> We get: NA fraction = (21 + 16 + 12 + 3.5 + 1.5 + 0.5 / (100 - 17))
>
>  66%
>
> And a the European fraction of about 23%.
>
> But that's pretty much neither here nor there.

Most traffic from NA, but not exclusively.

There is a chance that european users are more used to finding local
mirrors, but it is enough users to be even noticable is pure guessing.

> > [snip FTP performance test]
> > > So, quite decent transfer rates from North America.
> > >
> > > But from a good-network-citizen point of view, if most of our traffic
> > > is from North America, it seems that we probably should try to have
> > > ftp.gnome.org in North America.
> >
> > Well, it isn't that much traffic to begin with for us, or our network.
> > Assuming the xferlogs that I got from ftp.gnome.org are representative,
>
> They aren't, though unfortunately, they are the best we have.
> My guess is that on a ftp.gnome.org with plenty of bandwidth
> and a user limit of 100 we'd see about 10G of traffic a day
> steady-state with peaks of 20-30G or so around releases.
>
> But that's a pure blind guess.

Ok, then we wouldn't have a factor of 10 to spare before the load would be
noticable, I guess. But most of the load on the machine comes from rsync
connections, and I guess those are more relevant for the debian CDs than
gnome.

But most of this is blind guessing too. We did notice the load when potato
was released, but that was over 110G in one day. The days after (with
60-40G/day) felt just the same as when we only send out 10G (on a very
slow day).

But that is really noticing load on our server, things starting to get a
bit slower and so on. Not hitting the limit. And most of the load came
from the rsync connections, ftp and http are quite nice from that point of
view.

We are working to upgrade the hardware though, and will do that if it
starts getting slow. But from these guesses ftp.gnome.org won't do that
much of a difference for us.

> And don't ask me why so many people are downloading the source
> tarballs... (and often old, obsolete source tarballs as well!)

Probably the same reason why one host at an american university got a full
set of debian cd images from us, despite having one of the official
mirrors locally on campus. :)

To summarize: If a big release would happen at the same time as a big
debian release, or if the guesses would be off by more than a factor of
20, things might get crowded during peak hours and connections refused.

Other than that, I think we would make a good ftp.gnome.org, one that
takes running this service seriously. Until someone has a better place to
run it on at least.

But it all depends on how many other offers you have on providing
ftp.gnome.org, one probably has to make a choice between the existing
offers. There are plenty faster and topologically better placed sites out
there.

/Mattias Wadenstein - trying to make a point, just not sure what it is.