Re: build.gnome.org

[Iago Toral Quiroga <itoral igalia com>]

Hi John!

El mié, 23-04-2008 a las 21:23 +0100, John Carr escribió:
> Hi All
> 
> Tonight I have good news and bad news...
> 
> http://live.gnome.org/JohnCarr/Jhbuildbot has been updated a little,
> included the setup I go through to get a working master and slave from
> scratch. Importantly, the version of the code this deploys needs only
> *one* open port on the master \o/

Good! :)

> My victory is short lived however. After talking to API, I started up
> more slaves and quickly fell victim to various errors from the proxy:

that feeling is familiar to me too... :)

> 2008/04/23 21:07 +0100 [jhbuildbot.master.SocksFactory] Could not
> accept new connection (EMFILE)
> 
> And the web server was also not feeling well:
> 
> 2008/04/23 21:08 +0100 [twisted.web.server.Site] Could not accept new
> connection (EMFILE)
> 
> A quick grep | wc -l of a netstat revealed i have 826 connections in
> the master process. For 2 slaves. I think this is more than my test PC
> is happy with. Because i'm proxying I have twice the number of
> connections (slave to proxy, proxy to master) and there are > 160
> projects... So I think I hit a limit. This will occur with the
> multiplexer and the multi-port solution too, but they should be able
> to cope with twice as many nodes as me (which is only 4 so far).

Right, these are too many connections, and I think it will be worse in
the future, as the gnome moduleset does not stop to grow, including more
modules.

And yes, this will affect the multi-port solution too at some moment,
though it will have twice the room you have for connections since it
does not need proxy connections, as you have pointed out.

> If this is true, I think the only way to scale to a decent number of
> nodes is to move entirely to a single connection. Unless buildbot
> upstream is working on stuff that might help us, the way forward might
> be to implement a custom ITransport so that the masters and slaves
> don't open TCP/IP connections directly and instead send their data to
> a shared connection. I have some thoughts on how this would work, and
> its only a slight step away from the current proxying system really.
> Slave side this is (relatively) easy, but i need to investigate more
> on the master side.

I see your point, and think it is worth a try. Also, it would be a idea
to ask directly to buildbot developers and see if they working on
something similar or have some suggestion about the way to go. I'll send
an email to buildbot-devel and ask.

> NOTE that i'm making nasty assumptions about why its exploding. If
> someone with a working multi-port master and some spare pc's could
> quickly see how many slaves we can currently handle it would be a big
> help..

I think we can arrange something here to do those tests, I'll let you
know about the results as soon as I have done it. However, I expect to
get the same results due to the same reason, since the number of
connections will have a limit at some point.

Iago

> Playing with the instructions I linked to would also help. It's still
> quite possible the SOCKS code is leaking somewhere...
> 
> John
> 
> On Wed, Apr 23, 2008 at 8:10 AM, Iago Toral Quiroga <itoral igalia com> wrote:
> > Hi John!
> >
> >  El mar, 22-04-2008 a las 22:13 +0100, John Carr escribió:
> >
> > > Hi All
> >  >
> >  > Just a little status update.
> >  >
> >  > I've been looking at using a SOCKS Server/Client to get around the
> >  > ports issue.
> >  [...]
> >
> > > I plan to do a test with the SOCKS5 code to see if my plan works, and
> >  > then there are 2 paths i can take.
> >  >
> >  > (1) Use SOCKS5. There is code for client and server, but it means that
> >  > the master will depend on proxy65. Note that my jhbuild moduleset
> >  > should make it trivial to deploy anyway. We could also look at
> >  > including the SOCKS part of the code directly, but i'm not sure of the
> >  > copyright stuff.
> >
> >  If this SOCKS5 client/server code works for us then we can check both
> >  options, but in principle, as you say, a jhbuild moduleset for the
> >  buildbot would make this easy to handle.
> >
> >
> >  > Unfortunately the SOCKS5 client has no copyright
> >  > notice on it so we would need to contact him before we started
> >  > deploying it
> >
> >  Yes, good point. Googling a little bit it looks like the person that
> >  submitted that ticket is Daniel Henninger <jadestorm nc rr com>. I will
> >  try to contact him.
> >
> >
> >  > (2) Use SOCKS4. We'd be using the server that ships with twisted, but
> >  > would have to roll our own client..
> >
> >  I think SOCKSv5 client/server is the best option if it works fine for
> >  us, but I'll give you the final word on this, for you are the one
> >  actually testing them :)
> >
> >  Iago
> >
> >
>