Re: build.gnome.org

["John Carr" <john carr unrouted co uk>]

Hi All

Tonight I have good news and bad news...

http://live.gnome.org/JohnCarr/Jhbuildbot has been updated a little,
included the setup I go through to get a working master and slave from
scratch. Importantly, the version of the code this deploys needs only
*one* open port on the master \o/

My victory is short lived however. After talking to API, I started up
more slaves and quickly fell victim to various errors from the proxy:

2008/04/23 21:07 +0100 [jhbuildbot.master.SocksFactory] Could not
accept new connection (EMFILE)

And the web server was also not feeling well:

2008/04/23 21:08 +0100 [twisted.web.server.Site] Could not accept new
connection (EMFILE)

A quick grep | wc -l of a netstat revealed i have 826 connections in
the master process. For 2 slaves. I think this is more than my test PC
is happy with. Because i'm proxying I have twice the number of
connections (slave to proxy, proxy to master) and there are > 160
projects... So I think I hit a limit. This will occur with the
multiplexer and the multi-port solution too, but they should be able
to cope with twice as many nodes as me (which is only 4 so far).

If this is true, I think the only way to scale to a decent number of
nodes is to move entirely to a single connection. Unless buildbot
upstream is working on stuff that might help us, the way forward might
be to implement a custom ITransport so that the masters and slaves
don't open TCP/IP connections directly and instead send their data to
a shared connection. I have some thoughts on how this would work, and
its only a slight step away from the current proxying system really.
Slave side this is (relatively) easy, but i need to investigate more
on the master side.

NOTE that i'm making nasty assumptions about why its exploding. If
someone with a working multi-port master and some spare pc's could
quickly see how many slaves we can currently handle it would be a big
help..

Playing with the instructions I linked to would also help. It's still
quite possible the SOCKS code is leaking somewhere...

John

On Wed, Apr 23, 2008 at 8:10 AM, Iago Toral Quiroga <itoral igalia com> wrote:
> Hi John!
>
>  El mar, 22-04-2008 a las 22:13 +0100, John Carr escribió:
>
> > Hi All
>  >
>  > Just a little status update.
>  >
>  > I've been looking at using a SOCKS Server/Client to get around the
>  > ports issue.
>  [...]
>
> > I plan to do a test with the SOCKS5 code to see if my plan works, and
>  > then there are 2 paths i can take.
>  >
>  > (1) Use SOCKS5. There is code for client and server, but it means that
>  > the master will depend on proxy65. Note that my jhbuild moduleset
>  > should make it trivial to deploy anyway. We could also look at
>  > including the SOCKS part of the code directly, but i'm not sure of the
>  > copyright stuff.
>
>  If this SOCKS5 client/server code works for us then we can check both
>  options, but in principle, as you say, a jhbuild moduleset for the
>  buildbot would make this easy to handle.
>
>
>  > Unfortunately the SOCKS5 client has no copyright
>  > notice on it so we would need to contact him before we started
>  > deploying it
>
>  Yes, good point. Googling a little bit it looks like the person that
>  submitted that ticket is Daniel Henninger <jadestorm nc rr com>. I will
>  try to contact him.
>
>
>  > (2) Use SOCKS4. We'd be using the server that ships with twisted, but
>  > would have to roll our own client..
>
>  I think SOCKSv5 client/server is the best option if it works fine for
>  us, but I'll give you the final word on this, for you are the one
>  actually testing them :)
>
>  Iago
>
>