- From: "John Carr" <john carr unrouted co uk>
- To: "Iago Toral Quiroga" <itoral igalia com>
- Cc: build-brigade-list gnome org
- Subject: Re: build.gnome.org
- Date: Wed, 23 Apr 2008 21:23:41 +0100
Tonight I have good news and bad news...
http://live.gnome.org/JohnCarr/Jhbuildbot has been updated a little,
included the setup I go through to get a working master and slave from
scratch. Importantly, the version of the code this deploys needs only
*one* open port on the master \o/
My victory is short lived however. After talking to API, I started up
more slaves and quickly fell victim to various errors from the proxy:
2008/04/23 21:07 +0100 [jhbuildbot.master.SocksFactory] Could not
accept new connection (EMFILE)
And the web server was also not feeling well:
2008/04/23 21:08 +0100 [twisted.web.server.Site] Could not accept new
A quick grep | wc -l of a netstat revealed i have 826 connections in
the master process. For 2 slaves. I think this is more than my test PC
is happy with. Because i'm proxying I have twice the number of
connections (slave to proxy, proxy to master) and there are > 160
projects... So I think I hit a limit. This will occur with the
multiplexer and the multi-port solution too, but they should be able
to cope with twice as many nodes as me (which is only 4 so far).
If this is true, I think the only way to scale to a decent number of
nodes is to move entirely to a single connection. Unless buildbot
upstream is working on stuff that might help us, the way forward might
be to implement a custom ITransport so that the masters and slaves
don't open TCP/IP connections directly and instead send their data to
a shared connection. I have some thoughts on how this would work, and
its only a slight step away from the current proxying system really.
Slave side this is (relatively) easy, but i need to investigate more
on the master side.
NOTE that i'm making nasty assumptions about why its exploding. If
someone with a working multi-port master and some spare pc's could
quickly see how many slaves we can currently handle it would be a big
Playing with the instructions I linked to would also help. It's still
quite possible the SOCKS code is leaking somewhere...
On Wed, Apr 23, 2008 at 8:10 AM, Iago Toral Quiroga <itoral igalia com> wrote:
> Hi John!
> El mar, 22-04-2008 a las 22:13 +0100, John Carr escribió:
> > Hi All
> > Just a little status update.
> > I've been looking at using a SOCKS Server/Client to get around the
> > ports issue.
> > I plan to do a test with the SOCKS5 code to see if my plan works, and
> > then there are 2 paths i can take.
> > (1) Use SOCKS5. There is code for client and server, but it means that
> > the master will depend on proxy65. Note that my jhbuild moduleset
> > should make it trivial to deploy anyway. We could also look at
> > including the SOCKS part of the code directly, but i'm not sure of the
> > copyright stuff.
> If this SOCKS5 client/server code works for us then we can check both
> options, but in principle, as you say, a jhbuild moduleset for the
> buildbot would make this easy to handle.
> > Unfortunately the SOCKS5 client has no copyright
> > notice on it so we would need to contact him before we started
> > deploying it
> Yes, good point. Googling a little bit it looks like the person that
> submitted that ticket is Daniel Henninger <jadestorm nc rr com>. I will
> try to contact him.
> > (2) Use SOCKS4. We'd be using the server that ships with twisted, but
> > would have to roll our own client..
> I think SOCKSv5 client/server is the best option if it works fine for
> us, but I'll give you the final word on this, for you are the one
> actually testing them :)
] [Thread Prev