Re: build.gnome.org

["John Carr" <john carr unrouted co uk>]

Hi All

Just a little status update.

I've been looking at using a SOCKS Server/Client to get around the
ports issue. Currently native twisted supports SOCKS4 from the server
end, and provides no client protocol at all. The SOCKS4 implementation
can be hooked, so we can easily restrict where it proxies to (only
allow incoming connections to connect to our masters and nothing
else). But it does mean that we need to implement a twisted SOCKS4
client ourselves...

Looking further, I found a SOCKS5 client:
http://twistedmatrix.com/trac/ticket/1330

I also managed to find this:
http://code.google.com/p/proxy65/

This implements a SOCKS5 server.

I plan to do a test with the SOCKS5 code to see if my plan works, and
then there are 2 paths i can take.

(1) Use SOCKS5. There is code for client and server, but it means that
the master will depend on proxy65. Note that my jhbuild moduleset
should make it trivial to deploy anyway. We could also look at
including the SOCKS part of the code directly, but i'm not sure of the
copyright stuff. Unfortunately the SOCKS5 client has no copyright
notice on it so we would need to contact him before we started
deploying it

(2) Use SOCKS4. We'd be using the server that ships with twisted, but
would have to roll our own client..

John

On Tue, Apr 22, 2008 at 9:00 AM, Iago Toral Quiroga <itoral igalia com> wrote:
> Hi John!
>
>  El lun, 21-04-2008 a las 16:40 +0100, John Carr escribió:
>
> > Hi all
>  >
>  > What is the current status of the jhbuild buildbot project?
>  > What's the
>  > best way to get involved?
>
>  It is good to know you want to help, there're many things you can help
>  with! your proposal to simplify slave deployment is just great, I
>  encourage you to keep working on that, but if you want to work on
>  something else , here are some examples:
>
>  1.- current main page is too simple, I'd like to add  more info there,
>  like:
>    - What moduleset we are building
>    - Passed/Failed modules statistics per lave
>    - etc
>
>  2. Integration with repositories so we can check what changes have been
>  done between two builds of a project.
>
>  3. Apply the gnome web style to the view:
>  http://bugzilla.gnome.org/show_bug.cgi?id=485125
>
>  4. Reduce the number of ports required to communicate the master with a
>  slave.
>
>  5. Simplify builds for other people. For example, if could make our
>  jhbuildrc file go public others can use them (this should save some
>  problems to some).
>
>  6.- Investigate a way to provide packages for distributions as part of
>  the build process (deb, rpm, etc)
>
>  7.- Check failed builds and reports them as bugs to appropriate modules
>  in bugzilla or fix them (if that happens due to slave
>  missconfiguration).
>
>  Of course, these are only some examples, and you can also suggest
>  anything you find interesting!
>
>
>  >  I'd been following it for a while but
>  > haven't heard much about it recently. I noticed the sid slave has come
>  > back online and thought it was time to come out from under a rock...
>
>  you are right, we have been very silent for quite a while, more than we
>  would have liked to, really :(
>
>  The main reason for this was that, as you may know, we had a problem to
>  connect several slaves to the master instance due to the need of opening
>  many ports on the master's firewall side... So, we have been quite busy
>  trying to figure out a solution for this during all this time. The
>  problem is that we did not have experience with Twisted and it took some
>  time before we could start doing something productive... In the end we
>  managed to find a solution and implement it, but unfortunately, it does
>  not work fully well :(, it is quite weird because it works fine for a
>  while and from time to time we get a weird error from low level twisted
>  and we got a bit stuck there.
>
>  As you said, we had been quite silent for too much time, so we decided
>  to change the strategy so we could work on stuff and keep things moving
>  forwards, thus, we decided to move the master back to Igalia
>  temporarily, so we can open these ports and start accepting new slaves.
>  I completed this with Olav's help on Friday, and now we have a
>  Debian-Sid slave running at Igalia too.
>
>  I was going to post something in planet.gnome.org and this mailing list
>  announcing that we are open to accept new slaves during this week, but
>  consider this email as a formal announcement :)
>
>  Right now, I'm also checking out the gnome-mobile module set and we plan
>  to provide a buildbot for that software stack too.
>
>
>  > My main aim has been to try and reduce slave deployment to as close to
>  > this as possible:
>  >
>  > jhbuild build jhbuildbot-master
>  > <tweak-config> (as little as possible...)
>  > jhbuild run "slave start"
>
>  That would be really great! We have a lot to improve in this regard at
>  the moment, since the current deployment guide includes many manual
>  steps.
>
>
>  > On the way I switched to using buildbot 0.7.7, added some autotools
>  > foo to the scripts so jhbuild can work its magic on them, moved lots
>  > of code to site-packages/jhbuildbot and other piddly things. I'm eager
>  > to finish migrating to buildbot 0.7.7 (for now, i use html.py from
>  > buildbot 0.7.5 so that the web frontend still mostly works) and to
>  > look at not needing to open a port for each project that is added to
>  > the setup. But obviously I don't want to duplicate effort...
>
>  Please, keep us updated about your work, it'd be a great contribution if
>  you manage to simplify the slave deployment.
>
>
>  > I've got some basic notes here:
>  > http://live.gnome.org/JohnCarr/Jhbuildbot
>  >
>  > If anyone wants to talk about what i've been doing, I'm Jc2k in #build-brigade.
>
>  You're doing a great job. If you manage to get it done let me know and
>  I'll check it out, it'd be a great addition.
>
>  Also, bug me anytime on the IRC if you need help, want to know more
>  about the buildbrigade...
>
>  Cheers,
>  Iago
>
>