Re: Jenkins-based Continuous

[Vadim Rutkovsky <vrutkovs redhat com>]

Hey,

On Fri, Feb 27, 2015 at 2:35 PM, Colin Walters <walters verbum org> 
wrote:
> You're doing some great work here, and I don't want it to be stalled. 
>  I've been thinking about this in the background, and here's my 
> current conclusion:
>
> - I'm OK with using Jenkins as a control framework for *tests*, after 
> the build
> - I do not want actual *builds* (i.e. git -> tree) to involve Jenkins

> To implement this, we should run the tests and builds as separate 
> uids on the server.  Splitting up the build and test environment and 
> having them be separate "processes" (containers?) will allow us a lot 
> more agility on the test side too.  We can push updates to the test 
> suite faster.

Agreed on that, we should have build.gnome.org separated from tests - 
current implementation is rather useless for actual QA.

Since that I've come up with a different idea:
- Use OpenStack as a libvirt, which will start instances
- Use stable image, which is proven to be bootable and upgradable
- Connect to an instance via ssh to be able to run custom commands / 
scripts

So now I'm working on a POC of a system, which uses a different 
workflow:
 - A new instance based on stable image is started
 - A specific IP is assigned to an instance
 - Jenkins node runs 'ostree admin upgrade' via ssh, uses openstack 
API to reboot the instance
 - Jenkins node starts smoketests/integrationtests/applications tests 
via ssh
 - After the tests are complete, Jenkins can run arbitrary scripts on 
the instance:
   - collect coredumps and produce a gdb traceback
   - make a screenshot
   - etc.

Jenkins can support a lot of worker nodes, so we can tweak it to have 
several instances running in parallel - for instance, using different 
video-drivers, various CPU/RAM amounts etc.

Also, the only thing we refer in our tests is IP address, so we might 
experiment with bare-metal installations, based on Owen's work on 
perf.gnome.org.

Currently the system is very much WIP, I'll post the results on the 
list later on.

> This model addresses my concerns with security - if the Jenkins web 
> frontend is compromised[1], it doesn't allow an attacker to insert 
> malicious binaries.  It also clearly avoids a dependency on Jenkins 
> for developers doing builds locally.

I'd prefer to be even more radical about this - hide jenkins port from 
outer world and make a custom app, which will allow no modifications at 
all - Jenkins API is pretty powerful. This will hide unnecessary 
details from the builds and will help us to focus on test results.

Thanks,
 Vadim