Integration between buildservice and gnome.build

[API <apinheiro igalia com>]

Hello

sometime ago Rodrigo Moya proposed a kind of integration between the build-brigade
and the Novell BuildService [1]. The idea is that the build-brigade compile a
module, and if it is successful, use this code as a target to upload the
module to the build service, to have an updated SuSE package.

The idea is add a new step (currently we have 5: update, build, check, coverage
and clean), and execute a script. It is clear that it is not needed that all
the slaves have this additional step, so we could check how to configure it
as a conditional step or simply use a modified slave code in some concrete
machine.

This script basically takes the code, creates a tarball with this code, and
then use some osc in order to apply some SuSE patches, and upload the package.

Rodrigo Moya and Michael Wolf created a provisional script, and Iago and me
were reviewing it, so we thought that we should share it with the list. You can
check it here:

git clone http://www.gnome.org/~rodrigo/git/osc-plugins.git

There is a directory with the build-brigade stuff.

During my review I found some errors and added additional information to
the verbose mode (to debugging purpose), I attach a patch, so if you agree with
the changes, please Rodrigo commit it.

At this moment it have some problems, and in fact I was not able to get any
module uploaded. Some problems I found were:

  * In order to create the tarball it uses make dist
  * I'm not sure that all the gnome modules support correctly make dist
  * make dist require to have enabled gtk-doc in some modules, so it is required
    to do a ./autogen.sh --enable-gtk-doc
  * This slow the process and making my tests, in some modules (ie libglade) it
    fails with this error (although not all the times):
      configure.in:165: required file `doc/Makefile.in' not found
      configure.in:165: required file `doc/version.xml.in' not found
      configure.in:165: required file `tests/Makefile.in' not found
      Makefile.am:1: required directory ./tests does not exist
    But all theses files and directories exists
  * So, probably could be a good idea find another way to create the tarball with
    the source code.
  * Other problem is related with the fact that there are different name
    convention for the gnome moduels between jhbuild and the build service, so a
    name mapping is required (see packagemappings.py file). Currently the mapping
    list is incomplete.

In you want to do some tests, you'll need to do something like that (with the
proper local directories):

   ./builddir-prepare -p GNOME:Factory  -m  libglade -b ~/osc -v
   ./builddir-prepare -p GNOME:snapshots:unpatched  -m  libglade -b ~/osc -v
   ./upload-tarball -p GNOME:snapshots:unpatched  -m  libglade -s ~/gnome/work/src -b ~/osc -v

(About that, one idea could be include the builddir-prepare on the upload-tarball)

You'll need a account on the build service. There are currently one created for
build-brigade, to make the tests. If anyone have interest, I can send the login
and password.

Sorry for this über mail

BR

[1] http://en.opensuse.org/Build_Service/CLI
[2] http://en.opensuse.org/Build_Service/CLI#osc.2C_the_Python_command_line_client

===
API (apinheiro igalia com)

diff --git a/gnome-build-brigade/builddir-prepare b/gnome-build-brigade/builddir-prepare
index f39413c..6e60a41 100755
--- a/gnome-build-brigade/builddir-prepare
+++ b/gnome-build-brigade/builddir-prepare
@@ -62,8 +62,11 @@ def main():
     cmd = "osc checkout %s %s" % \
         (project, packagemappings.suse_package_name(package))
 
+    if optz.verbose:
+        print "About to execute %s" % cmd
     status, output = commands.getstatusoutput(cmd)
     if status != 0:
+        print "Fail\n %s" % output
         raise
 
     pushd.popd()
diff --git a/gnome-build-brigade/packagesnapshotter/osc.py b/gnome-build-brigade/packagesnapshotter/osc.py
index ba19b5f..b276638 100644
--- a/gnome-build-brigade/packagesnapshotter/osc.py
+++ b/gnome-build-brigade/packagesnapshotter/osc.py
@@ -3,19 +3,27 @@
 # misc osc commands.  Probably should use osc's api directly, of
 # course.  Maybe later.
 
+import commands
 import optz
 import packagemappings
 import pushd
+import oscpluginsgnome.pacversion
 
 def get_last_version(project, package):
 
-    dir = "%s/%s/%s" % (optz.builddir, project, package)
+    dir = "%s/%s/%s" % (optz.builddir, project, packagemappings.suse_package_name(package))
     pushd.pushd(dir)
 
     cmd = "osc pacversion %s %s" % \
         (project, packagemappings.suse_package_name(package))
 
+    if optz.verbose:
+        print "About to execute: \"%s\"" % cmd
+
     status, output = commands.getstatusoutput(cmd)
+    if optz.verbose:
+        print output
+
     if status != 0:
         raise
     
@@ -32,8 +40,13 @@ def checkout():
 def add(package):
     cmd = "osc add %s" % package
 
+    if optz.verbose:
+        print "About to execute %s" % cmd
+
     status, output = commands.getstatusoutput(cmd)
     if status != 0:
+        print "Error executing %s" % cmd
+        print "Output\n %s" % output
         raise
     pass
 
@@ -44,5 +57,7 @@ def commit(message):
     status, output = commands.getstatusoutput(cmd)
 
     if status != 0:
+        print "Error executing %s" % cmd
+        print "%s" % output
         raise
     pass
diff --git a/gnome-build-brigade/packagesnapshotter/svn.py b/gnome-build-brigade/packagesnapshotter/svn.py
index 29932f4..5c7e2b4 100644
--- a/gnome-build-brigade/packagesnapshotter/svn.py
+++ b/gnome-build-brigade/packagesnapshotter/svn.py
@@ -12,7 +12,7 @@ import pushd
 # XXX: We could probably do a lot of this stuff with svn's python bindings...
 
 def get_revision_no(svndir, module):
-    dir = "%s/%s/trunk" % (svndir, module)
+    dir = "%s/%s" % (svndir, module)
 
     if optz.verbose:
         print "getting the current revision number"
@@ -42,7 +42,7 @@ def update(svndir, module):
         print "doing an svn update"
         pass
 
-    dir = "%s/%s/trunk" % (svndir, module)
+    dir = "%s/%s" % (svndir, module)
     
     try:
         pushd.pushd(dir)
@@ -52,10 +52,13 @@ def update(svndir, module):
             pass
         cmd = "svn up"
         status, output = commands.getstatusoutput(cmd)
+        if optz.verbose:
+            print output
+
         if status != 0:
             raise
         
-        if opts.verbose:
+        if optz.verbose:
             print "svn update finished, apparently successfully."
             pass
 
@@ -69,7 +72,7 @@ def update(svndir, module):
     pass
 
 def checkout(svndir, module, root, nuke=False):
-    dir = "%s/%s/trunk" % (svndir, module)
+    dir = "%s/%s" % (svndir, module)
 
     if nuke == True:
         status, output = commands.getstatusoutput("rm -rf %s" % dir)
@@ -77,7 +80,7 @@ def checkout(svndir, module, root, nuke=False):
     
     try:
         pushd.pushd(svndir)
-        cmd = "svn co %s/%s/trunk %s/trunk" % (root, module, module)
+        cmd = "svn co %s/%s %s" % (root, module, module)
         status, output = commands.getstatusoutput(cmd)
         
     except:
diff --git a/gnome-build-brigade/packagesnapshotter/tarballs.py b/gnome-build-brigade/packagesnapshotter/tarballs.py
index 7e3fddf..a3e05a4 100644
--- a/gnome-build-brigade/packagesnapshotter/tarballs.py
+++ b/gnome-build-brigade/packagesnapshotter/tarballs.py
@@ -14,7 +14,7 @@ import versions
 # Return a dict of, at least, { 'path': <path>, 'version', <version> }
 def dist(baseversion, svndir, module):
     retval = {}
-    path = "%s/%s/trunk" % (svndir, module)
+    path = "%s/%s" % (svndir, module)
     pushd.pushd(path)
     
     # Nuke any tarballs that may be lying around
@@ -24,7 +24,10 @@ def dist(baseversion, svndir, module):
     if status != 0:
         raise
 
-    status, output = commands.getstatusoutput("./autogen.sh --enable-gtk-doc")
+    if optz.verbose:
+        print "About to execute ./autogen.sh --enable-gtk-doc"
+    
+    status, output = commands.getstatusoutput("jhbuild run ./autogen.sh --enable-gtk-doc")
     if status != 0:
         print output
         raise "autogen failed in %s :(" % path
@@ -37,13 +40,18 @@ def dist(baseversion, svndir, module):
     # autoconf
     # configure --enable-gtk-doc
 
-    status, output = commands.getstatusoutput("make dist")
+    if optz.verbose:
+        print "About to execute 'make dist-bzip2'"
+
+    status, output = commands.getstatusoutput("jhbuild run make dist-bzip2")
     if status != 0:
+        print "Error \n %s" % output
         raise
     
     cmd = "ls %s-*tar.*" % tn
     status, output = commands.getstatusoutput(cmd)
     if status != 0:
+        print "Error: \n %s" % output
         raise
     tarball = output
     
@@ -106,6 +114,7 @@ def rename_tarball(tarball, module, their_version, our_version):
 
     status, output = commands.getstatusoutput(cmd)
     if status != 0:
+        print "%s" % output
         raise
     
     nt = "%s.tar.bz2" % new_name
@@ -120,7 +129,7 @@ def rename_tarball(tarball, module, their_version, our_version):
 def make_our_version(baseversion, svndir, module):
     current_date = dates.current_date()
     revno = svn.get_revision_no(svndir, module)
-    claimed_version = versions.configure_thinks("%s/%s/trunk" % (svndir, module), module)
+    claimed_version = versions.configure_thinks("%s/%s" % (svndir, module), module)
 
     # baseversion+claimed_version+current_date+revno
     if optz.verbose:
diff --git a/gnome-build-brigade/upload-tarball b/gnome-build-brigade/upload-tarball
index f0d3372..b538706 100755
--- a/gnome-build-brigade/upload-tarball
+++ b/gnome-build-brigade/upload-tarball
@@ -28,7 +28,7 @@ def main():
     parser.add_option("-s", "--svndir", dest="svndir",
                       help="Specify where svn checkouts live.  Note that checkouts are relative to " +
                       "this path.  Thus, you only need to specify something like '~/svn/gnome'; this " +
-                      "program will automagically expand it into '/home/<you>/svn/gnome/glib/trunk'")
+                      "program will automagically expand it into '/home/<you>/svn/gnome/<module>'")
     parser.add_option("--skip-svn-update", dest="skipsvn", action="store_true",
                       help="Don't bother doing a svn update, to save time")
     parser.add_option("-b", "--builddir", dest="builddir",
@@ -66,6 +66,8 @@ def main():
         fail = True
         print "You need to specify where osc will work"
         pass
+    else:
+        optz.builddir = options.builddir
     
     if fail == True:
         parser.error("")