Re: Integration between buildservice and gnome.build

["John Carr" <john carr unrouted co uk>]

Hi

Interesting stuff. This touches on several areas that JHBuild is
thinking about (or at least, I am).

On jhbuild to distro mappings. This is useful information for other
distros wanting to do build.g.o / package integration and also could
be handy for eventual jhbuild / packagekit support. So it would be
nice to specify something that other distros could provide as well.

There is a jhbuild variant that can build debian packages (jhdebuild),
and when i experimented with that i had make dist problems too. It
would be nice to have an export method on the JHBuild Package class
that tried to do a make dist, but could fall pack to tarring the
source directory.

It would also be nice if this functionality was a jhbuild command so
that jhbuild users could just do 'jhbuild upload-package' or something
similiar, then buildbot could just do a normal "jhbuild bot --step"
type command.

John

On Fri, Jan 16, 2009 at 10:59 AM, API <apinheiro igalia com> wrote:
>
> Hello
>
> sometime ago Rodrigo Moya proposed a kind of integration between the build-brigade
> and the Novell BuildService [1]. The idea is that the build-brigade compile a
> module, and if it is successful, use this code as a target to upload the
> module to the build service, to have an updated SuSE package.
>
> The idea is add a new step (currently we have 5: update, build, check, coverage
> and clean), and execute a script. It is clear that it is not needed that all
> the slaves have this additional step, so we could check how to configure it
> as a conditional step or simply use a modified slave code in some concrete
> machine.
>
> This script basically takes the code, creates a tarball with this code, and
> then use some osc in order to apply some SuSE patches, and upload the package.
>
> Rodrigo Moya and Michael Wolf created a provisional script, and Iago and me
> were reviewing it, so we thought that we should share it with the list. You can
> check it here:
>
> git clone http://www.gnome.org/~rodrigo/git/osc-plugins.git
>
> There is a directory with the build-brigade stuff.
>
> During my review I found some errors and added additional information to
> the verbose mode (to debugging purpose), I attach a patch, so if you agree with
> the changes, please Rodrigo commit it.
>
> At this moment it have some problems, and in fact I was not able to get any
> module uploaded. Some problems I found were:
>
>  * In order to create the tarball it uses make dist
>  * I'm not sure that all the gnome modules support correctly make dist
>  * make dist require to have enabled gtk-doc in some modules, so it is required
>    to do a ./autogen.sh --enable-gtk-doc
>  * This slow the process and making my tests, in some modules (ie libglade) it
>    fails with this error (although not all the times):
>      configure.in:165: required file `doc/Makefile.in' not found
>      configure.in:165: required file `doc/version.xml.in' not found
>      configure.in:165: required file `tests/Makefile.in' not found
>      Makefile.am:1: required directory ./tests does not exist
>    But all theses files and directories exists
>  * So, probably could be a good idea find another way to create the tarball with
>    the source code.
>  * Other problem is related with the fact that there are different name
>    convention for the gnome moduels between jhbuild and the build service, so a
>    name mapping is required (see packagemappings.py file). Currently the mapping
>    list is incomplete.
>
> In you want to do some tests, you'll need to do something like that (with the
> proper local directories):
>
>   ./builddir-prepare -p GNOME:Factory  -m  libglade -b ~/osc -v
>   ./builddir-prepare -p GNOME:snapshots:unpatched  -m  libglade -b ~/osc -v
>   ./upload-tarball -p GNOME:snapshots:unpatched  -m  libglade -s ~/gnome/work/src -b ~/osc -v
>
> (About that, one idea could be include the builddir-prepare on the upload-tarball)
>
> You'll need a account on the build service. There are currently one created for
> build-brigade, to make the tests. If anyone have interest, I can send the login
> and password.
>
> Sorry for this über mail
>
> BR
>
> [1] http://en.opensuse.org/Build_Service/CLI
> [2] http://en.opensuse.org/Build_Service/CLI#osc.2C_the_Python_command_line_client
>
> ===
> API (apinheiro igalia com)
>
> diff --git a/gnome-build-brigade/builddir-prepare b/gnome-build-brigade/builddir-prepare
> index f39413c..6e60a41 100755
> --- a/gnome-build-brigade/builddir-prepare
> +++ b/gnome-build-brigade/builddir-prepare
> @@ -62,8 +62,11 @@ def main():
>     cmd = "osc checkout %s %s" % \
>         (project, packagemappings.suse_package_name(package))
>
> +    if optz.verbose:
> +        print "About to execute %s" % cmd
>     status, output = commands.getstatusoutput(cmd)
>     if status != 0:
> +        print "Fail\n %s" % output
>         raise
>
>     pushd.popd()
> diff --git a/gnome-build-brigade/packagesnapshotter/osc.py b/gnome-build-brigade/packagesnapshotter/osc.py
> index ba19b5f..b276638 100644
> --- a/gnome-build-brigade/packagesnapshotter/osc.py
> +++ b/gnome-build-brigade/packagesnapshotter/osc.py
> @@ -3,19 +3,27 @@
>  # misc osc commands.  Probably should use osc's api directly, of
>  # course.  Maybe later.
>
> +import commands
>  import optz
>  import packagemappings
>  import pushd
> +import oscpluginsgnome.pacversion
>
>  def get_last_version(project, package):
>
> -    dir = "%s/%s/%s" % (optz.builddir, project, package)
> +    dir = "%s/%s/%s" % (optz.builddir, project, packagemappings.suse_package_name(package))
>     pushd.pushd(dir)
>
>     cmd = "osc pacversion %s %s" % \
>         (project, packagemappings.suse_package_name(package))
>
> +    if optz.verbose:
> +        print "About to execute: \"%s\"" % cmd
> +
>     status, output = commands.getstatusoutput(cmd)
> +    if optz.verbose:
> +        print output
> +
>     if status != 0:
>         raise
>
> @@ -32,8 +40,13 @@ def checkout():
>  def add(package):
>     cmd = "osc add %s" % package
>
> +    if optz.verbose:
> +        print "About to execute %s" % cmd
> +
>     status, output = commands.getstatusoutput(cmd)
>     if status != 0:
> +        print "Error executing %s" % cmd
> +        print "Output\n %s" % output
>         raise
>     pass
>
> @@ -44,5 +57,7 @@ def commit(message):
>     status, output = commands.getstatusoutput(cmd)
>
>     if status != 0:
> +        print "Error executing %s" % cmd
> +        print "%s" % output
>         raise
>     pass
> diff --git a/gnome-build-brigade/packagesnapshotter/svn.py b/gnome-build-brigade/packagesnapshotter/svn.py
> index 29932f4..5c7e2b4 100644
> --- a/gnome-build-brigade/packagesnapshotter/svn.py
> +++ b/gnome-build-brigade/packagesnapshotter/svn.py
> @@ -12,7 +12,7 @@ import pushd
>  # XXX: We could probably do a lot of this stuff with svn's python bindings...
>
>  def get_revision_no(svndir, module):
> -    dir = "%s/%s/trunk" % (svndir, module)
> +    dir = "%s/%s" % (svndir, module)
>
>     if optz.verbose:
>         print "getting the current revision number"
> @@ -42,7 +42,7 @@ def update(svndir, module):
>         print "doing an svn update"
>         pass
>
> -    dir = "%s/%s/trunk" % (svndir, module)
> +    dir = "%s/%s" % (svndir, module)
>
>     try:
>         pushd.pushd(dir)
> @@ -52,10 +52,13 @@ def update(svndir, module):
>             pass
>         cmd = "svn up"
>         status, output = commands.getstatusoutput(cmd)
> +        if optz.verbose:
> +            print output
> +
>         if status != 0:
>             raise
>
> -        if opts.verbose:
> +        if optz.verbose:
>             print "svn update finished, apparently successfully."
>             pass
>
> @@ -69,7 +72,7 @@ def update(svndir, module):
>     pass
>
>  def checkout(svndir, module, root, nuke=False):
> -    dir = "%s/%s/trunk" % (svndir, module)
> +    dir = "%s/%s" % (svndir, module)
>
>     if nuke == True:
>         status, output = commands.getstatusoutput("rm -rf %s" % dir)
> @@ -77,7 +80,7 @@ def checkout(svndir, module, root, nuke=False):
>
>     try:
>         pushd.pushd(svndir)
> -        cmd = "svn co %s/%s/trunk %s/trunk" % (root, module, module)
> +        cmd = "svn co %s/%s %s" % (root, module, module)
>         status, output = commands.getstatusoutput(cmd)
>
>     except:
> diff --git a/gnome-build-brigade/packagesnapshotter/tarballs.py b/gnome-build-brigade/packagesnapshotter/tarballs.py
> index 7e3fddf..a3e05a4 100644
> --- a/gnome-build-brigade/packagesnapshotter/tarballs.py
> +++ b/gnome-build-brigade/packagesnapshotter/tarballs.py
> @@ -14,7 +14,7 @@ import versions
>  # Return a dict of, at least, { 'path': <path>, 'version', <version> }
>  def dist(baseversion, svndir, module):
>     retval = {}
> -    path = "%s/%s/trunk" % (svndir, module)
> +    path = "%s/%s" % (svndir, module)
>     pushd.pushd(path)
>
>     # Nuke any tarballs that may be lying around
> @@ -24,7 +24,10 @@ def dist(baseversion, svndir, module):
>     if status != 0:
>         raise
>
> -    status, output = commands.getstatusoutput("./autogen.sh --enable-gtk-doc")
> +    if optz.verbose:
> +        print "About to execute ./autogen.sh --enable-gtk-doc"
> +
> +    status, output = commands.getstatusoutput("jhbuild run ./autogen.sh --enable-gtk-doc")
>     if status != 0:
>         print output
>         raise "autogen failed in %s :(" % path
> @@ -37,13 +40,18 @@ def dist(baseversion, svndir, module):
>     # autoconf
>     # configure --enable-gtk-doc
>
> -    status, output = commands.getstatusoutput("make dist")
> +    if optz.verbose:
> +        print "About to execute 'make dist-bzip2'"
> +
> +    status, output = commands.getstatusoutput("jhbuild run make dist-bzip2")
>     if status != 0:
> +        print "Error \n %s" % output
>         raise
>
>     cmd = "ls %s-*tar.*" % tn
>     status, output = commands.getstatusoutput(cmd)
>     if status != 0:
> +        print "Error: \n %s" % output
>         raise
>     tarball = output
>
> @@ -106,6 +114,7 @@ def rename_tarball(tarball, module, their_version, our_version):
>
>     status, output = commands.getstatusoutput(cmd)
>     if status != 0:
> +        print "%s" % output
>         raise
>
>     nt = "%s.tar.bz2" % new_name
> @@ -120,7 +129,7 @@ def rename_tarball(tarball, module, their_version, our_version):
>  def make_our_version(baseversion, svndir, module):
>     current_date = dates.current_date()
>     revno = svn.get_revision_no(svndir, module)
> -    claimed_version = versions.configure_thinks("%s/%s/trunk" % (svndir, module), module)
> +    claimed_version = versions.configure_thinks("%s/%s" % (svndir, module), module)
>
>     # baseversion+claimed_version+current_date+revno
>     if optz.verbose:
> diff --git a/gnome-build-brigade/upload-tarball b/gnome-build-brigade/upload-tarball
> index f0d3372..b538706 100755
> --- a/gnome-build-brigade/upload-tarball
> +++ b/gnome-build-brigade/upload-tarball
> @@ -28,7 +28,7 @@ def main():
>     parser.add_option("-s", "--svndir", dest="svndir",
>                       help="Specify where svn checkouts live.  Note that checkouts are relative to " +
>                       "this path.  Thus, you only need to specify something like '~/svn/gnome'; this " +
> -                      "program will automagically expand it into '/home/<you>/svn/gnome/glib/trunk'")
> +                      "program will automagically expand it into '/home/<you>/svn/gnome/<module>'")
>     parser.add_option("--skip-svn-update", dest="skipsvn", action="store_true",
>                       help="Don't bother doing a svn update, to save time")
>     parser.add_option("-b", "--builddir", dest="builddir",
> @@ -66,6 +66,8 @@ def main():
>         fail = True
>         print "You need to specify where osc will work"
>         pass
> +    else:
> +        optz.builddir = options.builddir
>
>     if fail == True:
>         parser.error("")
>
> _______________________________________________
> Build-brigade-list mailing list
> Build-brigade-list gnome org
> http://mail.gnome.org/mailman/listinfo/build-brigade-list
>
>