Re: Announcement/RFC: jhbuild continuous integration testing -- mystery of recent flood of failures solved

[Tristan Van Berkom <tvb gnome org>]

On Tue, Feb 19, 2013 at 10:13 PM, Simon McVittie
<simon mcvittie collabora co uk> wrote:
> On 18/02/13 22:34, Martin Pitt wrote:
> > Please note that there is no system D-BUS and no default session D-BUS
> > running. If you need those, then the tests should start dbus-launch or
> > use GTestDBus.
>
> dbus-launch is not particularly suitable for regression tests: if you
> use it, you have to kill the resulting dbus-daemon yourself when you're
> finished with it. If not using GTestDBus, please use with-session-bus.sh
> from telepathy-glib, or review
> <https://bugs.freedesktop.org/show_bug.cgi?id=39196> so I can add
> dbus-run-session(1) to dbus.

This is a very interesting topic (and brings to mind Colin's ideas about
installed tests).

Here's some ideas worth consideration IMHO...

  o Unit Tests with GTestDBus

     GTestDBus is IMO ideal for regression testing (or in-tree unit tests),
     I made a short write-up on this not long ago in my blog[0].

     The idea here is that you want to be absolutely sure that you
     are testing isolated modules and services that are still in-tree,
     you want to test ideally your services alone without clouding
     your results with installed services. If your service relies on
     system installed services, you would ideally want to control
     which specific installed services get to run in your controlled
     D-Bus environment sandbox.

     I.e. if you have services in /usr/share/dbus-1, you dont want those
     mixed in to your sandboxed build path, colliding with services in
     /opt/devel/share/dbus-1/.

     You also probably want to control cases where fallbacks can be
     implemented, if your service/client can run without a complimentary
     service... you want to test a case where your client has access to
     an installed service vs. a case where a fallback is used instead.

 o Now that we are talking about a build server and building 'all-of-gnome'
    it becomes interesting to know if a service installed by some dependency
    effects any modules which depend on that service in a negative way.

    For this case (why I thought about Colin's ideas on installed tests), it
    suddenly becomes interesting to have tests which do not use GTestDBus
    in a controlled environment, but instead to test the system as a whole,
    running only services from ${build_prefix}/share/dbus-1 but certainly
    avoiding anything from /usr/share/dbus-1/ (or at least properly prioritizing
    the build prefix services over the system installed ones, if we can't avoid
    using those).

 o If we have these two theories of testing D-Bus services and clients which
    depend on them, we probably want to reuse the unit testing code as much
    as possible.

    Perhaps some additional features added to GTestDBus could allow us
    to run the same tests in both contexts (i.e. the installed test context
    with a "full gnome environment" vs. the isolated in-tree context).

Cheers,
          -Tristan

[0]: http://blogs.gnome.org/tvb/2012/12/20/isolated-unit-testing-of-d-bus-services/