Re: New GnomeGoal proposal: InstalledTests

[Travis Reitter <travis reitter collabora co uk>]

On Tue, 2013-04-30 at 10:27 -0700, Travis Reitter wrote:
> On Fri, 2013-04-26 at 22:47 +0900, Tristan Van Berkom wrote:
> > On Fri, Apr 26, 2013 at 9:46 PM, Matthias Clasen
> > <matthias clasen gmail com> wrote:
> >         On Thu, Apr 25, 2013 at 10:45 AM, Colin Walters
> >         <walters verbum org> wrote:
> >         
> >         >
> >         >> Do we have (makefile) infrastructure to allow GTests to run
> >         both
> >         >> uninstalled (during make *check) and installed?
> >         >
> >         > Not at this time; that'd be nice, but I think the jhbuild
> >         model
> >         > mostly obviates the need for uninstalled tests, because
> >         jhbuild by
> >         > nature is a sandboxed environment independent from the
> >         underlying
> >         > system.
> >         
> >         
> >         
> >         You are not going to get me to buy eagerly into a new
> >         installed tests
> >         scheme for glib if it means that I have to give up make check.
> >
> >
> > I just wanted to jump in and mention that, I'd really really like to
> > see
> > better all around relocatability of modules.
> >
> >
> > Ideally, I'd like to see the following things:
> >   o Ability to run installed tests, encapsulated in the jhbuild prefix
> >
> >   o Ability to use the same test cases that I use in-tree as installed
> >
> >      tests, hopefully by simply installing my in-tree tests somewhere
> >
> >      (perhaps they would run without some of the custom env vars
> >
> >      that I would use in-tree, but still re-use all the same test
> > suites
> >
> >      when installed)
> >
> >   o Overall relocated D-Bus and settings environment
> >
> >
> >
> > This last one is a really big deal, there are some really annoying
> > things right now, for instance I can in *no way* test GTK+ apps
> >
> > in a jhbuild shell and use the theme installed in the jhbuild prefix.
> > It seems natural that when typing 'jhbuild shell', any processes
> > that run should be accessing the settings persisted somewhere
> > in /opt/devel... instead I would have to slave my whole system 
> > just to see what an app looks like with the theme installed
> > in /opt/devel.
>
> This problem applies to a whole classes of Gnome development. I just
> lost of a lot of time yesterday trying to set up a development
> environment in a VM so I would have a full, graphical session (because
> apparently parts of the Shell are what normally start up some EDS
> components, so 'jhbuild shell' is insufficient). I still don't have a
> working solution here and I've lost track of the times I've tried to
> resolve this with existing components without success.
>
> Any long-term solution needs to be based within a standard VM setup that
> most developers (core and third-party) all use (for dog-fooding
> purposes). Many developers can get away with just using jhbuild now, but
> it's too different from a regular session for other developers.
>
> I think there was general agreement on this idea in the Gnome Developer
> Experience Hackfest in February. We need an SDK that is the official and
> de facto standard for first- and third-party Gnome development so we can
> all benefit from solutions to our common development problems.
>
> > GTestDBus has brought us at least part of the way there when
> > it comes to D-Bus, but if we run installed tests we'll want to disable
> > the relocation of services done by GTestDBus (so that some apps
> > and services can interact with other services installed into the
> >
> > build prefix, instead of running in complete isolation which is
> > more appropriate for 'make check'). Perhaps this can be done
> > by hardwiring some features directly into GTestDBus, of course
> > in this case we still would want to run a separate session bus for
> > the jhbuild environment, so that the whole relocated installed
> > test suite does not interfere with a real active session.
>
> When I'm developing, I honestly only care how my new code works when
> it's installed. I use build-tree tests to save the time of installing,
> starting a new session for the development user (if necessary), etc. The
> length of the development cycle between each edit and verifying it is
> critical. The difference between a 3s and 1m loop is huge in my
> productivity.
>
> But if a 1-minute loop meant not fighting issues related to build-tree
> vs. installed, full graphical session vs. jhbuild session, etc. issues,
> I'd very gladly take it. And 1 minute is just an arbitrary number. I'm
> sure we could minimize the lag added by installation, session
> initiation, etc.

In the nearer-term, if we could optimize 'make check' for quick sanity
checks (particularly if we could work out minimum dependencies
automatically) and run installed tests for a more comprehensive test, I
would probably move a fair number of the Folks tests to be installed.
That could speed up my development loop quite a bit without too much
work and would buy more time to speed up the "comprehensive path"
installed test loop. (They may be fundamentally slower, especially if we
emphasized a "quick vs. comprehensive" split, but it'd be good to work
out best practices and make them clear and commonly-followed).

-Travis