Re: [gnome-love] My proposals for Gnome

[Malcolm Tredinnick <malcolm commsecure com au>]

I have a few comments on the practicaltilty of one of the ideas Tom
raised. Having written this and looked over it, I hate myself. This is
really "the voice of negativity" speaking. :(

However, I will post it, since I think there is a lot of talk on various
Gnome mailing lists at the moment about what we 'could' do and we need
to start thinking whether things are practical. Mostly, I think Tom's
ideas are interesting and worth considering, but I am a long way from
being sold on the idea that there is some panacea in having testing and
other verification done automatically.

On Thu, Jun 07, 2001 at 12:16:41PM -0500, Tom Musgrove wrote:
[...snip...]
> 2) An automated bug discovery and reporting tool
>
> This should be a cooperative effort between Ximian, RH, SourceForge, and
> everyone else under the sun.  What I'd like to see is Tinderbox set up with
> additional functionality - so that upon CVS checkin a list of things happens
>
> a) Runs Lint b) Runs a Pretty Printer (checks for white space
> consistency...) c) Compiles software with a memory profiler debugger etc. d)
> Runs the software with a memory shaker, a tool that gives garbage input,
> etc.  5) Generates bug reports 'automagically' (just adding a comment to
> previous bug reports that give the same error message), and sends an email
> to whoever broke the CVS (assuming that it doesn't compile)

This is not quite as easy as it looks on paper, for the following
reason:

(1) The "pretty printer" idea: different projects have different
indentation styles. There is no one true layout method and to try and
insist on one will simply generate a few megabytes of email and no
consensus. For example, the GNU coding guidelines have been out for
years and some projects follow them. Others (even major Gnome
components) modify them slightly and still have readable code. Some big
project (e.g. linux kernel) throw them out entirely and still have
readable code. A lot of people know what the true and correct
indentation style is, the problem is that they all know it to be
something different from all the other people. The rest of us just
adjust to the different styles in different modules; it's not that hard.

Even were somebody to try and mandate that we will all use indentation
style X, it would be a travesty to go through CVS and reindent or
otherwise alter the existing code. You would screw up CVS's knowledge of
who last edited that line. This is part of the reason why people are
encouraged to stick with whatever coding style already exists in a
project, rather than doing mass reformattings -- it is *very* useful to
be able to see when a line was last changed and generally retain the
history of pertinent alterations.

(2) Automatic compiling: please remember that it is not a requirement
(and should not, in general, be one) that everything in CVS must compile
and run. For many (most?) Gnome modules, the contributors are spread
around the world. The main way they have to communicate their coding
changes to each other is through CVS, so sometimes code will be
committed simply for the purpose of sharing it with the other
developers. Especially between point releases the code on the various
CVS branches may well be broken and that is not a bad thing. 

Some people may argue that if code is going to break the compile, then
it should go on its own branch. That's not an unreasonable attitude. But
then the tinderbox maintainers have to keep track of which branches are
"stable" ones (e.g. the GNOME_1_4 branch in gnome-core) and which are
temporary development ones. Secondly, I am a believer that the
maintainers of the module should be the ones to determine if their main
branch must always compile. People who arbitrarily insist that CVS code
from times between releases must always compile and run correctly are,
in my opinion, contributing only negatively the project -- they remove
the ability to experiment in consultation with other developers. For
that matter, even point releases in unstable branches can have known
problems.

Because of these points, any sort of automatic setup needs a way for
maintainers to add or remove their module from the process (during
periods when they are aiming at a release or developing madly,
respectively). Otherwise you are going to have truckloads of spurious
bug reports and email spam about expected bugs and people will just
ignore them as "yet another thing we sort of expected".

(3) Giving garbage input is HARD at the moment. Most gnome applications
are based around some sort of GUI. It is possible to set up tests on a
case-by-case basis sometimes, but to do it for all apps, automatically
at the moment is an enormous undertaking.

Consider the work that must be done:
        (a) somebody has to write a tool that will automatically poke at
        buttons and know about entry widgets and stuff.

        (b) Each project must then have some sort of file that describes
        to this tool what it's GUI is like (I can't imagine how this
        would work, but it's required). This file must be kept up to
        date and since it's really a duplicate of work in the project's
        code and since it probably can't be automatically generated, it
        will often slip out of sync.

        (c) The application that does all this random poking about must
        be run in a very high-walled sandbox. For example, imagine
        you're testing a file manager via this method. Somewhere in
        there is the ability to delete files. You aren't going to run
        that in $HOME are you? :-)

Now, none of these problems are impossible to overcome. Late last year I
wrote down what a tool like that in (a) would look like and do (then the
problems in (b) and (c) became clear and I got depressed about it). The
Palm Pilot emulation stuff has a program called Gremlins, which randomly
pokes about at your application and if it crashes, records where it was
up to. This is all done via software emulation of the Palm (so it
effects the sandbox idea). Since sandboxing is a bit fiddly (because you
need to sandbox an entire Gnome setup to be accurate), I was also
thinking about scripting tests (as can be done with some tools on
Microsoft's systems). I got further here (there's a spec somewhere on my
backup disks if people care), but it's still not bullet-proof: for
example, the script says to click on the first button on the left, the
author recently changed that to make it the "delete" button and ... oh,
dear!

I think part (b) is the really hard bit. You have to sell everybody on
the idea that the advantages of automatic testing on many different
platforms (if it's only single-platform, it's worse than just letting
the developers manage the testing) is worth the effort of having yet
another dependent file. Then we need to come up with a format that
works.

[...snip...]
> 3) Ultimate Help System
>
> "In reality, I was thinking of an Admin setting up a local help channel.
> The admin then requires all (most?) help requests to be submitted via the
> help.
>
> The users aren't actually accessing IRC, nor do the admins actually need to
> be in IRC to receive the message - any sort of messaging service would work
> fine as long as there a is a way for the user to enter the question, and the
> admin to send the response.

I'm a bit confused as to what advanatages this has over mailing lists
(since even the realtime reponse of IRC is not considered necessary).

It sounds to me like a lot of what you are talking about here could be
accomplished by turning archives of gnome-list and gnome-hackers into
more searchable FAQ lists. But I'm probably missing something in your
proposal because I haven't really thought about this as much as the
above points.

Cheers,
Malcolm

-- 
Eagles may soar, but weasels don't get sucked into jet engines.