Re: Gtk+ unit tests (brainstorming)

[Tim Janik <timj imendio com>]

On Thu, 26 Oct 2006, Tristan Van Berkom wrote:

> Havoc Pennington wrote:
>
> > Michael Urman wrote:
> >
> >
> > > On 10/25/06, Tim Janik <timj imendio com> wrote:
> > >
> > >
> > > > - GLib based test programs should never produce a "CRITICAL **:" or
> > > >   "WARNING **:" message and succeed.
> > > It would be good not to make it impossible to test WARNINGs and
> > > CRITICALs. After all, error cases are often the least tested part of
> > > an application, so it's important to make sure the base library
> > > detects and handles the error cases correctly.
> >
> > Testing those is like testing segfault handling, i.e. just nuts. The
> > behavior is undefined once they print. (Well, for critical anyway.
> > g_warning seems to be less consistently used)
> On the other hand - its happened to me several times to have gtk+
> crash when setting up properties on an object genericly from glade,

please make sure to file these as bug reports against Gtk+.
gtk shouln't crash if properties are set within valid ranges.

> where a simple g_return_if_fail() guard on that public api would have
> saved me the trouble of a segfault

i reapet, relying on g_return_if_fail() for production code is at beast
futile and at worst dangerous. it might be defined to a NOP.

> (sometime one property has no
> meaning if another one hasnt been setup yet - in which case a
> g_return_if_fail() guard would be appropriate).

wrong, some proeprty values are intentionally set up to support 
free-order settings. e.g. foo=5; foo-set=true; or foo-set=true; foo=5;
every order restriction or property dependency that is introduced in the
proeprty API makes generic handling of properties harder or sometimes
impossible so should be avoided at all costs.
(examples for generic property handling code are language bindings,
property editors, glade, gtk-doc)

> Whether or not its really really relevent,  I think that the critical
> warnings from a function that wasnt fed the correct arguments,
> or are invalid because of the current object state - are part of
> the contract of the api, and for whatever thats worth, maybe
> worth testing for.

no, they are not. functions are simply just defined within the range
specified by the implementor/designer, and in this function for a lot
of glib/gtk functions, i'm informing you that occasional return_if_fail
statements that you may or not may happen to find in glib/gtk code base
are a pure convenience tool to catch programming mistakes easier.

--- that being said, please read on ;)

the above is intentionally worded conservatively, e.g. to allow removal
of return_if_fail statements in the future, so value ranges a function
operates on can be extended *compatibly*. in practice however, we're
doing our best to add missing warning/checks that help programmers
and users to code and use our libraries in the most correct and
reliable way. but by definition, a library/program is broken once a
warninig/critical/assertion/error has been triggered and there is no
point in verifying that these are in fact triggred. i.e. you'd
essentially try to verify that you can make the library/program enter
an undefined state by a certain sequence of usage patterns, and that
is rather pointless.

> Ummm, while I'm here - I'd also like to say that - (I'm not about to
> dig up the quote from the original email but) - I think that there
> is some value to reporting all the tests that fail - even after one
> of the tests has failed, based on the same principals that you'd
> want your compiler to tell you everything that went wrong in its
> parse (not just the first error),

well, try to get your compiler tell you about all compilation errors
in a project covering multiple files or directories if you broke a
multitude of files and in different directories/libraries. it simply
won't work for the vast majority of cases, and also isn't useful in
practice. similar to unit tests, most times a compiler yields errors,
you have to fix the first one before being able to move on.

> maybe not all spewed out to the console
> by default - but having the whole test results in a log file can
> save valuable developer time in some situations.

i don't quite see the scenrio here, so it might be good if you cared
to elaborate this. as described in my initial email, if anything breaks,
it either shouldn't be under test or should be fixed, and i provided
reasonings for why that is the case. if you really mean to challenge
this approach, please also make an effort to adress the reasoning
i provided.

> Cheers,
>                          -Tristan

---
ciaoTJ