GNOME.org
 

Re: Gtk+ unit tests (brainstorming)

Tim Janik wrote:


On Thu, 26 Oct 2006, Tristan Van Berkom wrote:


Havoc Pennington wrote:


Michael Urman wrote:



On 10/25/06, Tim Janik <timj imendio com> wrote:



- GLib based test programs should never produce a "CRITICAL **:" or
  "WARNING **:" message and succeed.



It would be good not to make it impossible to test WARNINGs and
CRITICALs. After all, error cases are often the least tested part of
an application, so it's important to make sure the base library
detects and handles the error cases correctly.



Testing those is like testing segfault handling, i.e. just nuts. The
behavior is undefined once they print. (Well, for critical anyway.
g_warning seems to be less consistently used)



On the other hand - its happened to me several times to have gtk+
crash when setting up properties on an object genericly from glade,



please make sure to file these as bug reports against Gtk+.
gtk shouln't crash if properties are set within valid ranges.


Yes siree, usually if not always - I do.


where a simple g_return_if_fail() guard on that public api would have
saved me the trouble of a segfault



i reapet, relying on g_return_if_fail() for production code is at beast
futile and at worst dangerous. it might be defined to a NOP.


Yes, my point here is only that having a warning for misusing the api,
particularly in /development/ code - should be part of the contract of
the api - maybe it doesnt make sence to test for it... fuzzy.


(sometime one property has no
meaning if another one hasnt been setup yet - in which case a
g_return_if_fail() guard would be appropriate).
wrong, some proeprty values are intentionally set up to support free-order settings. e.g. foo=5; foo-set=true; or foo-set=true; foo=5; 
every order restriction or property dependency that is introduced in the
proeprty API makes generic handling of properties harder or sometimes
impossible so should be avoided at all costs.
(examples for generic property handling code are language bindings,
property editors, glade, gtk-doc)
I think thats a little uncooperative - wouldnt you say on the other hand - that the independance of property ordering - aside from possible real weird corner cases - should be promoted as much as possible to make life easier for language 
binding writers, property editors and such ?
It is ofcourse a two-way street in the sence that one should not be mislead and 
not program defensively in this aspect.


Whether or not its really really relevent,  I think that the critical
warnings from a function that wasnt fed the correct arguments,
or are invalid because of the current object state - are part of
the contract of the api, and for whatever thats worth, maybe
worth testing for.



no, they are not. functions are simply just defined within the range
specified by the implementor/designer, and in this function for a lot
of glib/gtk functions, i'm informing you that occasional return_if_fail
statements that you may or not may happen to find in glib/gtk code base
are a pure convenience tool to catch programming mistakes easier.

--- that being said, please read on ;)

the above is intentionally worded conservatively, e.g. to allow removal
of return_if_fail statements in the future, so value ranges a function
operates on can be extended *compatibly*. in practice however, we're
doing our best to add missing warning/checks that help programmers
and users to code and use our libraries in the most correct and
reliable way. but by definition, a library/program is broken once a
warninig/critical/assertion/error has been triggered and there is no
point in verifying that these are in fact triggred. i.e. you'd
essentially try to verify that you can make the library/program enter
an undefined state by a certain sequence of usage patterns, and that
is rather pointless.


Yes I see, saying whether it is part of the "contract" may sound binding,
but an effort to report api misuse is definitly in need. Also adding
a g_return_if_fail() to your function should probably not constitute an
addition to your test suite (i.e. you'd usually progressively add tests
to ensure that your api complains properly when misused - well,
I guess its a lot of effort for a low yield).


Ummm, while I'm here - I'd also like to say that - (I'm not about to
dig up the quote from the original email but) - I think that there
is some value to reporting all the tests that fail - even after one
of the tests has failed, based on the same principals that you'd
want your compiler to tell you everything that went wrong in its
parse (not just the first error),



well, try to get your compiler tell you about all compilation errors
in a project covering multiple files or directories if you broke a
multitude of files and in different directories/libraries. it simply
won't work for the vast majority of cases, and also isn't useful in
practice. similar to unit tests, most times a compiler yields errors,
you have to fix the first one before being able to move on.



maybe not all spewed out to the console
by default - but having the whole test results in a log file can
save valuable developer time in some situations.



i don't quite see the scenrio here, so it might be good if you cared
to elaborate this. as described in my initial email, if anything breaks,
it either shouldn't be under test or should be fixed, and i provided
reasonings for why that is the case. if you really mean to challenge
this approach, please also make an effort to adress the reasoning
i provided.


so I did go dig up the original mail:

====================================

 having/letting more than one test fail and to continue work in an
 unrelated area rapidly leads to confusion about which tests are
 supposed to work and which aren't, especially in multi-contributor setups.
 figuring whether the right test passed, suddenly requires scanning of
 the test logs and remembering the last count of tests that may validly
 fail. this defeats the purpose using a single quick make check run to
 be confident that one's changes didn't introduce breakage.
 as a result, the whole test harness should always either succeed or
 be immediately fixed.

====================================

Sure I'll elaborate, what I mean is - if you dont want to pollute
the developers console with alot of errors from every failing test,
you can redirect them to some log file and only say
"at least one test failed" - maybe depict which one was the first
failure - and then point to the log.

Your arguments against collecting all the failures seem to
apply fine to gcc output too - and yes you might usually end up just
fixing the first error and recompiling, yet I manage to still appriciate
this feature of verbosity - delivered at the not too high cost of
wording a makefile a little differently (for unit tests that is) - i'd think. 

Sorry to make it sound like I can talk about something I didnt read
about - I did actually read your initial email with quite a hunger,
was pleased and digested it nicely I think - I do appriciate your
work and I dont want to come off to agressive with my sincere comments.

Cheers,
                                -Tristan
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]