Re: [Evolution-hackers] nightly builds of Evolution + testing withSyncEvolution

[Patrick Ohly <Patrick Ohly gmx de>]

On Mon, 2008-02-11 at 15:32 +0530, Srinivasa Ragavan wrote:
> On Sun, 2008-02-10 at 12:18 +0100, Patrick Ohly wrote:
> > I wonder whether this regular building and testing is of interest to
> > anybody else? The build script sends out a short summary email which
> > links to full logs for each night; I could easily add other recipients.
> > For the Evolution build these include the changes since the last build.
> > 
> I think it may prove very helpful in the long run, where some commits
> cause some side effects, which we may not easily identify in normal
> cases. But the test script gotto be foolproof to make this really
> useful.

They have worked fine so far. I'll see how it'll go in the future.

> > Are there other tests of the EDS API which I should run? If Novel
> > already does regular testing this probably isn't needed.
> AFAIK, we don't have much automated tests except the tests under the
> respective folders. Akhil, correct me if I'm wrong. Patrick, if your
> test script could cover some cases from those test files also, it will
> be really useful.

I can run "make check" after a build. Are all tests going to be run by
that?

How do I detect failures? I just tried it on evolution-data-server and
got e.g.:
        [...]
                Failed 48/662 tests, 92% okay
        
                Failed tests:
                  
                  2/23 2/24 2/26 2/27 2/28 2/29 2/32 2/47 2/48 2/50 2/51
        2/52 2/53 2/56 2/71 2/72 2/74 2/75 2/76 2/77 2/80 2/95 2/96 2/98
        2/99 2/100 2/101 2/104 2/119 2/120 2/122 2/123 2/124 2/125 
                  10/295 10/325 
                  14/344 14/345 14/346 
                  15/347 15/349 15/350 15/351 
                  27/486 
                  31/503 
                  32/505 
                  39/599 39/600 
        PASS: regression
        ==================
        All 1 tests passed
        ==================
        make[5]: Leaving directory
        `/tmp/runtests/head/tmp/evolutiontrunk-build/evolution-data-server/calendar/libical/src/test'

The "make check" at top level returned no error code. Looking at
calendar/libical/src/test/regression.c I see that it always returns 0,
so I guess I have to scan the log output to find failures, right?

Are the failures above something that should be reported? To make
regular testing useful failing tests either have to be skipped or fixed
in a timely fashion.

-- 
Bye, Patrick Ohly
--  
Patrick Ohly gmx de
http://www.estamos.de/