[xml] Python script for Regression Tests (for Linux and Windows)

  During a conversation with Daniel a few weeks ago, we talked about
the possibility of modifying the existing libxml2 "regression
tests" (which currently run under the gnu 'make' program) to run
under Python, so that the same test suite could be executed on
platforms other than gnu-based systems (in particular under
Windows).  I spent some time this last weekend and composed a
first attempt at such a script.

  There is still some amount of work to be done, but most of that
work is in adding to the existing parameter files (which is what I
classify as "donkey work", i.e. it takes time, but not much
careful thought).  Unfortunately, I'm going to be fairly busy over
the next week or more, and won't be able to put much more time
into this, so I thought it would be worthwhile to offer what I've
done so far to the list for anyone interested to "try out".

  I'm attaching to this post a .tgz file containing the script
(regressions.py) together with two parameter files
(regressions.std.xml for Linux, and regressions.win32.xml for
Windows).  The parameter files include the tests from "XML
regression tests" through "HTML regression tests".  I have tested
them on both Linux and Windows and (with a few exceptions) things
are working well.  The "exceptions" are a few tests which fail
under Windows - I haven't spent any time looking into where the
problem is (it could, for example, be something unique to my
installation), but rather I added an "exclude" parameter to take
them out of the Windows version. Similarly, the tests using the
"--memory" switch, as well as all of the SAX tests, have been
removed from the Windows parameter file.

  The biggest thing which I'm not comfortable with is the parsing of
the params.  I didn't want to spend any great amount of time on
that part of the operation, and I had never written an xmlreader
application before, so I took the opportunity to try xmlreader for
this job.  Again, I really wasn't interested in being clever in
the layout / design of the params, so the xml file I came up with
is practically brain-dead (but it works :-).  It's probably not
really a good idea to require the python bindings for libxml2 in
order to test libxml2, so it may be worthwhile to convert the
param parsing over to some completely different method / file
layout.  Within the script, the reading / parsing of the params is
just a few relatively small classes which result in a Python
dictionary containing the parsed params, so changing that part
would be relatively simple.

  There were a couple of more technically challenging items.  The
basic method used by the script to run the various tests is to use
python's os.popen3 (run a task using pipe files for stdin, stdout
and stderr).  The "task" can be xmllint, testXPath, testHTML or
whatever (controlled by the parameter file).  Unfortunately, there
is a problem in Python when trying to read the supplied file
descriptors for stdout and stderr - if lots of data is written to
one descriptor before anything is written to the other, the read
within the script can "block" waiting for the task to write data
to one descriptor, while the task blocks waiting for the script to
read the data it has written so far to the other descriptor. 
Since some of the tests write lots of data to stdout and nothing
to stderr, while other tests write lots of data to stderr and
little or nothing to stdout, one cannot arbitrarily choose one of
them to read first.  A good solution would be to use Python's
version of the select() function, but unfortunately that's
apparently not available under "some operating systems" (which,
according to the docs, includes Windows).  I finally had to settle
on some very basic multi-threading, with one thread handling
stdout and another thread handling stderr.

  For comparing the actual vs. expected results on each test, Python
has a "difflib" module which is really ideal for what we want to
do. I started out using it (very successfully), but while testing
the SAX routines I got into some situation where it still worked
perfectly on Linux, but on the Windows implementation the difflib
routine went into a dead loop :-(.  So, I abandoned that and just
did a simple compare of the results.  That works fine as long as
they are equal, it just isn't very nice when they are different
(i.e. if the test fails).

  Anyway, feedback is welcome- criticisms, additions, enhancements,
even downright insults will all be appreciated!



Attachment: regressions.tgz
Description: application/gzip-compressed

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]