During a conversation with Daniel a few weeks ago, we talked about the possibility of modifying the existing libxml2 "regression tests" (which currently run under the gnu 'make' program) to run under Python, so that the same test suite could be executed on platforms other than gnu-based systems (in particular under Windows). I spent some time this last weekend and composed a first attempt at such a script. There is still some amount of work to be done, but most of that work is in adding to the existing parameter files (which is what I classify as "donkey work", i.e. it takes time, but not much careful thought). Unfortunately, I'm going to be fairly busy over the next week or more, and won't be able to put much more time into this, so I thought it would be worthwhile to offer what I've done so far to the list for anyone interested to "try out". I'm attaching to this post a .tgz file containing the script (regressions.py) together with two parameter files (regressions.std.xml for Linux, and regressions.win32.xml for Windows). The parameter files include the tests from "XML regression tests" through "HTML regression tests". I have tested them on both Linux and Windows and (with a few exceptions) things are working well. The "exceptions" are a few tests which fail under Windows - I haven't spent any time looking into where the problem is (it could, for example, be something unique to my installation), but rather I added an "exclude" parameter to take them out of the Windows version. Similarly, the tests using the "--memory" switch, as well as all of the SAX tests, have been removed from the Windows parameter file. The biggest thing which I'm not comfortable with is the parsing of the params. I didn't want to spend any great amount of time on that part of the operation, and I had never written an xmlreader application before, so I took the opportunity to try xmlreader for this job. Again, I really wasn't interested in being clever in the layout / design of the params, so the xml file I came up with is practically brain-dead (but it works :-). It's probably not really a good idea to require the python bindings for libxml2 in order to test libxml2, so it may be worthwhile to convert the param parsing over to some completely different method / file layout. Within the script, the reading / parsing of the params is just a few relatively small classes which result in a Python dictionary containing the parsed params, so changing that part would be relatively simple. There were a couple of more technically challenging items. The basic method used by the script to run the various tests is to use python's os.popen3 (run a task using pipe files for stdin, stdout and stderr). The "task" can be xmllint, testXPath, testHTML or whatever (controlled by the parameter file). Unfortunately, there is a problem in Python when trying to read the supplied file descriptors for stdout and stderr - if lots of data is written to one descriptor before anything is written to the other, the read within the script can "block" waiting for the task to write data to one descriptor, while the task blocks waiting for the script to read the data it has written so far to the other descriptor. Since some of the tests write lots of data to stdout and nothing to stderr, while other tests write lots of data to stderr and little or nothing to stdout, one cannot arbitrarily choose one of them to read first. A good solution would be to use Python's version of the select() function, but unfortunately that's apparently not available under "some operating systems" (which, according to the docs, includes Windows). I finally had to settle on some very basic multi-threading, with one thread handling stdout and another thread handling stderr. For comparing the actual vs. expected results on each test, Python has a "difflib" module which is really ideal for what we want to do. I started out using it (very successfully), but while testing the SAX routines I got into some situation where it still worked perfectly on Linux, but on the Windows implementation the difflib routine went into a dead loop :-(. So, I abandoned that and just did a simple compare of the results. That works fine as long as they are equal, it just isn't very nice when they are different (i.e. if the test fails). Anyway, feedback is welcome- criticisms, additions, enhancements, even downright insults will all be appreciated! Regards, Bill
Attachment:
regressions.tgz
Description: application/gzip-compressed