Re: [xml] XML regression test cases...

["Jain, Nilesh" <nilesh jain intel com>]

On Tue, 2005-10-11 at 15:15 -0400, Daniel Veillard wrote:

>   I will try to take an example putting this in parallel. If you were testing
> the Mozilla rendering on a set of complex HTML pages, would you consider 
> failure to match an exact rendering at the pixel level for those pages
> to be wrong. Obviously no! This is the same for libxml2 processing on some 
> complex processin/transformations, it will usually give exactly the same
> output on two runs in different environemnt, but failure to do so doesn't
> mean it's broken.

I understand that and agree with you but one thing which still bothers
me is when I run these regression test from 2.20 against code base of
2.22 and see around 51 test errors. Theoretically I should not see
errors if ABI behavior is constant and running in same environment. Am I
correct or missing something?

Why I am looking from that point of view is if I include these test
cases into LSB runtime for conformance, and let say spec is based on
2.22 and distro is running 2.24/2.25.. test cases which is confirming
the specs should not fail in same environment. 

BTW the error messages I get when I run 2.20 against 2.22 code base is:
## XML regression tests
File ./test/xhtml1 generated an error
## XML regression tests on memory
Result for ./test/xhtml1 failed
File ./test/xhtml1 generated an error
## XML entity subst regression tests
File ./test/xhtml1 generated an error
## XML Namespaces regression tests
## Error cases regression tests
## Error cases stream regression tests
## Reader regression tests
## Reader entities substitution regression tests
## Reader on memory regression tests
## Walker regression tests
## SAX1 callbacks regression tests
Got a difference for ./test/ent2
File ./test/ent2 generated an error
Got a difference for ./test/ent7
File ./test/ent7 generated an error
Got a difference for ./test/xml2
File ./test/xml2 generated an error
## SAX2 callbacks regression tests
Got a difference for ./test/ent2
File ./test/ent2 generated an error
Got a difference for ./test/ent7
File ./test/ent7 generated an error
Got a difference for ./test/xml2
File ./test/xml2 generated an error
## XML push regression tests
Result for ./test/xhtml1 failed
File ./test/xhtml1 generated an error
## HTML regression tests
Result for ./test/HTML/Down.html failed
File ./test/HTML/Down.html generated an error
Result for ./test/HTML/attrents.html failed
File ./test/HTML/attrents.html generated an error
Result for ./test/HTML/cf_128.html failed
File ./test/HTML/cf_128.html generated an error
Result for ./test/HTML/doc2.htm failed
File ./test/HTML/doc2.htm generated an error
Result for ./test/HTML/doc3.htm failed
File ./test/HTML/doc3.htm generated an error
Result for ./test/HTML/fp40.htm failed
File ./test/HTML/fp40.htm generated an error
Result for ./test/HTML/liclose.html failed
File ./test/HTML/liclose.html generated an error
Result for ./test/HTML/pre.html failed
File ./test/HTML/pre.html generated an error
Result for ./test/HTML/python.html failed
File ./test/HTML/python.html generated an error
Result for ./test/HTML/test2.html failed
File ./test/HTML/test2.html generated an error
Result for ./test/HTML/test3.html failed
File ./test/HTML/test3.html generated an error
Result for ./test/HTML/wired.html failed
File ./test/HTML/wired.html generated an error
## Push HTML regression tests
Result for ./test/HTML/Down.html failed
File ./test/HTML/Down.html generated an error
Result for ./test/HTML/attrents.html failed
File ./test/HTML/attrents.html generated an error
Result for ./test/HTML/cf_128.html failed
File ./test/HTML/cf_128.html generated an error
Result for ./test/HTML/doc2.htm failed
File ./test/HTML/doc2.htm generated an error
Result for ./test/HTML/doc3.htm failed
File ./test/HTML/doc3.htm generated an error
Result for ./test/HTML/fp40.htm failed
File ./test/HTML/fp40.htm generated an error
Result for ./test/HTML/liclose.html failed
File ./test/HTML/liclose.html generated an error
Result for ./test/HTML/pre.html failed
File ./test/HTML/pre.html generated an error
Result for ./test/HTML/python.html failed
File ./test/HTML/python.html generated an error
Result for ./test/HTML/test2.html failed
File ./test/HTML/test2.html generated an error
Result for ./test/HTML/test3.html failed
File ./test/HTML/test3.html generated an error
Result for ./test/HTML/wired.html failed
File ./test/HTML/wired.html generated an error
## HTML SAX regression tests
Got a difference for ./test/HTML/Down.html
File ./test/HTML/Down.html generated an error
Got a difference for ./test/HTML/attrents.html
File ./test/HTML/attrents.html generated an error
Got a difference for ./test/HTML/cf_128.html
File ./test/HTML/cf_128.html generated an error
Got a difference for ./test/HTML/doc2.htm
File ./test/HTML/doc2.htm generated an error
Got a difference for ./test/HTML/doc3.htm
File ./test/HTML/doc3.htm generated an error
Got a difference for ./test/HTML/fp40.htm
File ./test/HTML/fp40.htm generated an error
Got a difference for ./test/HTML/liclose.html
File ./test/HTML/liclose.html generated an error
Got a difference for ./test/HTML/pre.html
File ./test/HTML/pre.html generated an error
Got a difference for ./test/HTML/python.html
File ./test/HTML/python.html generated an error
Got a difference for ./test/HTML/reg1.html
File ./test/HTML/reg1.html generated an error
Got a difference for ./test/HTML/reg2.html
File ./test/HTML/reg2.html generated an error
Got a difference for ./test/HTML/reg3.html
File ./test/HTML/reg3.html generated an error
Got a difference for ./test/HTML/reg4.html
File ./test/HTML/reg4.html generated an error
Got a difference for ./test/HTML/script.html
File ./test/HTML/script.html generated an error
Got a difference for ./test/HTML/test2.html
File ./test/HTML/test2.html generated an error
Got a difference for ./test/HTML/test3.html
File ./test/HTML/test3.html generated an error
Got a difference for ./test/HTML/wired.html
File ./test/HTML/wired.html generated an error
## Valid documents regression tests
## Validity checking regression tests
## General documents valid regression tests
## XInclude regression tests
## XInclude xmlReader regression tests
## XInclude regression tests stripping include nodes
## XInclude xmlReader regression tests stripping include nodes
## XPath expressions regression tests
## XPath document queries regression tests
## XPointer document queries regression tests
## xml:id regression tests
## URI parsing tests
## URI base composition tests
## Schemas regression tests
Error for ./test/schemas/any3_0.xml on ./test/schemas/any3_0.xsd failed
Error for ./test/schemas/bug303566_1.xml
on ./test/schemas/bug303566_1.xsd failed
Error for ./test/schemas/changelog093_0.xml
on ./test/schemas/changelog093_1.xsd
 failed
Result for ./test/schemas/derivation-ok-extension_0.xml
on ./test/schemas/deriva
tion-ok-extension_0.xsd failed
Error for ./test/schemas/derivation-ok-extension_0.xml
on ./test/schemas/derivat
ion-ok-extension_0.xsd failed
## Relax-NG regression tests
## Relax-NG streaming regression tests
## Pattern regression tests
## C14N with comments regression tests
## C14N without comments regression tests
## C14N exclusive without comments regression tests
## Catalog and Threads regression tests
Total 2557 tests, 51 errors, 0 leaks

I also see one error message while running 2.22 code base against 2.22
code.
## Schemas regression tests
Error for ./test/schemas/derivation-ok-extension_0.xml
on ./test/schemas/derivation-ok-extension_0.xsd failed

Thanks,
Nilesh