Re: [xml] XML regression test cases...

["Jain, Nilesh" <nilesh jain intel com>]

On Tue, 2005-10-11 at 17:25 -0400, Daniel Veillard wrote:
> On Tue, Oct 11, 2005 at 02:01:56PM -0700, Jain, Nilesh wrote:
> > On Tue, 2005-10-11 at 15:15 -0400, Daniel Veillard wrote:
> >
> > >   I will try to take an example putting this in parallel. If you were testing
> > > the Mozilla rendering on a set of complex HTML pages, would you consider 
> > > failure to match an exact rendering at the pixel level for those pages
> > > to be wrong. Obviously no! This is the same for libxml2 processing on some 
> > > complex processin/transformations, it will usually give exactly the same
> > > output on two runs in different environemnt, but failure to do so doesn't
> > > mean it's broken.
> >
> > I understand that and agree with you but one thing which still bothers
> > me is when I run these regression test from 2.20 against code base of
> > 2.22 and see around 51 test errors. Theoretically I should not see
> > errors if ABI behavior is constant and running in same environment. Am I
> > correct or missing something?
> >
> > Why I am looking from that point of view is if I include these test
> > cases into LSB runtime for conformance, and let say spec is based on
> > 2.22 and distro is running 2.24/2.25.. test cases which is confirming
> > the specs should not fail in same environment. 
> >
> > BTW the error messages I get when I run 2.20 against 2.22 code base is:
> > ## XML regression tests
> > File ./test/xhtml1 generated an error
>
>   Please look at the difference between the ./test/xhtml1 output with 2.20
> against what is expected from 2.22
>   This was discussed on this list during the month of August IIRC.

I tried to find the discussion but couldn't, could you please send me
the pointer.

I see a slight difference between input (&t1) and output (<) file, when
I compare against the newer version, but still not clear about this
change.

> > ## SAX1 callbacks regression tests
> > Got a difference for ./test/ent2
> > File ./test/ent2 generated an error
>
>   Same thing. Could you make a diff and check by yourselves ?

I don't see any change in ent2 input file between 2.6.20 and 2.6.22, but
behavior is different.

> > Got a difference for ./test/ent7
>
>   I saw those changes, and validated them, otherwise they would not have gone
> in CVS. I take a user viewpoint when I make those decisions, if you think
> they should not have gone in, please explain why :-)

I don't see any change in the input file for ent7 also.

> > ## Schemas regression tests
> > Error for ./test/schemas/any3_0.xml on ./test/schemas/any3_0.xsd failed
> > Error for ./test/schemas/bug303566_1.xml
> > on ./test/schemas/bug303566_1.xsd failed
>
>   yes the XML Schemas implementation improved between those 2 versions.
> Don't worry, even Microsoft is being fixing their XML Schemas implementation
> as you can see on xmlschemas-dev w3 org mailing list !
> And the complexity of the spec even warrant the garantee that you will never
> see change in interpretation of the spec, there are parts nobody is 100% sure
> about. This doesn't mean XSD support in libxml2 is not useful, nor that
> peopel are not using it.

This what worries me most. As mentioned earlier, stability of ABI
behavior is big concern for LSB, if it changes between minor version,
then it become very difficult for LSB certification to change. This can
be a big issues because if LSB based it specification and runtime test
on 2.6.20 and let us say Suse picks 2.6.21 and Mandirva 2.6.22 in there
release for LSB certification, then it will fail because ABI behavior is
not compatible.. I hope I have articulated the problem correctly.

Another aspect I am looking at is the usage of model these ABI.. how
application uses these ABI.. Does application really care about these
changes? could you help me understand that, or send me some pointer to
explore that. 

BTW: If somewhere I talking absurd, please accept my apologies as being
novice about XML.

Thanks,
Nilesh