Re: [xml] HTML-parser: encoding?

[Melvyn Sopacua <mdev idg nl>]

At 20:51 11/29/2001 +0100, Elizabeth Mattijsen wrote:

> At 07:01 PM 11/29/01 +0100, Melvyn Sopacua wrote:
> > At 15:52 11/29/2001 +0100, you wrote:
> > > I would propose that _if_ the HTML-parser is used _and_ there is _no_ 
> > > encoding  specification found, that libxml _then_ would check all of the 
> > > text in the tree for characters illegal for the ISO-Latin-1 encoding and 
> > > replace these with spaces (so that the size of the buffer used is not changed).
> > Personally, I think that would be quite expensive...
>
> Expensive in what way?  I always thought that libxml was made for complete 
> functionality, not speed.  And it would only happen _if_ you are using the 
> HTML-parser _and_ no encoding information was found.

Agreed. But what would the overhead be, of checking every character in a 
document - not alone speed, but memory wise. Anyway - that would be for 
Daniel to answer.

> > ..., while there are utils out there, that can pre-process such files.
> >  In any case - it would break with the infamous standards violation by 
> > Microsoft and it's implementation of the 'curly quotes' which often turn 
> > up in HTML documents deriving from MS Word files (ASCII character range 
> > 128-159). Iconv doesn't handle this, for one.
>
> But how can you pre-process reliably if you don't know the encoding of the 
> document?  E.g. if a document is encoded in UTF-16, how are you sure that a
>
>   $document =~ s/[\x00-\x08\x0b-\x1f\x80-\x9f]/ /sg;
>
> (in Perl speak) would not affect certain valid characters?

You don't.
In your case, HTML is fetched via the network, and you have the advantage 
of HTTP headers. As I understand it, you buffer the incoming streams, which 
allows you to build an encoding map, which can be as simple as a textfile 
consisting of "filename","encoding"\n.
If the HTTP headers don't provide information, the meta tags can be read.
As a last resort, you can make a default based on originating country 
(REMOTE_ADDR or formfield) - this is the experience I'm refering to. At 
some point you need a default and can pass that to iconv. Iconv breaks when 
it encounters an invalid character and gives accurate positions. So the 
C-source should provide you with some pointers on how to catch this 
exception. There's also the Text::Iconv module. I have something lying 
around somewhere, which used that module. I'll check if it's useful when I 
get home.

> > This is probably a better solution, since I'm certain there are documents 
> > out there, which are encoding B, but -  because it's the default setting 
> > in the HTML editor - encoding A is specified. This would mean, that every 
> > HTML document should first be parsed for encoding errors, regardless of 
> > the encoding specification.
>
> Or maybe xmllint could need an extra parameter to transform any characters 
> not legal in the encoding of the document, to be replaced by another 
> character.  That would make it more general...

Hmm. It does have it's advantages.

> > Even if Daniel would choose to implement it, I would opt for underscores 
> > or a question-mark instead of spaces. But it's not a clean solution, to a 
> > problem that is IMHO outside of the scope for the library and can easily 
> > be corrected by a pre-processing filter into a more elegant solution, 
> > adjustable by analyzing the experience of handled input.
>
> I don't agree with the pre-processing, but _can_ agree with the iconv 
> post-processing.  Would be nice if it would be part of xmllint, though...

Which would need to link iconv. I can already picture the many headaches 
that will cause Daniel - since that is the number 1 problem on the 
Sablotron XSL list and very badly handled by the auto* and libtool packages 
:-).
But then again - that shouldn't be a reason to not implement it, if it's as 
useful as it looks to be.



Best regards,

Melvyn Sopacua
WebMaster IDG.nl
_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
If it applies, where it applies - this email is a personal
contribution and does not reflect the views of my employer
IDG.nl.
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\