Re: [xml] HTML-parser: encoding?

At 20:51 11/29/2001 +0100, Elizabeth Mattijsen wrote:

At 07:01 PM 11/29/01 +0100, Melvyn Sopacua wrote:
At 15:52 11/29/2001 +0100, you wrote:
I would propose that _if_ the HTML-parser is used _and_ there is _no_ encoding specification found, that libxml _then_ would check all of the text in the tree for characters illegal for the ISO-Latin-1 encoding and replace these with spaces (so that the size of the buffer used is not changed).
Personally, I think that would be quite expensive...

Expensive in what way? I always thought that libxml was made for complete functionality, not speed. And it would only happen _if_ you are using the HTML-parser _and_ no encoding information was found.

Agreed. But what would the overhead be, of checking every character in a document - not alone speed, but memory wise. Anyway - that would be for Daniel to answer.

..., while there are utils out there, that can pre-process such files.
In any case - it would break with the infamous standards violation by Microsoft and it's implementation of the 'curly quotes' which often turn up in HTML documents deriving from MS Word files (ASCII character range 128-159). Iconv doesn't handle this, for one.

But how can you pre-process reliably if you don't know the encoding of the document? E.g. if a document is encoded in UTF-16, how are you sure that a

  $document =~ s/[\x00-\x08\x0b-\x1f\x80-\x9f]/ /sg;

(in Perl speak) would not affect certain valid characters?

You don't.
In your case, HTML is fetched via the network, and you have the advantage of HTTP headers. As I understand it, you buffer the incoming streams, which allows you to build an encoding map, which can be as simple as a textfile consisting of "filename","encoding"\n.
If the HTTP headers don't provide information, the meta tags can be read.
As a last resort, you can make a default based on originating country (REMOTE_ADDR or formfield) - this is the experience I'm refering to. At some point you need a default and can pass that to iconv. Iconv breaks when it encounters an invalid character and gives accurate positions. So the C-source should provide you with some pointers on how to catch this exception. There's also the Text::Iconv module. I have something lying around somewhere, which used that module. I'll check if it's useful when I get home.

This is probably a better solution, since I'm certain there are documents out there, which are encoding B, but - because it's the default setting in the HTML editor - encoding A is specified. This would mean, that every HTML document should first be parsed for encoding errors, regardless of the encoding specification.

Or maybe xmllint could need an extra parameter to transform any characters not legal in the encoding of the document, to be replaced by another character. That would make it more general...

Hmm. It does have it's advantages.

Even if Daniel would choose to implement it, I would opt for underscores or a question-mark instead of spaces. But it's not a clean solution, to a problem that is IMHO outside of the scope for the library and can easily be corrected by a pre-processing filter into a more elegant solution, adjustable by analyzing the experience of handled input.

I don't agree with the pre-processing, but _can_ agree with the iconv post-processing. Would be nice if it would be part of xmllint, though...

Which would need to link iconv. I can already picture the many headaches that will cause Daniel - since that is the number 1 problem on the Sablotron XSL list and very badly handled by the auto* and libtool packages :-). But then again - that shouldn't be a reason to not implement it, if it's as useful as it looks to be.

Best regards,

Melvyn Sopacua
If it applies, where it applies - this email is a personal
contribution and does not reflect the views of my employer

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]