Re: [xml] HTML-parser: encoding?
- From: Melvyn Sopacua <mdev idg nl>
- To: Elizabeth Mattijsen <liz dijkmat nl>
- Cc: xml gnome org
- Subject: Re: [xml] HTML-parser: encoding?
- Date: Thu, 29 Nov 2001 21:27:36 +0100
At 20:51 11/29/2001 +0100, Elizabeth Mattijsen wrote:
At 07:01 PM 11/29/01 +0100, Melvyn Sopacua wrote:
At 15:52 11/29/2001 +0100, you wrote:
I would propose that _if_ the HTML-parser is used _and_ there is _no_
encoding specification found, that libxml _then_ would check all of the
text in the tree for characters illegal for the ISO-Latin-1 encoding and
replace these with spaces (so that the size of the buffer used is not changed).
Personally, I think that would be quite expensive...
Expensive in what way? I always thought that libxml was made for complete
functionality, not speed. And it would only happen _if_ you are using the
HTML-parser _and_ no encoding information was found.
Agreed. But what would the overhead be, of checking every character in a
document - not alone speed, but memory wise. Anyway - that would be for
Daniel to answer.
..., while there are utils out there, that can pre-process such files.
In any case - it would break with the infamous standards violation by
Microsoft and it's implementation of the 'curly quotes' which often turn
up in HTML documents deriving from MS Word files (ASCII character range
128-159). Iconv doesn't handle this, for one.
But how can you pre-process reliably if you don't know the encoding of the
document? E.g. if a document is encoded in UTF-16, how are you sure that a
$document =~ s/[\x00-\x08\x0b-\x1f\x80-\x9f]/ /sg;
(in Perl speak) would not affect certain valid characters?
In your case, HTML is fetched via the network, and you have the advantage
of HTTP headers. As I understand it, you buffer the incoming streams, which
allows you to build an encoding map, which can be as simple as a textfile
consisting of "filename","encoding"\n.
If the HTTP headers don't provide information, the meta tags can be read.
As a last resort, you can make a default based on originating country
(REMOTE_ADDR or formfield) - this is the experience I'm refering to. At
some point you need a default and can pass that to iconv. Iconv breaks when
it encounters an invalid character and gives accurate positions. So the
C-source should provide you with some pointers on how to catch this
exception. There's also the Text::Iconv module. I have something lying
around somewhere, which used that module. I'll check if it's useful when I
This is probably a better solution, since I'm certain there are documents
out there, which are encoding B, but - because it's the default setting
in the HTML editor - encoding A is specified. This would mean, that every
HTML document should first be parsed for encoding errors, regardless of
the encoding specification.
Or maybe xmllint could need an extra parameter to transform any characters
not legal in the encoding of the document, to be replaced by another
character. That would make it more general...
Hmm. It does have it's advantages.
Even if Daniel would choose to implement it, I would opt for underscores
or a question-mark instead of spaces. But it's not a clean solution, to a
problem that is IMHO outside of the scope for the library and can easily
be corrected by a pre-processing filter into a more elegant solution,
adjustable by analyzing the experience of handled input.
I don't agree with the pre-processing, but _can_ agree with the iconv
post-processing. Would be nice if it would be part of xmllint, though...
Which would need to link iconv. I can already picture the many headaches
that will cause Daniel - since that is the number 1 problem on the
Sablotron XSL list and very badly handled by the auto* and libtool packages
But then again - that shouldn't be a reason to not implement it, if it's as
useful as it looks to be.
If it applies, where it applies - this email is a personal
contribution and does not reflect the views of my employer
] [Thread Prev