Re: [xml] HTML-parser: encoding?

[Elizabeth Mattijsen <liz dijkmat nl>]

At 09:27 PM 11/29/01 +0100, Melvyn Sopacua wrote:
> In your case, HTML is fetched via the network, and you have the advantage 
> of HTTP headers.

Unfortunately, these files reside on disk and don't have any header info 
anymore  ;-(


>  As I understand it, you buffer the incoming streams, which allows you to 
> build an encoding map, which can be as simple as a textfile consisting of 
> "filename","encoding"\n.
> If the HTTP headers don't provide information, the meta tags can be read.

That's what the HTML-parser is already doing, isn't it?


> As a last resort, you can make a default based on originating country 
> (REMOTE_ADDR or formfield) - this is the experience I'm refering to. At 
> some point you need a default and can pass that to iconv. Iconv breaks 
> when it encounters an invalid character and gives accurate positions. So 
> the C-source should provide you with some pointers on how to catch this 
> exception. There's also the Text::Iconv module. I have something lying 
> around somewhere, which used that module. I'll check if it's useful when I 
> get home.

The point is that I'm not really interested in a 100% correct result.  I'm 
talking 500K+ documents here for this particular case (although the script 
I'm working on should be usable by anyone).  I don't want to manually edit 
each one of those that has some error in it.  We're trying to get as much 
out of it with as little (manual) effort possible...

> > Or maybe xmllint could need an extra parameter to transform any 
> > characters not legal in the encoding of the document, to be replaced by 
> > another character.  That would make it more general...
> Hmm. It does have it's advantages.

Changing it to numeric entities would actually be best, as it wouldn't lose 
any information.  Hmm...  but can you actually do that?  Wouldn't the next 
time you read this into an xml parser, re-create the encoding error again 
(having the entity processed)?


> > I don't agree with the pre-processing, but _can_ agree with the iconv 
> > post-processing.  Would be nice if it would be part of xmllint, though...
> Which would need to link iconv. I can already picture the many headaches 
> that will cause Daniel - since that is the number 1 problem on the 
> Sablotron XSL list and very badly handled by the auto* and libtool 
> packages :-).
> But then again - that shouldn't be a reason to not implement it, if it's 
> as useful as it looks to be.

Hmmm... availability of iconv is not guaranteed on a lot of platforms, 
isn't it?  Which would be a reason for me not to use this, as the script 
should be generally usable.


Elizabeth Mattijsen