Re: [xml] UTF8ToHtml entity handling problem

[Daniel Veillard <veillard redhat com>]

On Fri, Nov 17, 2006 at 02:54:56PM -0600, R. Steven Rainwater wrote:
> In the process of doing maintenance on the mod_virgule (advogato.org)
> codebase, I've run across what I think is a minor bug in the libxml2
> function, UTF8ToHtml(). I've checked the bug database and didn't see any
> reports on this. I thought it would be a good idea to post some
> background on how I'm using UTF8ToHtml() before filing a bug report in
> case I'm misunderstanding the intention of this function. 
>
> Mod_virgule accepts HTML form submissions encoded as UTF-8 data. This
> data often includes HTML markup as well as various international
> characters. The data must be must be processed by older mod_virgule code
> that can handle only ASCII, not UTF-8 data. So the raw UTF-8 data is
> passed through UTF8ToHtml() to convert it to ASCII HTML with entity
> encoding of non-ASCII characters.
>
> This works great when the input is English or most languages that use a
> Latin character set. But when valid HTML markup encoded as UTF-8
> contains more exotic characters, such as Han ideographs, it causes
> UTF8ToHtml() to fail, returning an error code of -2. This is unexpected
> since the input was valid UTF-8.
>
> I examined the code of the UTF8ToHtml() function and discovered that it
> fails with error -2 because the input contains UTF-8 characters for
> which libxml2 does not know a named entity value (e.g. "É").
> Since there are tens of thousdands of possible UTF-8 characters and
> libxml2 only knows names for a couple of hundred, this seems to suggest
> UTF8ToHtml() will fail most of the time if the input includes non-Latin
> character sets.
>
> By making a trivial change to the code in UTF8ToHtml(), I was able to
> correct this behavior. When a named entity value cannot be found in the
> internal libxml2 entity table, a numeric entity value (e.g. "兡")
> is used instead. 
>
> Here's the original code where the problem lies:
>
>           /*
>            * Try to lookup a predefined HTML entity for it
>            */
>
>           ent = htmlEntityValueLookup(c);
>           if (ent == NULL) {
>               /* no chance for this in Ascii */
>               *outlen = out - outstart;
>               *inlen = processed - instart;
>               return(-2);
>           }
>
> And here's the same piece of my revised code that seems to have fixed
> the problem:
>
>             /*
>              * Try to lookup a predefined HTML entity for it
>              */
>  
>             ent = htmlEntityValueLookup(c);
>             if (ent == NULL) {
>               snprintf(nbuf, sizeof(nbuf), "#%u", c);
>               cp = nbuf;
>             }
>
>
> I can file a bug report and attach a full patch for this if desired.
> Otherwise, maybe somebody can explain where I've gone wrong. Thanks!

  No that sounds just right ! Using a character reference (technically
"兡" is not an entity reference :-) is the right thing to do there.
Patch would be gratefully accepted, I will just need to check effect on
libxml2 and libxslt regression tests.

Daniel

-- 
Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
veillard redhat com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine  http://rpmfind.net/