[xml] Another encoding problem when not using iconv
- From: <mark itzcovitz ntlworld com>
- To: xml gnome org
- Subject: [xml] Another encoding problem when not using iconv
- Date: Wed, 30 Jul 2003 14:54:24 +0000
There is a problem in the UTF8ToUTF16xx conversion routines in encoding.c. The last parameter is a pointer to
the length of the input, which xmlCharEncOutFunc expects to be modified to be the number of characters in the
input that have been processed. Other routines such as UTF8Toisolat1 do this but the UTF8ToUTF16xx routines
seem to set it to 0. This means that xmlCharEncOutFunc doesn't remove the processed input from the input
buffer.
The problem can be verified by running the attached xml document (converted to utf-16 from slashdot.xml in
the test suite) through a version of xmllint built without iconv - it never ends!
A patch is attached containing changes for UTF8ToUTF16LE and UTF8ToUTF16BE.
-----------------------------------------
Email provided by http://www.ntlhome.com/
ÿþ< ? x m l v e r s i o n = " 1 . 0 " e n c o d i n g = " u t f - 1 6 " ? >
< u l t r a m o d e >
< s t o r y >
< t i t l e > 1 0 0 M b i t / s o n F i b r e t o t h e h o m e < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a r t i c l e s / 9 9 / 0 6 / 0 6 / 1 4 4 0 2 1 1 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 6 1 4 : 3 9 : 5 9 < / t i m e >
< a u t h o r > C m d r T a c o < / a u t h o r >
< d e p a r t m e n t > w o u l d n ' t - i t - b e - n i c e < / d e p a r t m e n t >
< t o p i c > i n t e r n e t < / t o p i c >
< c o m m e n t s > 2 0 < / c o m m e n t s >
< s e c t i o n > a r t i c l e s < / s e c t i o n >
< i m a g e > t o p i c i n t e r n e t . j p g < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > G i m p 1 . 2 P r e v i e w < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a r t i c l e s / 9 9 / 0 6 / 0 6 / 1 4 3 8 2 4 6 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 6 1 4 : 3 8 : 4 0 < / t i m e >
< a u t h o r > C m d r T a c o < / a u t h o r >
< d e p a r t m e n t > s t u f f - t o - r e a d < / d e p a r t m e n t >
< t o p i c > g i m p < / t o p i c >
< c o m m e n t s > 1 2 < / c o m m e n t s >
< s e c t i o n > a r t i c l e s < / s e c t i o n >
< i m a g e > t o p i c g i m p . g i f < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > S o n y ' s A I B O r o b o t S o l d O u t < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a r t i c l e s / 9 9 / 0 6 / 0 6 / 1 4 3 2 2 5 6 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 6 1 4 : 3 2 : 5 1 < / t i m e >
< a u t h o r > C m d r T a c o < / a u t h o r >
< d e p a r t m e n t > s t u f f - t o - s e e < / d e p a r t m e n t >
< t o p i c > t e c h < / t o p i c >
< c o m m e n t s > 1 0 < / c o m m e n t s >
< s e c t i o n > a r t i c l e s < / s e c t i o n >
< i m a g e > t o p i c t e c h 2 . j p g < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > A s k S l a s h d o t : A n o t h e r W o r d f o r
" H a c k e r " ? < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a s k s l a s h d o t / 9 9 / 0 6 / 0 5 / 1 8 1 5 2 2 5 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 5 2 0 : 0 0 : 0 0 < / t i m e >
< a u t h o r > C l i f f < / a u t h o r >
< d e p a r t m e n t > h a c k e r - v s - c r a c k e r < / d e p a r t m e n t >
< t o p i c > n e w s < / t o p i c >
< c o m m e n t s > 3 8 5 < / c o m m e n t s >
< s e c t i o n > a s k s l a s h d o t < / s e c t i o n >
< i m a g e > t o p i c n e w s . g i f < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > 1 0 0 M b i t / s o n F i b r e t o t h e h o m e < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a r t i c l e s / 9 9 / 0 6 / 0 6 / 1 4 4 0 2 1 1 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 6 1 4 : 3 9 : 5 9 < / t i m e >
< a u t h o r > C m d r T a c o < / a u t h o r >
< d e p a r t m e n t > w o u l d n ' t - i t - b e - n i c e < / d e p a r t m e n t >
< t o p i c > i n t e r n e t < / t o p i c >
< c o m m e n t s > 2 0 < / c o m m e n t s >
< s e c t i o n > a r t i c l e s < / s e c t i o n >
< i m a g e > t o p i c i n t e r n e t . j p g < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > G i m p 1 . 2 P r e v i e w < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a r t i c l e s / 9 9 / 0 6 / 0 6 / 1 4 3 8 2 4 6 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 6 1 4 : 3 8 : 4 0 < / t i m e >
< a u t h o r > C m d r T a c o < / a u t h o r >
< d e p a r t m e n t > s t u f f - t o - r e a d < / d e p a r t m e n t >
< t o p i c > g i m p < / t o p i c >
< c o m m e n t s > 1 2 < / c o m m e n t s >
< s e c t i o n > a r t i c l e s < / s e c t i o n >
< i m a g e > t o p i c g i m p . g i f < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > S o n y ' s A I B O r o b o t S o l d O u t < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a r t i c l e s / 9 9 / 0 6 / 0 6 / 1 4 3 2 2 5 6 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 6 1 4 : 3 2 : 5 1 < / t i m e >
< a u t h o r > C m d r T a c o < / a u t h o r >
< d e p a r t m e n t > s t u f f - t o - s e e < / d e p a r t m e n t >
< t o p i c > t e c h < / t o p i c >
< c o m m e n t s > 1 0 < / c o m m e n t s >
< s e c t i o n > a r t i c l e s < / s e c t i o n >
< i m a g e > t o p i c t e c h 2 . j p g < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > A s k S l a s h d o t : A n o t h e r W o r d f o r
" H a c k e r " ? < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a s k s l a s h d o t / 9 9 / 0 6 / 0 5 / 1 8 1 5 2 2 5 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 5 2 0 : 0 0 : 0 0 < / t i m e >
< a u t h o r > C l i f f < / a u t h o r >
< d e p a r t m e n t > h a c k e r - v s - c r a c k e r < / d e p a r t m e n t >
< t o p i c > n e w s < / t o p i c >
< c o m m e n t s > 3 8 5 < / c o m m e n t s >
< s e c t i o n > a s k s l a s h d o t < / s e c t i o n >
< i m a g e > t o p i c n e w s . g i f < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > C o r e l L i n u x F A Q < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a r t i c l e s / 9 9 / 0 6 / 0 5 / 1 8 4 2 2 1 8 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 5 1 8 : 4 2 : 0 6 < / t i m e >
< a u t h o r > C m d r T a c o < / a u t h o r >
< d e p a r t m e n t > s t u f f - t o - r e a d < / d e p a r t m e n t >
< t o p i c > c o r e l < / t o p i c >
< c o m m e n t s > 1 6 4 < / c o m m e n t s >
< s e c t i o n > a r t i c l e s < / s e c t i o n >
< i m a g e > t o p i c c o r e l . g i f < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > U p s i d e d o w n s i d e s M P 3 . C O M . < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a r t i c l e s / 9 9 / 0 6 / 0 5 / 1 5 5 8 2 1 0 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 5 1 5 : 5 6 : 4 5 < / t i m e >
< a u t h o r > C m d r T a c o < / a u t h o r >
< d e p a r t m e n t > s t u f f - t o - t h i n k - a b o u t < / d e p a r t m e n t >
< t o p i c > m u s i c < / t o p i c >
< c o m m e n t s > 4 8 < / c o m m e n t s >
< s e c t i o n > a r t i c l e s < / s e c t i o n >
< i m a g e > t o p i c m u s i c . g i f < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > 2 T e r a b i t s o f B a n d w i d t h < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a r t i c l e s / 9 9 / 0 6 / 0 5 / 1 5 5 4 2 5 8 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 5 1 5 : 5 3 : 4 3 < / t i m e >
< a u t h o r > C m d r T a c o < / a u t h o r >
< d e p a r t m e n t > f a s t e r - p o r n < / d e p a r t m e n t >
< t o p i c > i n t e r n e t < / t o p i c >
< c o m m e n t s > 6 6 < / c o m m e n t s >
< s e c t i o n > a r t i c l e s < / s e c t i o n >
< i m a g e > t o p i c i n t e r n e t . j p g < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > S u p p r e s s i o n o f c o l d f u s i o n
r e s e a r c h ? < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a r t i c l e s / 9 9 / 0 6 / 0 4 / 2 3 1 3 2 0 0 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 4 2 3 : 1 2 : 2 9 < / t i m e >
< a u t h o r > H e m o s < / a u t h o r >
< d e p a r t m e n t > p o s s i b l y - p r o b a b l y < / d e p a r t m e n t >
< t o p i c > s c i e n c e < / t o p i c >
< c o m m e n t s > 2 1 7 < / c o m m e n t s >
< s e c t i o n > a r t i c l e s < / s e c t i o n >
< i m a g e > t o p i c s c i e n c e . g i f < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > C a l i f o r n i a G o v . H a l t s W a g e I n f o
S a l e < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a r t i c l e s / 9 9 / 0 6 / 0 4 / 2 3 5 2 5 6 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 4 2 3 : 0 5 : 3 4 < / t i m e >
< a u t h o r > H e m o s < / a u t h o r >
< d e p a r t m e n t > w o o - h o o ! < / d e p a r t m e n t >
< t o p i c > u s a < / t o p i c >
< c o m m e n t s > 1 6 < / c o m m e n t s >
< s e c t i o n > a r t i c l e s < / s e c t i o n >
< i m a g e > t o p i c u s . g i f < / i m a g e >
< / s t o r y >
< s t o r y >
< t i t l e > R e d H a t A n n o u n c e s I P O < / t i t l e >
< u r l > h t t p : / / s l a s h d o t . o r g / a r t i c l e s / 9 9 / 0 6 / 0 4 / 0 8 4 9 2 0 7 . s h t m l < / u r l >
< t i m e > 1 9 9 9 - 0 6 - 0 4 1 9 : 3 0 : 1 8 < / t i m e >
< a u t h o r > J u s t i n < / a u t h o r >
< d e p a r t m e n t > d e t a i l s - s k e t c h y < / d e p a r t m e n t >
< t o p i c > r e d h a t < / t o p i c >
< c o m m e n t s > 1 5 5 < / c o m m e n t s >
< s e c t i o n > a r t i c l e s < / s e c t i o n >
< i m a g e > t o p i c r e d h a t . g i f < / i m a g e >
< / s t o r y >
< / u l t r a m o d e >
# diff -c encoding_orig.c encoding.c
*** encoding_orig.c Wed Jul 30 14:45:20 2003
--- encoding.c Wed Jul 30 14:47:14 2003
***************
*** 824,829 ****
--- 824,830 ----
{
unsigned short* out = (unsigned short*) outb;
const unsigned char* processed = in;
+ const unsigned char *const instart = in;
unsigned short* outstart= out;
unsigned short* outend;
const unsigned char* inend= in+*inlen;
***************
*** 858,864 ****
else if (d < 0xC0) {
/* trailing byte in leading position */
*outlen = (out - outstart) * 2;
! *inlen = processed - in;
return(-2);
} else if (d < 0xE0) { c= d & 0x1F; trailing= 1; }
else if (d < 0xF0) { c= d & 0x0F; trailing= 2; }
--- 859,865 ----
else if (d < 0xC0) {
/* trailing byte in leading position */
*outlen = (out - outstart) * 2;
! *inlen = processed - instart;
return(-2);
} else if (d < 0xE0) { c= d & 0x1F; trailing= 1; }
else if (d < 0xF0) { c= d & 0x0F; trailing= 2; }
***************
*** 866,872 ****
else {
/* no chance for this in UTF-16 */
*outlen = (out - outstart) * 2;
! *inlen = processed - in;
return(-2);
}
--- 867,873 ----
else {
/* no chance for this in UTF-16 */
*outlen = (out - outstart) * 2;
! *inlen = processed - instart;
return(-2);
}
***************
*** 920,926 ****
processed = in;
}
*outlen = (out - outstart) * 2;
! *inlen = processed - in;
return(0);
}
--- 921,927 ----
processed = in;
}
*outlen = (out - outstart) * 2;
! *inlen = processed - instart;
return(0);
}
***************
*** 1035,1040 ****
--- 1036,1042 ----
{
unsigned short* out = (unsigned short*) outb;
const unsigned char* processed = in;
+ const unsigned char *const instart = in;
unsigned short* outstart= out;
unsigned short* outend;
const unsigned char* inend= in+*inlen;
***************
*** 1069,1075 ****
else if (d < 0xC0) {
/* trailing byte in leading position */
*outlen = out - outstart;
! *inlen = processed - in;
return(-2);
} else if (d < 0xE0) { c= d & 0x1F; trailing= 1; }
else if (d < 0xF0) { c= d & 0x0F; trailing= 2; }
--- 1071,1077 ----
else if (d < 0xC0) {
/* trailing byte in leading position */
*outlen = out - outstart;
! *inlen = processed - instart;
return(-2);
} else if (d < 0xE0) { c= d & 0x1F; trailing= 1; }
else if (d < 0xF0) { c= d & 0x0F; trailing= 2; }
***************
*** 1077,1083 ****
else {
/* no chance for this in UTF-16 */
*outlen = out - outstart;
! *inlen = processed - in;
return(-2);
}
--- 1079,1085 ----
else {
/* no chance for this in UTF-16 */
*outlen = out - outstart;
! *inlen = processed - instart;
return(-2);
}
***************
*** 1128,1134 ****
processed = in;
}
*outlen = (out - outstart) * 2;
! *inlen = processed - in;
return(0);
}
--- 1130,1136 ----
processed = in;
}
*outlen = (out - outstart) * 2;
! *inlen = processed - instart;
return(0);
}
#
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]
