<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Helvetica Neue";
panose-1:2 0 5 3 0 0 0 2 0 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi Yuval,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It is the correct method to use, however, you need to specify a list of charsets that it should even attempt to try.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What you need to do is:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">static const char **charsets = { “big5”, “shift-jis”, “euc-jis”, “cp1255”, NULL };<o:p></o:p></p>
<p class="MsoNormal">options = g_mime_parser_options_clone (NULL);<o:p></o:p></p>
<p class="MsoNormal">g_mime_parser_options_set_fallback_charsets (options, charsets);<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Then pass those options into decode_8bit().<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hope that helps,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Jeff<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">gmime-devel-list <gmime-devel-list-bounces@gnome.org> on behalf of Yuval Peduel via gmime-devel-list <gmime-devel-list@gnome.org><br>
<b>Reply-To: </b>Yuval Peduel <ypeduel@yahoo-inc.com><br>
<b>Date: </b>Wednesday, August 9, 2017 at 2:00 PM<br>
<b>To: </b>"gmime-devel-list@gnome.org" <gmime-devel-list@gnome.org><br>
<b>Subject: </b>[gmime-devel] determining encodings<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black">Most messages with subjects and From: headers using characters outside the ASCII set now use the RFC-2047 encoding to keep the actual
bytes in the message "7-bit safe". But there are still a significant number of messages coming in which use national encoding: big5 from China, Taiwan, and Singapore; EUC-JIS and shift-JIS from Japan; cp1255 from Israel; etc.<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black">What is the best way to convert these strings into UTF-8?<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black">Since these contain 8-bit characters, I tried using g_mime_utils_decode_8bit with a NULL encoding, assuming it would determine the
best one to use. But in my test, this didn't work at all. (My test consisted of:<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black">- starting with one UTF-8 string for each of 4 encodings, the equivalent of <o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"> - "Happy New Year" in Chinese (big5<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"> - "Good Morning" for shift-JIS<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"> - "Good Evening" for EUC-JIS<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"> - "Peace unto you" for cp1255<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black">- I converted the UTF-8 to a byte sequence using the corresponding encoding.<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black">- I then fed the four resulting byte sequences to g_mime_utils_decode_8bit and wrote out the results<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black">I confirmed that the input to g_mime_utils_decode_8bit were correctly encoded by decoding them with the proper decoding.<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black">So:<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black">1. is g_mime_utils_decode_8bit the right tool for the job? I assume it works properly when one actually knows the encoding, but
when one doesn't?<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black">2. if so, how should I be using it, because:<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"> <b id="yui_3_16_0_ym19_1_1502300068142_11808">output_ptr = g_mime_utils_decode_8bit(NULL, input_ptr, input_length);</b><o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"> isn't doing it.<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black">3. if it isn't, what is the right way?<o:p></o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div id="yui_3_16_0_ym19_1_1502300068142_11392">
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Helvetica Neue",sans-serif;color:black">TIA.<o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>