Ok, I've found a workaround for this behavior.
1. It seems that the iconv returns errno ERANGE in case of
that german umlaut character, if source encoding is set to
UTF-8. So I added this ERANGE error code to the last if block "if (errno
== EINVAL
|| errno
== EILSEQ
|| errno
== ERANGE)".
In that case we do not run into an infinite loop.
But this is not the correct solution because the result
string contains a question mark instead the correct german
umlaut, so I take a look at the source charset which are passed
to the iconv function and found that the
only source charset tried by gmime was UTF-8. The gmime library
tries to get the local_charset by calling setlocale in
gmime-charset.c (g_mime_charset_map_init) - but this doesn't
seem to work on my Windows environment. The value returned for
LC_CTYPE is "German_Germany.1252" which results in locale =
"1252" and this is not a valid iconv charset.
As a bugfix for Windows I add the following lines of code to
determine the current charset via glib.
#ifdef G_OS_WIN32
g_get_charset(&locale_charset);
#else
locale = setlocale (LC_ALL, NULL);
.
.
.
#endif