Re: Charsets, Gtkmm and MySQL



Hello,

well, first it was hard to figure out where the problem was.

I played around with different charsets on My Linux-Box, on the database
server and the webserver (with my PHP frontend for the same data).

After some (unsuccessful) attempts, I figured out, that everything was
ok on the client and on the server side, also with the conversion
from/to ustring<->strings in My Gtkmm application.

The solution was just be a conversion from html-entities wich are also
utf-8 characters to be valid c-lib utf-8 chars.

I simply used the multibyte-conversion function from php to convert the
data from my html-entities-utf-8 PHP-app to real utf-8 data and stored
them in My DB.

The embrassing thing is, that I should know, that http-form data are not
stored as the same character data, as it is in real applications.
(Never trust what your browser shows you. :-)) )
I think that have to do with the character-conversion-table in PHP.

So I just simply store and load them from My PHP-app as
multibyte-converted-strings. With this conversion, they are also
compatible to My Gtkmm application.

I'm not sure if this is a good way, but it works. :D

Thanks to your suggestions about the operand "<<" ">>" some posts ago.

If someone has a better or other way, feel free, to show Me another
point of view.

Best regards

Bernd


Am Samstag, den 03.01.2009, 15:59 +0100 schrieb Milosz Derezynski:
> How did you fix it? After all this back-and-forth it would be really
> interesting to know.
> 
> Regards,
> M.
> 
> On Sat, Jan 3, 2009 at 11:03 AM, Bernd Robertz
> <bernd robertz cssweb de> wrote:
>         Hello again,
>         
>         I fixed the problem.
>         
>         I know why I don't receive an answer about that. The solution
>         was so
>         easy that it was a bit embrassing. :-)
>         
>         Sorry for bothering and thanks to everyone who helped Me out.
>         
>         Greetings
>         
>         Bernd
>         
>         
>         Am Freitag, den 02.01.2009, 14:30 +0100 schrieb Bernd Robertz:
>         
>         > Hello all,
>         >
>         > I hope everyone is well arrived in the new year.
>         >
>         > I come back to my last post.
>         > It seems to be, that I have some weired missconfiguration
>         with my
>         > charsets.
>         > If I'm inserting a row from My gtkmm-app, it will be shown
>         correctly in
>         > My app, but not in other apps like my PHP webfrontend or
>         phpMyAdmin.
>         >
>         > I think I know how to fix it.
>         >
>         > Has anyone suggestions wich charset should generally be used
>         i. e. for
>         > client and server apps (Gtkmm, webserver, db, etc.)?
>         >
>         > Thanks everyone who give Me the flashlight to get the
>         problem.
>         >
>         > Many thanks
>         >
>         > Bernd
>         >
>         >
>         > Am Montag, den 29.12.2008, 13:09 +0100 schrieb Bernd
>         Robertz:
>         > > Hello,
>         > >
>         > > thanks for the code.
>         > >
>         > > result is an object of mysqlpp::StoreQueryResult in the
>         sql-libs.
>         > > It stores the query data.
>         > > But in the docs of libmysql++ is described, that are the
>         stored data
>         > > handeld as a derived std::string. Also compatible to the
>         original
>         > > std::string.
>         > >
>         > > I inserted your code as you described and it returns the
>         following
>         > > output (2 textviews as example, one with shown data, one
>         with empty)
>         > > The output is taken from console:
>         > >
>         > > :::::::::::::::::: SNIP first example
>         data :::::::::::::::::
>         > >
>         > > ** Message: UTF-8 valid: 0: Demozugang ist der gleiche.
>         > >
>         > > Auch f\xfcr gesch\xe4ftliche Aktivit\xe4ten (Streeting
>         > > Unternehmensvertrieb, siehe Unterlagen).
>         > >
>         > > Vertriebsnummer: 00028
>         > >
>         > > Vertriebsseite:
>         > > URL: http://vertrieb.streeting.com
>         > > Benutzer: xxxx
>         > > Kennwort: xxxx
>         > >
>         > > Abrechnungssystem (im Vertriebslogin):
>         > > Benutzer: xxxx
>         > > Kennwort: xxxx
>         > >
>         > > :::::::::::::::::: SNIP second example
>         data :::::::::::::::::
>         > >
>         > > ** Message: UTF-8 valid: 1: Daten CySoTec:
>         > > --------------
>         > >
>         > > Jabba (XMPP) Messenger Client:
>         > > Benutzer: xxxx
>         > > Kennwort: xxxx
>         > > Server: cysotec.info
>         > > Jabba ID: xrobbi cysotec info
>         > > [...]
>         > >
>         > > :::::::::::::::::: SNIP example data end :::::::::::::::::
>         > >
>         > > As far as I can see, the one with empty data is not UTF-8
>         valid
>         > > and the other is.
>         > > So, what can I do, to convert the Data from the SQL server
>         to My app in
>         > > a valid UTF-8, or on the other hand,
>         > > what collation should I use in on My SQL server?
>         > >
>         > > Thank you alot
>         > >
>         > > Bernd
>         > >
>         > >
>         > > Am Montag, den 29.12.2008, 00:47 +0100 schrieb Milosz
>         Derezynski:
>         > > > Of what type is 'result'?
>         > > >
>         > > > Also, could you add, before this line:
>         > > >
>         > > >    tmp.push_back((std::string)result[row][i]);
>         > > >
>         > > > this code:
>         > > >
>         > > >    Glib::ustring utf8_str(result[row][i]);
>         > > >    g_message("UTF-8 valid: %d: %s",
>         int(utf8_str.validate()),
>         > > > utf8_str.c_str());
>         > > >
>         > > > and see what it gives?
>         > > >
>         > > >
>         > > > M.
>         > > >
>         > > > P.S.:
>         > > > Yes std::string and ustring are generally compatible,
>         with the main
>         > > > differences being that ustring works by-character not
>         by-byte (e.g.
>         > > > find(), operator[], etc), ustring has methods for
>         validation of UTF-8
>         > > > and related functionality; the main potential culprit
>         here is
>         > > > operator<< and operator>> of ustring. I had really big
>         problems with
>         > > > these using boost::format() until i realised that
>         internally, it's a
>         > > > stream to which the args are fed using operator<<() of
>         ustring.
>         > > >
>         > > >
>         > > > On Mon, Dec 29, 2008 at 12:01 AM, Bernd Robertz
>         > > > <bernd robertz cssweb de> wrote:
>         > > >         Thanks alot for your fast reply,
>         > > >
>         > > >         I wrote a simple MySQL-Class wich is derived
>         from the libsql++
>         > > >         class.
>         > > >
>         > > >         In this class I used std::string to get the
>         columns of a row.
>         > > >         The columns (strings) are stored in a
>         vector<string> wich I
>         > > >         return to
>         > > >         Glib::ustring in My treeview-data-window.
>         > > >
>         > > >         Hiere are some snippets:
>         > > >
>         > > >         ::::::::::::::::::::::::
>         SNIP ::::::::::::::::::::::
>         > > >         database.cpp
>         > > >         (My simple MySQL database class.)
>         > > >
>         > > >         DataBase::Database()
>         > > >         {
>         > > >                // db-connect
>         > > >                // query-setup
>         > > >                // other connect related stuff
>         > > >
>         > > >                mysqlpp::StoreQueryResult result; //
>         Setup the result
>         > > >         object
>         > > >                int rows; // Keep track of result counts
>         > > >         }
>         > > >
>         > > >         void DataBase::returnrow(int dataid)
>         > > >         {
>         > > >                // Setup db-query with dataid to get
>         requested db-row
>         > > >                //
>         > > >                mysqlpp::Query query =
>         dbcon.query(queries.alldata); //
>         > > >         DB Query
>         > > >                result = query.store(); // Store the
>         results
>         > > >                rows = result.num_rows(); // Set the
>         number of results
>         > > >         }
>         > > >
>         > > >         std::vector<std::string> DataBase::fetchdata(int
>         row)
>         > > >         {
>         > > >                std::vector<std::string> tmp; // Temp
>         vector for return
>         > > >
>         > > >                for (int i = 0; i <
>         result.field_names()->size(); ++i)
>         > > >                {
>         > > >                        // Every column will be stored in
>         the temp
>         > > >         vector
>         > > >
>          tmp.push_back((std::string)result[row][i]);
>         > > >                }
>         > > >                return tmp; // Return the temp-vector
>         with the
>         > > >         [columns]=data
>         > > >         }
>         > > >
>         > > >         ------------
>         > > >
>         > > >         datawindow.cpp
>         > > >         (The window in where the data is displayed after
>         dbl-click on
>         > > >         treeview)
>         > > >
>         > > >         DataWindow::DataWindow()
>         > > >         {
>         > > >                // Widgets setup
>         > > >                // Database connect via DataBase-Object
>         > > >         }
>         > > >
>         > > >         void DataWindow::getid(int dataid)
>         > > >         {
>         > > >                // From the treeview I call this method
>         to set the
>         > > >                // requested row-id
>         > > >                mysql.returnrow(dataid);
>         > > >
>         > > >                // Call this method to setup the entries
>         and textview
>         > > >                setdata();
>         > > >         }
>         > > >
>         > > >         void DataWindow::setdata()
>         > > >         {
>         > > >                l_dataid.set_text(mysql.fetchdata(0)[0]);
>         > > >                Gtk::TreeModel::Row selrow =
>         *(ls_cbox->prepend());
>         > > >                selrow[c_mod.c_cat] =
>         mysql.fetchdata(0)[1];
>         > > >                cbox.set_active(0);
>         > > >
>         > > >                // Here comes the problem, I get every
>         column from My
>         > > >                // database-object wich I store in the
>         entries.
>         > > >                e_name.set_text(mysql.fetchdata(0)[2]);
>         > > >                e_number.set_text(mysql.fetchdata(0)[3]);
>         > > >                e_user.set_text(mysql.fetchdata(0)[4]);
>         > > >                e_pass.set_text(mysql.fetchdata(0)[5]);
>         > > >                e_mail.set_text(mysql.fetchdata(0)[6]);
>         > > >                e_url.set_text(mysql.fetchdata(0)[7]);
>         > > >                e_other.set_text(mysql.fetchdata(0)[8]);
>         > > >
>         > > >                // Here we have the main problem: the
>         textview-buffer
>         > > >                // Some rows are shown here, some not.
>         > > >                b_descr->set_text(mysql.fetchdata(0)[9]);
>         > > >         }
>         > > >
>         > > >         ::::::::::::::::::::::::: SNIP
>         > > >         END ::::::::::::::::::::::::::::
>         > > >
>         > > >         That is the main code where the problem occours.
>         > > >
>         > > >         The DataWindow is called on doubleclick on a
>         treeview-row.
>         > > >         With the call of the DataWindow I call the
>         DataView::getid()
>         > > >         wich receive the database row-id.
>         > > >         Then I fill with the DataView::setdata() method
>         the widgets
>         > > >         in the new window.
>         > > >
>         > > >         Some chars are, as I told, glyphs, but the
>         textbuffer is the
>         > > >         main
>         > > >         problem.
>         > > >
>         > > >         As far as I know, the entries and textbuffers
>         are expecting
>         > > >         Glib::ustring, but I thought the conversion from
>         std::string
>         > > >         to
>         > > >         Glib::ustring is quite compatible between these
>         types.
>         > > >         I used std::string in the database class,
>         because I wanted to
>         > > >         isolated
>         > > >         it from Gtkmm for other not Gtkmm-related
>         projects.
>         > > >
>         > > >         Thank you for helping.
>         > > >
>         > > >         Bernd
>         > > >
>         > > >
>         > > >         Am Sonntag, den 28.12.2008, 22:26 +0100 schrieb
>         Milosz
>         > > >         Derezynski:
>         > > >
>         > > >         > Are you using Glib::ustring anywhere in the
>         process of
>         > > >         getting the
>         > > >         > data from the SQL database to the view?
>         > > >         >
>         > > >         > Glib::ustring implicitly (and in my opinion
>         still at least
>         > > >         not well
>         > > >         > enough documented/hinted at) converts from
>         UTF-8
>         > > >         > to the current locale with operators << and
>         >>. If you are
>         > > >         using a
>         > > >         > stringstream anywhere in the process, or are
>         > > >         > using operator<< or operator>> in any other
>         way, this could
>         > > >         then be
>         > > >         > the cause of the problem.
>         > > >         >
>         > > >         > Otherwise, and in the above described case too
>         actually it
>         > > >         would help
>         > > >         > very much if you could post (if possible, all
>         of)
>         > > >         > the relevant code online.
>         > > >         >
>         > > >         > Regards,
>         > > >         > M.
>         > > >         >
>         > > >         > On Sun, Dec 28, 2008 at 8:53 PM, Bernd Robertz
>         > > >         > <bernd robertz cssweb de> wrote:
>         > > >         >         Hello list,
>         > > >         >
>         > > >         >         I'm not really sure, but I hope My
>         question is in
>         > > >         the right
>         > > >         >         place here.
>         > > >         >
>         > > >         >         I wrote a simple database app like
>         this:
>         > > >         >
>         > > >         >         Connect to a MySQL server via libmysql
>         ++,
>         > > >         >         and then list the rows in a treeview.
>         > > >         >         Doubleclick on a row shows a window
>         with some
>         > > >         entries and a
>         > > >         >         textview
>         > > >         >         with the content of the selected row.
>         > > >         >         The textview is associated with a type
>         text in the
>         > > >         DB.
>         > > >         >
>         > > >         >         Everything is fine.
>         > > >         >
>         > > >         >         So but I ran really into problems with
>         the charsets.
>         > > >         >         german umlaute (ö, ä, etc.) and some
>         other
>         > > >         characters are
>         > > >         >         glyphs in the
>         > > >         >         treeview. Ok, thats not the main
>         problem.
>         > > >         >
>         > > >         >         That is: The entries in the detailed
>         view still
>         > > >         shows the
>         > > >         >         glyphs for
>         > > >         >         some chars like ö, ä.
>         > > >         >         But on some rows, the represented
>         textbuffer for
>         > > >         textview is
>         > > >         >         not
>         > > >         >         visible. Wich means the textview is in
>         some rows
>         > > >         empty.
>         > > >         >
>         > > >         >         I run My app from a console and I got
>         these error
>         > > >         messages:
>         > > >         >
>         > > >         >         :::::::::
>         > > >         >         Gtk-CRITICAL **:
>         gtk_text_buffer_emit_insert:
>         > > >         assertion
>         > > >         >         `g_utf8_validate
>         > > >         >         (text, len, NULL)' failed
>         > > >         >         :::::::::
>         > > >         >
>         > > >         >         In other rows, the textbuffer is
>         correct shown,
>         > > >         exepct some
>         > > >         >         glyphs.
>         > > >         >
>         > > >         >         If I print the textbuffer as a
>         simple .c_str() in a
>         > > >         console it
>         > > >         >         will be
>         > > >         >         shown correctly (inklusive ä, ö,
>         ü...).
>         > > >         >         (Tried to show the textview as
>         a .c_str(), without
>         > > >         any
>         > > >         >         effect.)
>         > > >         >
>         > > >         >         The SQL connection is
>         serverside-configured as UTF-8
>         > > >         Unicode.
>         > > >         >         The SQL table collation is UTF-8
>         General CI (Case
>         > > >         >         Insensitive).
>         > > >         >         The Application runs on a DE_de Euro
>         (ISO-8859-15).
>         > > >         >
>         > > >         >         All characters are shown correctly in
>         a PHP app wich
>         > > >         does the
>         > > >         >         same, just
>         > > >         >         only online. :-)
>         > > >         >
>         > > >         >         I hope someone could turn on a light
>         above my head,
>         > > >         why I
>         > > >         >         don't get the
>         > > >         >         right charactes.
>         > > >         >
>         > > >         >         Thanks alot in advance
>         > > >         >
>         > > >         >         Bernd
>         > > >         >
>         > > >         >
>         > > >         >
>         _______________________________________________
>         > > >         >         gtkmm-list mailing list
>         > > >         >         gtkmm-list gnome org
>         > > >         >
>         http://mail.gnome.org/mailman/listinfo/gtkmm-list
>         > > >         >
>         > > >         >
>         > > >         >
>         > > >         > --
>         > > >         > Please note that according to the German law
>         on data
>         > > >         retention,
>         > > >         > information on every electronic information
>         exchange with me
>         > > >         is
>         > > >         > retained for a period of six months.
>         > > >         > [Bitte beachten Sie, dass dem Gesetz zur
>         > > >         Vorratsdatenspeicherung
>         > > >         > zufolge
>         > > >         > jeder elektronische Kontakt mit mir sechs
>         Monate lang
>         > > >         gespeichert
>         > > >         > wird.]
>         > > >
>         > > >
>         > > >
>         > > >
>         > > >
>         > > > --
>         > > > Please note that according to the German law on data
>         retention,
>         > > > information on every electronic information exchange
>         with me is
>         > > > retained for a period of six months.
>         > > > [Bitte beachten Sie, dass dem Gesetz zur
>         Vorratsdatenspeicherung
>         > > > zufolge
>         > > > jeder elektronische Kontakt mit mir sechs Monate lang
>         gespeichert
>         > > > wird.]
>         > >
>         > > _______________________________________________
>         > > gtkmm-list mailing list
>         > > gtkmm-list gnome org
>         > > http://mail.gnome.org/mailman/listinfo/gtkmm-list
>         >
>         > _______________________________________________
>         > gtkmm-list mailing list
>         > gtkmm-list gnome org
>         > http://mail.gnome.org/mailman/listinfo/gtkmm-list
>         
>         _______________________________________________
>         gtkmm-list mailing list
>         gtkmm-list gnome org
>         http://mail.gnome.org/mailman/listinfo/gtkmm-list
>         
> 
> 
> 
> -- 
> Please note that according to the German law on data retention,
> information on every electronic information exchange with me is
> retained for a period of six months.
> [Bitte beachten Sie, dass dem Gesetz zur Vorratsdatenspeicherung
> zufolge
> jeder elektronische Kontakt mit mir sechs Monate lang gespeichert
> wird.]



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]