Re: Charsets, Gtkmm and MySQL
- From: Bernd Robertz <bernd robertz cssweb de>
- To: Milosz Derezynski <internalerror gmail com>
- Cc: gtkmm-list gnome org
- Subject: Re: Charsets, Gtkmm and MySQL
- Date: Sat, 03 Jan 2009 16:27:19 +0100
Hello,
well, first it was hard to figure out where the problem was.
I played around with different charsets on My Linux-Box, on the database
server and the webserver (with my PHP frontend for the same data).
After some (unsuccessful) attempts, I figured out, that everything was
ok on the client and on the server side, also with the conversion
from/to ustring<->strings in My Gtkmm application.
The solution was just be a conversion from html-entities wich are also
utf-8 characters to be valid c-lib utf-8 chars.
I simply used the multibyte-conversion function from php to convert the
data from my html-entities-utf-8 PHP-app to real utf-8 data and stored
them in My DB.
The embrassing thing is, that I should know, that http-form data are not
stored as the same character data, as it is in real applications.
(Never trust what your browser shows you. :-)) )
I think that have to do with the character-conversion-table in PHP.
So I just simply store and load them from My PHP-app as
multibyte-converted-strings. With this conversion, they are also
compatible to My Gtkmm application.
I'm not sure if this is a good way, but it works. :D
Thanks to your suggestions about the operand "<<" ">>" some posts ago.
If someone has a better or other way, feel free, to show Me another
point of view.
Best regards
Bernd
Am Samstag, den 03.01.2009, 15:59 +0100 schrieb Milosz Derezynski:
> How did you fix it? After all this back-and-forth it would be really
> interesting to know.
>
> Regards,
> M.
>
> On Sat, Jan 3, 2009 at 11:03 AM, Bernd Robertz
> <bernd robertz cssweb de> wrote:
> Hello again,
>
> I fixed the problem.
>
> I know why I don't receive an answer about that. The solution
> was so
> easy that it was a bit embrassing. :-)
>
> Sorry for bothering and thanks to everyone who helped Me out.
>
> Greetings
>
> Bernd
>
>
> Am Freitag, den 02.01.2009, 14:30 +0100 schrieb Bernd Robertz:
>
> > Hello all,
> >
> > I hope everyone is well arrived in the new year.
> >
> > I come back to my last post.
> > It seems to be, that I have some weired missconfiguration
> with my
> > charsets.
> > If I'm inserting a row from My gtkmm-app, it will be shown
> correctly in
> > My app, but not in other apps like my PHP webfrontend or
> phpMyAdmin.
> >
> > I think I know how to fix it.
> >
> > Has anyone suggestions wich charset should generally be used
> i. e. for
> > client and server apps (Gtkmm, webserver, db, etc.)?
> >
> > Thanks everyone who give Me the flashlight to get the
> problem.
> >
> > Many thanks
> >
> > Bernd
> >
> >
> > Am Montag, den 29.12.2008, 13:09 +0100 schrieb Bernd
> Robertz:
> > > Hello,
> > >
> > > thanks for the code.
> > >
> > > result is an object of mysqlpp::StoreQueryResult in the
> sql-libs.
> > > It stores the query data.
> > > But in the docs of libmysql++ is described, that are the
> stored data
> > > handeld as a derived std::string. Also compatible to the
> original
> > > std::string.
> > >
> > > I inserted your code as you described and it returns the
> following
> > > output (2 textviews as example, one with shown data, one
> with empty)
> > > The output is taken from console:
> > >
> > > :::::::::::::::::: SNIP first example
> data :::::::::::::::::
> > >
> > > ** Message: UTF-8 valid: 0: Demozugang ist der gleiche.
> > >
> > > Auch f\xfcr gesch\xe4ftliche Aktivit\xe4ten (Streeting
> > > Unternehmensvertrieb, siehe Unterlagen).
> > >
> > > Vertriebsnummer: 00028
> > >
> > > Vertriebsseite:
> > > URL: http://vertrieb.streeting.com
> > > Benutzer: xxxx
> > > Kennwort: xxxx
> > >
> > > Abrechnungssystem (im Vertriebslogin):
> > > Benutzer: xxxx
> > > Kennwort: xxxx
> > >
> > > :::::::::::::::::: SNIP second example
> data :::::::::::::::::
> > >
> > > ** Message: UTF-8 valid: 1: Daten CySoTec:
> > > --------------
> > >
> > > Jabba (XMPP) Messenger Client:
> > > Benutzer: xxxx
> > > Kennwort: xxxx
> > > Server: cysotec.info
> > > Jabba ID: xrobbi cysotec info
> > > [...]
> > >
> > > :::::::::::::::::: SNIP example data end :::::::::::::::::
> > >
> > > As far as I can see, the one with empty data is not UTF-8
> valid
> > > and the other is.
> > > So, what can I do, to convert the Data from the SQL server
> to My app in
> > > a valid UTF-8, or on the other hand,
> > > what collation should I use in on My SQL server?
> > >
> > > Thank you alot
> > >
> > > Bernd
> > >
> > >
> > > Am Montag, den 29.12.2008, 00:47 +0100 schrieb Milosz
> Derezynski:
> > > > Of what type is 'result'?
> > > >
> > > > Also, could you add, before this line:
> > > >
> > > > tmp.push_back((std::string)result[row][i]);
> > > >
> > > > this code:
> > > >
> > > > Glib::ustring utf8_str(result[row][i]);
> > > > g_message("UTF-8 valid: %d: %s",
> int(utf8_str.validate()),
> > > > utf8_str.c_str());
> > > >
> > > > and see what it gives?
> > > >
> > > >
> > > > M.
> > > >
> > > > P.S.:
> > > > Yes std::string and ustring are generally compatible,
> with the main
> > > > differences being that ustring works by-character not
> by-byte (e.g.
> > > > find(), operator[], etc), ustring has methods for
> validation of UTF-8
> > > > and related functionality; the main potential culprit
> here is
> > > > operator<< and operator>> of ustring. I had really big
> problems with
> > > > these using boost::format() until i realised that
> internally, it's a
> > > > stream to which the args are fed using operator<<() of
> ustring.
> > > >
> > > >
> > > > On Mon, Dec 29, 2008 at 12:01 AM, Bernd Robertz
> > > > <bernd robertz cssweb de> wrote:
> > > > Thanks alot for your fast reply,
> > > >
> > > > I wrote a simple MySQL-Class wich is derived
> from the libsql++
> > > > class.
> > > >
> > > > In this class I used std::string to get the
> columns of a row.
> > > > The columns (strings) are stored in a
> vector<string> wich I
> > > > return to
> > > > Glib::ustring in My treeview-data-window.
> > > >
> > > > Hiere are some snippets:
> > > >
> > > > ::::::::::::::::::::::::
> SNIP ::::::::::::::::::::::
> > > > database.cpp
> > > > (My simple MySQL database class.)
> > > >
> > > > DataBase::Database()
> > > > {
> > > > // db-connect
> > > > // query-setup
> > > > // other connect related stuff
> > > >
> > > > mysqlpp::StoreQueryResult result; //
> Setup the result
> > > > object
> > > > int rows; // Keep track of result counts
> > > > }
> > > >
> > > > void DataBase::returnrow(int dataid)
> > > > {
> > > > // Setup db-query with dataid to get
> requested db-row
> > > > //
> > > > mysqlpp::Query query =
> dbcon.query(queries.alldata); //
> > > > DB Query
> > > > result = query.store(); // Store the
> results
> > > > rows = result.num_rows(); // Set the
> number of results
> > > > }
> > > >
> > > > std::vector<std::string> DataBase::fetchdata(int
> row)
> > > > {
> > > > std::vector<std::string> tmp; // Temp
> vector for return
> > > >
> > > > for (int i = 0; i <
> result.field_names()->size(); ++i)
> > > > {
> > > > // Every column will be stored in
> the temp
> > > > vector
> > > >
> tmp.push_back((std::string)result[row][i]);
> > > > }
> > > > return tmp; // Return the temp-vector
> with the
> > > > [columns]=data
> > > > }
> > > >
> > > > ------------
> > > >
> > > > datawindow.cpp
> > > > (The window in where the data is displayed after
> dbl-click on
> > > > treeview)
> > > >
> > > > DataWindow::DataWindow()
> > > > {
> > > > // Widgets setup
> > > > // Database connect via DataBase-Object
> > > > }
> > > >
> > > > void DataWindow::getid(int dataid)
> > > > {
> > > > // From the treeview I call this method
> to set the
> > > > // requested row-id
> > > > mysql.returnrow(dataid);
> > > >
> > > > // Call this method to setup the entries
> and textview
> > > > setdata();
> > > > }
> > > >
> > > > void DataWindow::setdata()
> > > > {
> > > > l_dataid.set_text(mysql.fetchdata(0)[0]);
> > > > Gtk::TreeModel::Row selrow =
> *(ls_cbox->prepend());
> > > > selrow[c_mod.c_cat] =
> mysql.fetchdata(0)[1];
> > > > cbox.set_active(0);
> > > >
> > > > // Here comes the problem, I get every
> column from My
> > > > // database-object wich I store in the
> entries.
> > > > e_name.set_text(mysql.fetchdata(0)[2]);
> > > > e_number.set_text(mysql.fetchdata(0)[3]);
> > > > e_user.set_text(mysql.fetchdata(0)[4]);
> > > > e_pass.set_text(mysql.fetchdata(0)[5]);
> > > > e_mail.set_text(mysql.fetchdata(0)[6]);
> > > > e_url.set_text(mysql.fetchdata(0)[7]);
> > > > e_other.set_text(mysql.fetchdata(0)[8]);
> > > >
> > > > // Here we have the main problem: the
> textview-buffer
> > > > // Some rows are shown here, some not.
> > > > b_descr->set_text(mysql.fetchdata(0)[9]);
> > > > }
> > > >
> > > > ::::::::::::::::::::::::: SNIP
> > > > END ::::::::::::::::::::::::::::
> > > >
> > > > That is the main code where the problem occours.
> > > >
> > > > The DataWindow is called on doubleclick on a
> treeview-row.
> > > > With the call of the DataWindow I call the
> DataView::getid()
> > > > wich receive the database row-id.
> > > > Then I fill with the DataView::setdata() method
> the widgets
> > > > in the new window.
> > > >
> > > > Some chars are, as I told, glyphs, but the
> textbuffer is the
> > > > main
> > > > problem.
> > > >
> > > > As far as I know, the entries and textbuffers
> are expecting
> > > > Glib::ustring, but I thought the conversion from
> std::string
> > > > to
> > > > Glib::ustring is quite compatible between these
> types.
> > > > I used std::string in the database class,
> because I wanted to
> > > > isolated
> > > > it from Gtkmm for other not Gtkmm-related
> projects.
> > > >
> > > > Thank you for helping.
> > > >
> > > > Bernd
> > > >
> > > >
> > > > Am Sonntag, den 28.12.2008, 22:26 +0100 schrieb
> Milosz
> > > > Derezynski:
> > > >
> > > > > Are you using Glib::ustring anywhere in the
> process of
> > > > getting the
> > > > > data from the SQL database to the view?
> > > > >
> > > > > Glib::ustring implicitly (and in my opinion
> still at least
> > > > not well
> > > > > enough documented/hinted at) converts from
> UTF-8
> > > > > to the current locale with operators << and
> >>. If you are
> > > > using a
> > > > > stringstream anywhere in the process, or are
> > > > > using operator<< or operator>> in any other
> way, this could
> > > > then be
> > > > > the cause of the problem.
> > > > >
> > > > > Otherwise, and in the above described case too
> actually it
> > > > would help
> > > > > very much if you could post (if possible, all
> of)
> > > > > the relevant code online.
> > > > >
> > > > > Regards,
> > > > > M.
> > > > >
> > > > > On Sun, Dec 28, 2008 at 8:53 PM, Bernd Robertz
> > > > > <bernd robertz cssweb de> wrote:
> > > > > Hello list,
> > > > >
> > > > > I'm not really sure, but I hope My
> question is in
> > > > the right
> > > > > place here.
> > > > >
> > > > > I wrote a simple database app like
> this:
> > > > >
> > > > > Connect to a MySQL server via libmysql
> ++,
> > > > > and then list the rows in a treeview.
> > > > > Doubleclick on a row shows a window
> with some
> > > > entries and a
> > > > > textview
> > > > > with the content of the selected row.
> > > > > The textview is associated with a type
> text in the
> > > > DB.
> > > > >
> > > > > Everything is fine.
> > > > >
> > > > > So but I ran really into problems with
> the charsets.
> > > > > german umlaute (ö, ä, etc.) and some
> other
> > > > characters are
> > > > > glyphs in the
> > > > > treeview. Ok, thats not the main
> problem.
> > > > >
> > > > > That is: The entries in the detailed
> view still
> > > > shows the
> > > > > glyphs for
> > > > > some chars like ö, ä.
> > > > > But on some rows, the represented
> textbuffer for
> > > > textview is
> > > > > not
> > > > > visible. Wich means the textview is in
> some rows
> > > > empty.
> > > > >
> > > > > I run My app from a console and I got
> these error
> > > > messages:
> > > > >
> > > > > :::::::::
> > > > > Gtk-CRITICAL **:
> gtk_text_buffer_emit_insert:
> > > > assertion
> > > > > `g_utf8_validate
> > > > > (text, len, NULL)' failed
> > > > > :::::::::
> > > > >
> > > > > In other rows, the textbuffer is
> correct shown,
> > > > exepct some
> > > > > glyphs.
> > > > >
> > > > > If I print the textbuffer as a
> simple .c_str() in a
> > > > console it
> > > > > will be
> > > > > shown correctly (inklusive ä, ö,
> ü...).
> > > > > (Tried to show the textview as
> a .c_str(), without
> > > > any
> > > > > effect.)
> > > > >
> > > > > The SQL connection is
> serverside-configured as UTF-8
> > > > Unicode.
> > > > > The SQL table collation is UTF-8
> General CI (Case
> > > > > Insensitive).
> > > > > The Application runs on a DE_de Euro
> (ISO-8859-15).
> > > > >
> > > > > All characters are shown correctly in
> a PHP app wich
> > > > does the
> > > > > same, just
> > > > > only online. :-)
> > > > >
> > > > > I hope someone could turn on a light
> above my head,
> > > > why I
> > > > > don't get the
> > > > > right charactes.
> > > > >
> > > > > Thanks alot in advance
> > > > >
> > > > > Bernd
> > > > >
> > > > >
> > > > >
> _______________________________________________
> > > > > gtkmm-list mailing list
> > > > > gtkmm-list gnome org
> > > > >
> http://mail.gnome.org/mailman/listinfo/gtkmm-list
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Please note that according to the German law
> on data
> > > > retention,
> > > > > information on every electronic information
> exchange with me
> > > > is
> > > > > retained for a period of six months.
> > > > > [Bitte beachten Sie, dass dem Gesetz zur
> > > > Vorratsdatenspeicherung
> > > > > zufolge
> > > > > jeder elektronische Kontakt mit mir sechs
> Monate lang
> > > > gespeichert
> > > > > wird.]
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Please note that according to the German law on data
> retention,
> > > > information on every electronic information exchange
> with me is
> > > > retained for a period of six months.
> > > > [Bitte beachten Sie, dass dem Gesetz zur
> Vorratsdatenspeicherung
> > > > zufolge
> > > > jeder elektronische Kontakt mit mir sechs Monate lang
> gespeichert
> > > > wird.]
> > >
> > > _______________________________________________
> > > gtkmm-list mailing list
> > > gtkmm-list gnome org
> > > http://mail.gnome.org/mailman/listinfo/gtkmm-list
> >
> > _______________________________________________
> > gtkmm-list mailing list
> > gtkmm-list gnome org
> > http://mail.gnome.org/mailman/listinfo/gtkmm-list
>
> _______________________________________________
> gtkmm-list mailing list
> gtkmm-list gnome org
> http://mail.gnome.org/mailman/listinfo/gtkmm-list
>
>
>
>
> --
> Please note that according to the German law on data retention,
> information on every electronic information exchange with me is
> retained for a period of six months.
> [Bitte beachten Sie, dass dem Gesetz zur Vorratsdatenspeicherung
> zufolge
> jeder elektronische Kontakt mit mir sechs Monate lang gespeichert
> wird.]
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]
