GNOME.org
 

libgda r3301 - in trunk: . doc/C doc/C/tmpl libgda providers/firebird providers/jdbc providers/mysql providers/postgres providers/skel-implementation/capi

Author: vivien
Date: Mon Jan 26 19:54:57 2009
New Revision: 3301
URL: http://svn.gnome.org/viewvc/libgda?rev=3301&view=rev

Log:
2009-01-26  Vivien Malerba <malerba gnome-db org>

	* doc/C:
	  - added section about SQL parsers and how to write a database specific
	    one in a database provider
	  - misc. improvements
	* providers/skel-implementation/capi/gen_def.c:
	* providers/firebird/gen_def.c:
	* providers/mysql/gen_def.c:
	* providers/postgres/gen_def.c: removed unused #define in generated header file
	* libgda/libgda.symbols: removed symbols not existing anymore
	* providers/jdbc/Makefile.am: include the MANIFEST.MF in distributed tarballs,
	fixes bugs #568388 and #568353
	* providers/gda-mysql-recordset.c: add unsigned information when binding
	for output to avoid data being truncated, fixes bug #561748


Added:
   trunk/doc/C/parser_gen.dia   (contents, props changed)
   trunk/doc/C/parser_gen.png   (contents, props changed)
   trunk/doc/C/parser_prov.dia   (contents, props changed)
   trunk/doc/C/parser_prov.png   (contents, props changed)
Modified:
   trunk/ChangeLog
   trunk/doc/C/   (props changed)
   trunk/doc/C/Makefile.am
   trunk/doc/C/prov-writing.xml
   trunk/doc/C/tmpl/gda-sql-parser.sgml
   trunk/libgda/libgda.symbols
   trunk/providers/firebird/gen_def.c
   trunk/providers/jdbc/Makefile.am
   trunk/providers/mysql/gda-mysql-recordset.c
   trunk/providers/mysql/gen_def.c
   trunk/providers/postgres/gen_def.c
   trunk/providers/skel-implementation/capi/gen_def.c

Modified: trunk/doc/C/Makefile.am
==============================================================================
--- trunk/doc/C/Makefile.am	(original)
+++ trunk/doc/C/Makefile.am	Mon Jan 26 19:54:57 2009
@@ -53,7 +53,8 @@
 	writable_data_model.png GdaDataModelIter.png \
 	data_validation_holder.png data_validation_proxy.png data_validation_set.png \
 	data_proxy1.png data_proxy2.png data_proxy3.png data_proxy4.png data_proxy5.png \
-	gda-sql-graph.png howto-exec.png
+	gda-sql-graph.png howto-exec.png \
+	parser_gen.png parser_prov.png
 
 # Extra options to supply to gtkdoc-fixref
 FIXXREF_OPTIONS=

Added: trunk/doc/C/parser_gen.dia
==============================================================================
Binary file. No diff available.

Added: trunk/doc/C/parser_gen.png
==============================================================================
Binary file. No diff available.

Added: trunk/doc/C/parser_prov.dia
==============================================================================
Binary file. No diff available.

Added: trunk/doc/C/parser_prov.png
==============================================================================
Binary file. No diff available.

Modified: trunk/doc/C/prov-writing.xml
==============================================================================
--- trunk/doc/C/prov-writing.xml	(original)
+++ trunk/doc/C/prov-writing.xml	Mon Jan 26 19:54:57 2009
@@ -676,6 +676,148 @@
   </sect2>
 </chapter>
 
+<chapter id="libgda-provider-parser">
+  <title>SQL parser</title>
+  <para>
+    &LIBGDA; implements a generic SQL parser which creates <link linkend="GdaStatement">GdaStatement</link> objects from
+    an SQL string. If the database provider needs to implement its own parser because the generic one does not handle
+    the database specific SQL syntax, it can be done using instructions in this chapter. Otherwise, the provider's sources
+    can be cleared of any code related to the parser.
+  </para>
+  <sect1>
+    <title>Implementation overview</title>
+    <para>
+      This section describes how the generic SQL parser and a provider specific parser are built regarding the files
+      and programs which are involved.
+    </para>
+    <sect2>
+      <title>Generic SQL parser</title>
+      <para>
+	The <link linkend="GdaSqlParser">GdaSqlParser</link> object can parse any SQL string of any SQL dialect,
+	while always identifying the variables (which have a &LIBGDA;'s specific syntax) in the string. If the parser
+	can identify a structure in the SQL string it understands, then it internally builds a
+	<link linkend="GdaSqlStatement">GdaSqlStatement</link> structure of the correct type, and if it cannot then is simply
+	delimits parts in the SQL string to identify variables and also builds a
+	<link linkend="GdaSqlStatement">GdaSqlStatement</link> structure but of 
+	<link linkend="GDA-SQL-STATEMENT-UNKNOWN:CAPS">GDA_SQL_STATEMENT_UNKNOWN</link>. If the string
+	cannot be delimited and variables identified, then it returns an error (usually there is a quotes mismatch problem
+	within the SQL string).
+      </para>
+      <para>
+	Failing to identify a known structure in the SQL string can have several reasons:
+	<itemizedlist>
+	   <listitem><para>the SQL string is not one of the known types of statements (see 
+	       <link linkend="GdaSqlStatementType">GdaSqlStatementType</link>)</para></listitem>
+	   <listitem><para>the SQL uses some database specific extensions</para></listitem>
+	</itemizedlist>
+      </para>
+      <para>
+	The generic SQL parser implementation has its source files in the 
+	<filename class="directory">libgda/sql-parser</filename> directory; the files which actually implement
+	the parser itself are the <filename>parser.y</filename>, <filename>delimiter.y</filename> and
+	 <filename>parser_tokens.h</filename> files:
+	 <itemizedlist>
+	   <listitem><para>The <filename>parser.y</filename> file contains the grammar used by the parser</para></listitem>
+	   <listitem><para>The <filename>delimiter.y</filename> file contains the grammar used by the parser when it 
+	       is operating as a delimiter</para></listitem>
+	   <listitem><para>The <filename>parser_tokens.h</filename> defines some hard coded tokens</para></listitem>
+	 </itemizedlist>
+      </para>
+      <para>
+	The parser grammar files use the <ulink url="http://www.hwaci.com/sw/lemon/";>Lemon parser generator</ulink> syntax
+	which is a LALR parser similar to <application>YACC</application> or <application>bison</application>. The lexer part
+	however is not <application>LEX</application> but is a custom one integrated in the
+	<filename>gda-sql-parser.c</filename> file (this allows a better integration between the lexer and parser parts).
+      </para>
+      <para>
+	The following figure illustrates the files involved and how they are produced and used to create
+	the generic SQL parser.
+	<mediaobject>
+          <imageobject role="html">
+            <imagedata fileref="parser_gen.png" format="PNG"/>
+          </imageobject>
+          <textobject> 
+            <phrase>Generic SQL parser's implementation</phrase>
+          </textobject>
+	</mediaobject>
+	<itemizedlist>
+	   <listitem><para>The white background indicate files which are sources
+	       (part of &LIBGDA;'s distribution)</para></listitem>
+	   <listitem><para>The blue background indicate files that they are produced dynamically</para></listitem>
+	   <listitem><para>The pink background indicate programs that are compiled and used themselves in
+	       the compilation process to generate files. These programs are:
+	       <itemizedlist>
+		 <listitem><para><application>lemon</application>: the lemon parser itself</para></listitem>
+		 <listitem><para><application>gen_def</application>: generated the "converters" arrays (see blow)</para>
+		 </listitem>
+	       </itemizedlist>
+	       Note that none of these programs gets installed (and when cross compiling, they are compiled as programs
+	       executing on the host actually making the compilation).
+	   </para></listitem>
+	   <listitem><para>The green background identifies components which are reused when implementing provider specific
+	       parsers</para></listitem>
+	 </itemizedlist>
+      </para>
+      <para>
+	The tokenizer (AKA lexer) generates values identified by the "L_" prefix (for example "L_BEGIN"). Because
+	the GdaSqlParser object uses the same lexer with at least two different parsers (namely the parser and delimiter
+	mentionned earlier), and because the Lemon parser generator generates its own value identifiers for tokens, there
+	is a conversion step (the "converter" block in the diagram) which converts the "L_" prefixed tokens with the ones
+	useable by each parser (both converters are defined as arrays in the <filename>token_types.h</filename> file.
+      </para>
+    </sect2>
+    <sect2>
+      <title>Provider specific SQL parser</title>
+      <para>
+	One wants to write a database specific SQL parser when:
+	<itemizedlist>
+	  <listitem><para>the SQL understood by the database differs from the generic SQL. For example
+	      PostgreSQL associates priorities to the compound statement in a different way as the generic SQL.
+	      In this case it is strongly recommended to write a custom SQL parser</para></listitem>
+	  <listitem><para>the SQL understood by the database has specific extensions</para></listitem>
+	</itemizedlist>
+      </para>
+      <para>
+	Using the same background color conventions as the previous diagram, the following diagram illustrates
+	the files involved and how they are produced and used to create a provider specific SQL parser:
+      </para>
+      <para>
+	<mediaobject>
+          <imageobject role="html">
+            <imagedata fileref="parser_prov.png" format="PNG"/>
+          </imageobject>
+          <textobject> 
+            <phrase>Provider specific SQL parser's implementation</phrase>
+          </textobject>
+	</mediaobject>
+      </para>
+      <para>
+	The key differences are:
+	<itemizedlist>
+	  <listitem><para>The delimiter part of the <link linkend="GdaSqlParser">GdaSqlParser</link> object
+	      is the same as for the generic SQL parser implementation</para></listitem>
+	  <listitem><para>While the <application>lemon</application> program is the same as for the generic SQL parser,
+	      the <application>gen_def</application> is different, and takes as its input the ".h" file produced by
+	      the <application>lemon</application> program and the <filename>libgda/sql-parser/token_types.h</filename>.
+	  </para></listitem>
+	</itemizedlist>
+      </para>
+    </sect2>
+  </sect1>
+  <sect1>
+    <title>Tips to write a custom parser</title>
+    <para>
+      <itemizedlist>
+	<listitem><para>Write a new <filename>parser.y</filename> file (better to copy and adapt 
+	    it than starting from scratch)</para></listitem>
+	<listitem><para>Sub class the <link linkend="GdaSqlParser">GdaSqlParser</link> class and "connect" the
+	    class's virtual methods to the new generated parser</para></listitem>
+	<listitem><para>Start from the skeleton implementation</para></listitem>
+      </itemizedlist>
+    </para>
+  </sect1>
+</chapter>
+
 <chapter id="libgda-provider-pack">
   <title>Assembling all the parts</title>
   <para>

Modified: trunk/doc/C/tmpl/gda-sql-parser.sgml
==============================================================================
--- trunk/doc/C/tmpl/gda-sql-parser.sgml	(original)
+++ trunk/doc/C/tmpl/gda-sql-parser.sgml	Mon Jan 26 19:54:57 2009
@@ -71,6 +71,15 @@
 ##-5::timestamp
 </programlisting>
 </para>
+<para>
+  Also note that variables should not be used when an SQL identifier is expected. For example the following
+  examples <emphasis>should be avoided</emphasis> because they may not work properly (depending on the database being used):
+<programlisting>
+SELECT * FROM ##tablename::string;
+DELETE FROM mytable WHERE ##tcol::string = 5;
+ALTER GROUP mygroup ADD USER ##name::gchararray;
+</programlisting>
+</para>
 
 <para>
   The #GdaSqlParser object internally uses a LEMON generated parser (the same as the one used by SQLite).

Modified: trunk/libgda/libgda.symbols
==============================================================================
--- trunk/libgda/libgda.symbols	(original)
+++ trunk/libgda/libgda.symbols	Mon Jan 26 19:54:57 2009
@@ -497,8 +497,6 @@
 	gda_server_operation_save_data_to_xml
 	gda_server_operation_set_value_at
 	gda_server_operation_type_get_type
-	gda_server_provider_blob_list_for_delete
-	gda_server_provider_blob_list_for_update
 	gda_server_provider_create_operation
 	gda_server_provider_create_parser
 	gda_server_provider_error_get_type
@@ -522,8 +520,6 @@
 	gda_server_provider_perform_operation
 	gda_server_provider_perform_operation_default
 	gda_server_provider_render_operation
-	gda_server_provider_select_query_has_blobs
-	gda_server_provider_split_update_query
 	gda_server_provider_string_to_value
 	gda_server_provider_supports_feature
 	gda_server_provider_supports_operation

Modified: trunk/providers/firebird/gen_def.c
==============================================================================
--- trunk/providers/firebird/gen_def.c	(original)
+++ trunk/providers/firebird/gen_def.c	Mon Jan 26 19:54:57 2009
@@ -56,7 +56,6 @@
 	FILE *fd_imposed;
 	FILE *fd_parser;
 	HashEntry *illegal_entry;
-	HashEntry *rawstring_entry;
 
 	memset (entries, 0, sizeof (entries));
 	/* printf ("Imposed header: %s\n", IMPOSED_HEADER); */
@@ -85,12 +84,7 @@
 		" * DO NOT EDIT MANUALLY\n */\n\n\n");
 
 	/* output */
-	for (i = 0; i < nb_entries; i++) {
-		HashEntry *entry = &(entries[i]);
-		printf ("#define L_%s \t\t %d\n", entry->key, i);
-	}
 	illegal_entry = find_entry_for_token ("ILLEGAL");
-	rawstring_entry = find_entry_for_token ("RAWSTRING");
 	printf ("gint firebird_parser_tokens[] = {\n");
 	for (i = 0; i < nb_entries; i++) {
 		HashEntry *entry = &(entries[i]);

Modified: trunk/providers/jdbc/Makefile.am
==============================================================================
--- trunk/providers/jdbc/Makefile.am	(original)
+++ trunk/providers/jdbc/Makefile.am	Mon Jan 26 19:54:57 2009
@@ -111,7 +111,8 @@
 
 EXTRA_DIST = $(xml_in_files) libgda-jdbc-4.0.pc.in \
 	$(jdbcprov_sources) \
-	$(doc_files)
+	$(doc_files) \
+	MANIFEST.MF
 DISTCLEANFILES = $(xml_DATA)
 
 
@@ -125,4 +126,5 @@
 gda_list_jdbc_providers_4_0_SOURCES = gda-list-jdbc-providers.c
 gda_list_jdbc_providers_4_0_LDADD = \
         $(LIBGDA_LIBS) \
-        $(top_builddir)/libgda/libgda-4.0.la
\ No newline at end of file
+        $(top_builddir)/libgda/libgda-4.0.la
+

Modified: trunk/providers/mysql/gda-mysql-recordset.c
==============================================================================
--- trunk/providers/mysql/gda-mysql-recordset.c	(original)
+++ trunk/providers/mysql/gda-mysql-recordset.c	Mon Jan 26 19:54:57 2009
@@ -111,7 +111,7 @@
 gint
 gda_mysql_recordset_get_chunk_size (GdaMysqlRecordset  *recset)
 {
-	g_return_if_fail (GDA_IS_MYSQL_RECORDSET (recset));
+	g_return_val_if_fail (GDA_IS_MYSQL_RECORDSET (recset), -1);
 	return recset->priv->chunk_size;
 }
 
@@ -142,7 +142,7 @@
 gint
 gda_mysql_recordset_get_chunks_read (GdaMysqlRecordset  *recset)
 {
-	g_return_if_fail (GDA_IS_MYSQL_RECORDSET (recset));
+	g_return_val_if_fail (GDA_IS_MYSQL_RECORDSET (recset), -1);
 	return recset->priv->chunks_read;
 }
 
@@ -320,9 +320,6 @@
 
 	/* make sure @ps reports the correct number of columns using the API*/
         if (_GDA_PSTMT (ps)->ncols < 0) {
-                /*_GDA_PSTMT (ps)->ncols = ...;*/
-		// TO_IMPLEMENT;
-		
 		_GDA_PSTMT(ps)->ncols = mysql_stmt_field_count (cdata->mysql_stmt);
 	}
 
@@ -364,8 +361,6 @@
 			GdaColumn *column = GDA_COLUMN (list->data);
 
 			/* use C API to set columns' information using gda_column_set_*() */
-			// TO_IMPLEMENT;
-			
 			MYSQL_FIELD *field = &mysql_fields[i];
 
 			GType gtype = _GDA_PSTMT(ps)->types[i];
@@ -377,7 +372,9 @@
 			gda_column_set_name (column, field->name);
 			gda_column_set_description (column, field->name);
 			
+			/* binding results with types */
 			mysql_bind_result[i].buffer_type = field->type;
+			mysql_bind_result[i].is_unsigned = field->flags & UNSIGNED_FLAG ? TRUE : FALSE;
 			switch (mysql_bind_result[i].buffer_type) {
 			case MYSQL_TYPE_TINY:
 			case MYSQL_TYPE_SHORT:
@@ -421,7 +418,8 @@
 				g_warning (_("Invalid column bind data type. %d\n"),
 					   mysql_bind_result[i].buffer_type);
 			}
-			//g_print ("%s: NAME=%s, TYPE=%d, GTYPE=%s\n", __func__, field->name, field->type, g_type_name (gtype));
+			/* g_print ("%s(): NAME=%s, TYPE=%d, GTYPE=%s\n", 
+			   __FUNCTION__, field->name, field->type, g_type_name (gtype)); */
 		}
 		
                 if (mysql_stmt_bind_result (cdata->mysql_stmt, mysql_bind_result)) {
@@ -482,16 +480,17 @@
 
 
 static GdaRow *
-new_row_from_mysql_stmt (GdaMysqlRecordset  *imodel,
-			 gint                rownum)
+new_row_from_mysql_stmt (GdaMysqlRecordset  *imodel, gint rownum, GError **error)
 {
-	/* g_print ("%s(): NCOLS=%d  ROWNUM=%d\n", __func__, */
-	/* 	 ((GdaDataSelect *) imodel)->prep_stmt->ncols, rownum); */
+	//g_print ("%s(): NCOLS=%d  ROWNUM=%d\n", __func__, ((GdaDataSelect *) imodel)->prep_stmt->ncols, rownum);
 
 	g_return_val_if_fail (imodel->priv->mysql_stmt != NULL, NULL);
 
-	if (mysql_stmt_fetch (imodel->priv->mysql_stmt))
+	if (mysql_stmt_fetch (imodel->priv->mysql_stmt)) {
+		g_set_error (error, GDA_SERVER_PROVIDER_ERROR, GDA_SERVER_PROVIDER_DATA_ERROR,
+			     _("Can't fetch data from server"));
 		return NULL;
+	}
 	
 	MYSQL_BIND *mysql_bind_result = ((GdaMysqlPStmt *) ((GdaDataSelect *) imodel)->prep_stmt)->mysql_bind_result;
 	g_assert (mysql_bind_result);
@@ -504,9 +503,9 @@
 		GValue *value = gda_row_get_value (row, i);
 		GType type = ((GdaDataSelect *) imodel)->prep_stmt->types[i];
 		gda_value_reset_with_type (value, type);
-		//
+		
 		//g_print ("%s: #%d : TYPE=%d, GTYPE=%s\n", __func__, i, mysql_bind_result[i].buffer_type, g_type_name (type));
-		//
+		
 		
 		int intvalue = 0;
 		long long longlongvalue = 0;
@@ -578,8 +577,10 @@
 				};
 				gda_value_set_timestamp (value, &timestamp);
 			} else {
-				g_warning (_("Type %s not mapped for value %p"),
-					   g_type_name (type), timevalue);
+				g_warning (_("Type %s not mapped for value %d/%d/%d %d:%d:%d.%d"),
+					   g_type_name (type), timevalue.year, timevalue.month, 
+					   timevalue.day, timevalue.hour, timevalue.minute, 
+					   timevalue.second, timevalue.second_part);
 			}
 
 			break;
@@ -675,7 +676,10 @@
 	if (*row)
 		return TRUE;
 	
-	*row = new_row_from_mysql_stmt (imodel, rownum);
+	*row = new_row_from_mysql_stmt (imodel, rownum, error);
+	if (!*row)
+		return FALSE;
+
 	gda_data_select_take_row (model, *row, rownum);
 	
 	if (model->nb_stored_rows == model->advertized_nrows) {
@@ -737,11 +741,11 @@
 		imodel->priv->tmp_row = NULL;
 	}
 	
-	*row = new_row_from_mysql_stmt (imodel, rownum);
+	*row = new_row_from_mysql_stmt (imodel, rownum, error);
 
 	imodel->priv->tmp_row = *row;
 
-	return TRUE;
+	return *row ? TRUE : FALSE;
 }
 
 /*

Modified: trunk/providers/mysql/gen_def.c
==============================================================================
--- trunk/providers/mysql/gen_def.c	(original)
+++ trunk/providers/mysql/gen_def.c	Mon Jan 26 19:54:57 2009
@@ -56,7 +56,6 @@
 	FILE *fd_imposed;
 	FILE *fd_parser;
 	HashEntry *illegal_entry;
-	HashEntry *rawstring_entry;
 
 	memset (entries, 0, sizeof (entries));
 	/* printf ("Imposed header: %s\n", IMPOSED_HEADER); */
@@ -85,12 +84,7 @@
 		" * DO NOT EDIT MANUALLY\n */\n\n\n");
 
 	/* output */
-	for (i = 0; i < nb_entries; i++) {
-		HashEntry *entry = &(entries[i]);
-		printf ("#define L_%s \t\t %d\n", entry->key, i);
-	}
 	illegal_entry = find_entry_for_token ("ILLEGAL");
-	rawstring_entry = find_entry_for_token ("RAWSTRING");
 	printf ("gint mysql_parser_tokens[] = {\n");
 	for (i = 0; i < nb_entries; i++) {
 		HashEntry *entry = &(entries[i]);

Modified: trunk/providers/postgres/gen_def.c
==============================================================================
--- trunk/providers/postgres/gen_def.c	(original)
+++ trunk/providers/postgres/gen_def.c	Mon Jan 26 19:54:57 2009
@@ -56,7 +56,6 @@
 	FILE *fd_imposed;
 	FILE *fd_parser;
 	HashEntry *illegal_entry;
-	HashEntry *rawstring_entry;
 
 	memset (entries, 0, sizeof (entries));
 	/* printf ("Imposed header: %s\n", IMPOSED_HEADER); */
@@ -85,12 +84,7 @@
 		" * DO NOT EDIT MANUALLY\n */\n\n\n");
 
 	/* output */
-	for (i = 0; i < nb_entries; i++) {
-		HashEntry *entry = &(entries[i]);
-		printf ("#define L_%s \t\t %d\n", entry->key, i);
-	}
 	illegal_entry = find_entry_for_token ("ILLEGAL");
-	rawstring_entry = find_entry_for_token ("RAWSTRING");
 	printf ("gint postgres_parser_tokens[] = {\n");
 	for (i = 0; i < nb_entries; i++) {
 		HashEntry *entry = &(entries[i]);

Modified: trunk/providers/skel-implementation/capi/gen_def.c
==============================================================================
--- trunk/providers/skel-implementation/capi/gen_def.c	(original)
+++ trunk/providers/skel-implementation/capi/gen_def.c	Mon Jan 26 19:54:57 2009
@@ -56,7 +56,6 @@
 	FILE *fd_imposed;
 	FILE *fd_parser;
 	HashEntry *illegal_entry;
-	HashEntry *rawstring_entry;
 
 	memset (entries, 0, sizeof (entries));
 	/* printf ("Imposed header: %s\n", IMPOSED_HEADER); */
@@ -85,12 +84,7 @@
 		" * DO NOT EDIT MANUALLY\n */\n\n\n");
 
 	/* output */
-	for (i = 0; i < nb_entries; i++) {
-		HashEntry *entry = &(entries[i]);
-		printf ("#define L_%s \t\t %d\n", entry->key, i);
-	}
 	illegal_entry = find_entry_for_token ("ILLEGAL");
-	rawstring_entry = find_entry_for_token ("RAWSTRING");
 	printf ("gint capi_parser_tokens[] = {\n");
 	for (i = 0; i < nb_entries; i++) {
 		HashEntry *entry = &(entries[i]);
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]