Re: [gnome-db] Update meta store (again)

[Vivien Malerba <vmalerba gmail com>]

On 9 March 2012 12:16, Piotr Pokora <piotrek pokora gmail com> wrote:

Hi!

I am quite confused with meta store functionality.
First, all operations are very slow (if you compare to direct RDBM
queries) and frnakly speaking I have no idea how to improve them.
Second, I have problems with GdaConnection options.

All is working fine, when I open connection with GDA_CONNECTION_OPTIONS_NONE

and update meta store on demand:
https://github.com/midgardproject/midgard-core/blob/ratatoskr/src/midgard_core_query.c#L58

I tried to set up the same database as meta store, which seems to
improve performance a bit, but fails with postgres provider:

syntax error at or near "'Nullable'"
LINE 1: ...eric_precision, c.numeric_scale, c.is_nullable AS
'Nullable'...^,
No:(-1), src:gda-postgres, SQL: 42601

This definitely looks like a bug which should be corrected. I'll work on it ASAP.
 

Setting GDA_CONNECTION_OPTIONS_AUTO_META_DATA doesn't help, as meta
store has to be updated still after any table layout change.

I am finding some examples here and there, and my question is:
What is the best way to manage meta store?

It's up to you to define when you need information from the meta store (i.e. when you need information about tables, columns, ...). Maybe you don't need that information at all (I don't know how Midgard works regarding the tables it creates).

What I tend to do is create a small separate SQLite database to keep the meta data as it is very fast to write to and read from.
 

Code complexity doesn't matter. It should work and have acceptable
performance.

Agreed. However, my guess is that the actual SQL code used in the PostgreSQL provider to actually fetch metadata should benefit from being simplified (some queries are very complex). However I don't have the time to work on this for now. Of course there may be some other bottlenecks in Libgda, and probably using a profiler would help.
 
Also keep in mind that Libgda tries to keep all the data in a meta store valid, even if out of date for some parts (meaning for example you can't update only the tables list: when you do that it also updates the columns list of each table, etc).

Regards,

Vivien