Re: Making metadata storage SQL-driven

[Jamie McCracken <jamiemcc blueyonder co uk>]

Alexander Larsson wrote:
> On Thu, 2005-09-01 at 12:16 +0200, Christian Neumair wrote:
>
> > I really wonder why we rely on files for our metadata. It has issues [1]
> > where a synchronous fopen and read or write operation would really take
> > too long, requiring us to schedule reads/writes. Without doing any
> > performance measures, I think this could significantly speed up our
> > metadata code for big directories. I think we should do the same as
> > beagle, and rely on a tiny SQL server, doing all our metadata operations
> > synchronously. I'm not a big fan of EAs, since they seem to be some
> > different flavors and inconsistencies among various implementations.
> >
> > [1] http://makeashorterlink.com/?Y364247BB
>
>
> I don't understand why a SQL server would be any slower than read() and
> write(). I mean, it would still end up doing read/write calls. It seems
> to me that all it does is highly complicating a simple key/value system.
> And furthermore, doing i/o synchronously is definitely a no-no since not
> doing that is at the core of the nautilus design.
>
> I'm not saying the current system is great, but I don't see using an SQL
> database as a better solution than just fixing stuff to be better.
> Databases tend to have all sorts of complications like on-disk format
> changes, complexity of installation, opaque file-formats, locking issue
> with shared homedirs, fragility (if some part goes bad the whole db can
> be corrupt), etc. Furthermore, we wouldn't have much use for most of the
> features you get from an SQL database. 

thats true for some DBs but not all.

For instance the embedded mysql needs no set up and it will work out of 
the box (its just a shared library) so installation and setup is a 
non-issue. It has no locking issues as the INNODB stuff in mysql is 
multi generational so readers and writers dont block each other and its 
totally safe as a write generates a new copy of a record and the commit 
on it is atomic so even of there's a power failure during a write 
corruption will not occur to anything committed.

With a dbus interface all access can be both sync or async so I dont see 
a problem there.

Also you need multiple tables to store or use RDF and contextual data 
which is highly relational in nature and cannot be efficiently stored in 
a single key value table.

My plan is to have a gnome-vfs driver to it so you can use nautilus to 
contextual browse your file metadata so you could for example enter a 
folder called "emails from bob" which would show corresponding emails 
etc. Contextual data is very hierarchical so this would be a good case 
for it.

--
Mr Jamie McCracken
http://www.advogato.org/person/jamiemcc/