Re: Making metadata storage SQL-driven

[Manuel Amador <rudd-o amautacorp com>]

Hi there, everyone participating on this list.

I really don't like the direction this thread is going.

I see you guys getting all wired up on tech and performance and I bet no
one in this thread has actually sat down and fleshed out all
requirements in a detailed fashion.

Let's visit a quite simple example of the things we should be thinking
of, which is simply *infeasible* with this cobbled-up embedded MySQL
solution.

There are two kinds of metadata, which need to be handled differently:
- per-user metadata: song ratings, file emblems, notes
- per-system metadata: document author, song tempo, file MIME type, 
                       file thumbnail preview

The MySQL embedded / DBUS solution everyone is drumming up here is quite
appropriate for per-user metadata.  But it will make our lives extremely
hard for per-system metadata.

The fact that users may also want to set "per-user metadata" system-wide
(say, an admin setting emblems on company folders for all their users)
is also something that needs to be accounted for.

I hope you guys won't take this longish remark personally.

Getting back to the embedded MySQL thingie.  I've seen many of you bash
EAs as not being "performant enough", as if that was the only thing that
mattered, or as if it were more of a show stopper than a SIGSEGV.  Even
if that was true:

* Performance is not the biggest concern here.  It's functionality and
coherence!  I really, really propose per-system metadata should be
stored in an *extended attribute*: it's a POSIX standard, and it will
make cooperation between projects (I'm thinking KDE and console-based
utilities here) much, much, much, much easier and less troublesome.  If
current implementations have bugs, it's time to start putting pressure
on implementors to fix their fucked-up things and get on with it.

* Even if EAs turned out to be horribly slow (which they are not, on
most filesystems) and inefficient, you still need a common ground for
many apps to get metadata.  Command-line apps and the like.  KDE.  Do
you honestly expect they'll link to your metadata libraries?  The KDE
guys haven't linked to glib ever, so I say, hell no way that's gonna
happen!  You *need* to store and read them from extended attributes.
I'm sure you'll then build a cache or something, or even start using
Beagle to cache and query metadata.

Start lifting use cases from competing operating systems (Mac comes to
mind), and come up with a few use cases yourselves.  Examples:

* nautilus visits a directory, and determines file MIME type, based on
contents/extension/whatever.  It then saves the mime type to files' EAs,
if there was no EA available first.  Subsequently it uses the EAs (or
the Beagle cache for them) as an authoritative source for MIME type.
Same can be said of apps (save/load), where GNOME should provide an API
to abstract low-level details (think of it as, like, gnome_play_sound
() ).


El jue, 01-09-2005 a las 16:42 +0100, Jamie McCracken escribió:
> John (J5) Palmieri wrote:
> 
> >>
> >>with a dbus wrapper around it the license becomes irrelevant. The dbus 
> >>server end of DDS will be GPL but the client end libDDS will be LGPL. 
> >>IPC is a great way to eliminate GPL dependencies!
> > 
> > 
> > That is a runaround the GPL.  Not something I would like to see GNOME
> > set precedence on.
> > 
> 
> The purpose of the Dbus wrapper is not to act as a run around the GPL. 
> The dbus wrapper is their to create a per session database thats fast 
> and efficient memory wise (by using threads and threadpools). The fact 
> that it also allows us to bypass restrictive licenses in this instance 
> is a bonus :)
> 
> That said, DDS does not have to be part of Gnome. GConf may always 
> default to a text file backend (IE lowest common denominator that works 
> on all systems whether its NFS or a platform that does not support 
> threads) and likwise a metadata server for Nautilus may well use text 
> files by default for similiar reasons. However thanks to Dbus, DDS can 
> reimplement any of these interfaces and so give the user or distro a 
> choice of whether to use the text-based default or a faster DB based one.
> 
> -- 
> Mr Jamie McCracken
> http://www.advogato.org/person/jamiemcc/
-- 
Manuel Amador                   <rudd-o amautacorp com>
http://www.amautacorp.com/            +593 (4) 220-7010