Re: [Rhythmbox-devel] rhythmdb post-mortem and future

[Bastien Nocera <hadess hadess net>]

Hey Colin,

On Fri, 2003-11-14 at 22:22, Colin Walters wrote:
> So I think at this point we can call RhythmDB a success, but it wasn't
> as big a one as I had hoped.  There are still issues with locking,
> although fewer than with RBNode.  Also it will need reworking to solve
> issues like filesystem coherency.  So, let's investigate.
> 
> * Locking
<snip>

I think Jamboreee has shown that multithreading isn't really needed.

> * Filesystem coherency
> 
> In summary:
> http://bugzilla.gnome.org/show_bug.cgi?id=125177
> This is also related to:
> http://bugzilla.gnome.org/show_bug.cgi?id=125452
<snip>
> Here are my thoughts on this.  Currently, RhythmDB just stores a set of
> songs (set in the mathematical sense, so no ordering at all).  Each song
> has a URI associated with it; RhythmDB maintains a bijective mapping
> between URIs and songs.
> 
> One issue with this is that RhythmDB has no notion of directories.  But
> it should, because this allows us to make some important optimizations. 
> For example, as I mentioned in the bug, we can just stat() each toplevel
> directory on startup, and if those haven't changed, we're done.
> 
> More generally though, we want to monitor directories for updates at
> runtime as well.  So what we really need is for RhythmDB to maintain a
> specialized kind of indexed filesystem, and to keep it in sync with the
> real filesystem.  The RhythmDB filesystem will never be written to
> directly.  When the user clicks delete on a song, we will simply delete
> it from the real filesystem; then, the fs thread will eventually notify
> us of this deletion, because it'll be watching the directory.
> 
> RhythmDB should primarily store (filesystem, inode) pairs, for both
> directories and files.  That's sufficient for unique identification. 
> These are the "device" and "inode" members of a GnomeVFSFileInfo.  Note
> that storing things this way solves the recursive symlink problem; if
> during a recursive directory traversal we hit a (filesystem, inode) pair
> that's already in the DB, we just ignore it.
> 
> However, this doesn't work for non-local filesystems.  So we'll still
> have to store URIs for those.  So I am thinking the structure will look
> something like this:
> 
> struct RhythmDBFile
> {
>   gboolean is_local;
>   gboolean is_dir;
>   dev_t device;               /* Both of these are invalid if */
>   GnomeVFSInodeNumber inode;  /* is_local is false */
>   time_t mtime;
>   char *name;  /* UTF-8 */
>   union {
>     GHashTable *dirents;  /* used if this is a dir */
>     RhythmDBEntry *entry; /* used if this is a song */
>   } data;
> };
>  
> Now, RhythmDB stores a set of RhythmDBFiles that are the "roots".  It
> also stores a hash table which maps (device, inode) pairs to
> RhythmDBFiles.  This is because updates could come in two forms; our fs
> monitoring thread will tell us things like (3, 3532523) has been
> deleted.  Then we need to look that up in the hash table, map it to a
> RhythmDBEntry, and signal the main thread that that entry has been
> deleted.  
> 
> But we also need to be able to handle plain URIs; as I mentioned for
> remote filesystems we don't have inodes.  So the strategy there will be
> to try matching the URI against each root in turn.  If there's a match,
> then we recursively traverse the directory pointers, matching up URI
> components, until we get to a RhythmDBEntry.  This should be reasonably
> efficient.

That sounds good, but remember that the overhead in monitoring a huge
number of files is not in fam (which monitors the directory above) but
in gnome-vfs.

> So, thoughts?

Looks good to me, at least you won't be the only person understanding
the database code after that ;)

---
Bastien Nocera <hadess@hadess.net> 
She had a deep, throaty, genuine laugh, like that sound a dog makes just
before it throws up.