Re: Performance

["Curtis C. Hovey" <sinzui cox net>]

On Mon, 2003-11-10 at 22:28, Darryl Rees wrote:
> Murray Cumming Comneon com wrote:
> 
> >>To: Alexander Larsson
> >>    
> >>
> >>(stuff deleted)
> >>what about storing the mime type in an extended attribute if 
> >>the file system supports them. If the file system doesn't it 
> >>will work as always has (doing mime type sniffing) but if 
> >>there is an EA containing the mime type we would have to go 
> >>trouht the sniffing process, and thus it would be faster.

EA wont fix everyone's problems since not everyone has a file system
that support them.  Building a half solution means building the other
half solution later.  EA are faster, but because they dispersed across
the file system they must be accessed individually; a directory with a
thousand files needs a thousand lookups.  Worse still, EAs take up a few
megs per gig of file space, but you cannot really query the EA space to
locate a group of matching files.

> >That sounds like just a more elegant cache to me, which wouldn't solve the
> >staleness problem.

Cache needs a smart clearing mechanism.  It can become a burden.  If
some the the file system's are not attached to FAM, Nautilus may not
display the correct directory contents.

> What about if it were an XDG standard? Ultimately you'd think real 
> metadata would be a better solution than either sniffing or file 
> extensions...

I think a metadata API was discussed a few weeks ago on a GNOME list.

I've put some thought to the issues here from search angle.  The Medusa
indexer and search engine knows quite a lot about the posix file
information and keeps it in on central place for fast access.  I intend
to extend replace Medusa's backend storage with a real database and a
new API that will store EA information for search and for any app that
wants to manage data (file manager, music manager, etc.).  

My ideas for a metadata database sit somewhere between the current
Medusa and Storage, and relates some what to the nautilus performance
problem.  The advantage of a VFS metadata database is that it works for
all file systems, it can be queried for a single file or a group of
files, it is accessible as a file system and an API for smart apps. 
It's weaknesses, like the cache problem, is that it must struggle to
keep current.  Using and indexer and FAM, will help, but there will be a
delay.  An app like Nautilus will require a sanity check to be confident
of the results it gets from the metadata database.

I've got detailed thoughts on the issue at
http://members.cox.net/sinzui/medusa/index.html
http://members.cox.net/sinzui/sutra/index.html
http://members.cox.net/sinzui/blog.html




-- 
__C U R T I S  C.  H O V E Y____________________
sinzui cox net
Guilty of stealing everything I am.