Re: Performance

["Manuel Amador (Rudd-O)" <amadorm usm edu ec>]

> EA wont fix everyone's problems 

thats true. but it will fix some.  or most.

> since not everyone has a file system
> that support them.  Building a half solution means building the other
> half solution later.

you can either build the second half or wait till everyone rallies round the
first one, which is very probable - after all AFAIK 1) they're a posix standard
2) they're already included in fairly recent Linux kernels, which represent the
great majority of UNIX desktops running GNOME.

>  EA are faster,

true

> but because they dispersed across
> the file system they must be accessed individually;
> a directory with a
> thousand files needs a thousand lookups.

depending upon the implementation, the thousand lookups might mean only two or
three disk reads.  You're already *accessing the files* to sniff for content,
which is much slower.

>  Worse still, EAs take up a few
> megs per gig of file space,

depending upon the implementation.  The ReiserFS implementation does not, and
it's perhaps the strongest candidate.

> but you cannot really query the EA space to
> locate a group of matching files.

You will at some point.  For now, sniffing the files for content is a much, much
more inefficient solution, and a terrible burden on network file systems.

> 
> Cache needs a smart clearing mechanism.  It can become a burden.  If
> some the the file system's are not attached to FAM, Nautilus may not
> display the correct directory contents.

Or it can switch to polling.

> 
> > What about if it were an XDG standard? Ultimately you'd think real 
> > metadata would be a better solution than either sniffing or file 
> > extensions...
> 
> I think a metadata API was discussed a few weeks ago on a GNOME list.
> 
> I've put some thought to the issues here from search angle.  The Medusa
> indexer and search engine knows quite a lot about the posix file
> information and keeps it in on central place for fast access.  I intend
> to extend replace Medusa's backend storage with a real database and a
> new API that will store EA information for search and for any app that
> wants to manage data (file manager, music manager, etc.).  

Medusa is useful, but you can't really expect it to become more prevalent than
ReiserFS for example.  Medusa does mean a reliable metadata store, but a
filesystem solution (now available) is a much more robust solution with less work.

I don't think we should abandon medusa - it does provide valuable services,
speeds searches up for example, but I don't think it's the right store for
metadata.  I think metadata belongs closest to the data, and Medusa should be
only a CACHE, useful for searches, but not the authoritative source of
information.  Keeping Medusa updated in real-time is bound to be more difficult
than keeping EAs updated.

> 
> My ideas for a metadata database sit somewhere between the current
> Medusa and Storage, and relates some what to the nautilus performance
> problem.  The advantage of a VFS metadata database is that it works for
> all file systems,

This is true.  I mean that the implementation for local file systems should rely
on EAs, and use the medusa database for caching EAs and full-text search data.

> it can be queried for a single file or a group of
> files, it is accessible as a file system and an API for smart apps. 
> It's weaknesses, like the cache problem, is that it must struggle to
> keep current.  Using and indexer and FAM, will help, but there will be a
> delay.  An app like Nautilus will require a sanity check to be confident
> of the results it gets from the metadata database.

How feasible would it be to create a medusa-index-queued that used fam to detect
file changes, log the changed files into a queue (NOT UNIX fifos, think about
queues), and have the medusa-indexd reinded the changed files, on a first-come,
first-served basis?

This would nearly eliminate the need for periodic reindexing.

> 
> I've got detailed thoughts on the issue at
> http://members.cox.net/sinzui/medusa/index.html
> http://members.cox.net/sinzui/sutra/index.html
> http://members.cox.net/sinzui/blog.html
> 
> 
> 
> 
> -- 
> __C U R T I S  C.  H O V E Y____________________
> sinzui cox net
> Guilty of stealing everything I am.
> 
> -- 
> nautilus-list mailing list
> nautilus-list gnome org
> http://mail.gnome.org/mailman/listinfo/nautilus-list
> 


    suerte,

    Rudd-O

===========================================================
     UNIVERSIDAD TECNICA FEDERICO SANTA MARIA
                 CAMPUS GUAYAQUIL
        CENTRO DE SERVICIOS INFORMATICOS
Mail enviado a traves de IMP-USM: http://www.usm.edu.ec/imp
    Los invitamos a visitar  http://www.usm.edu.ec
===========================================================