Re: [GNOME VFS] Re: Daemons [Was: gob inside gnome-vfs ...]

[Seth Nickell <snickell stanford edu>]

On Thu, 2002-06-27 at 20:24, Maciej Stachowiak wrote:
> On 27Jun2002 07:52PM (-0700), Seth Nickell wrote:
> > 
> > I still don't see how this is different from any other file in the
> > system, presuming we store metadata as a file. 
> 
> It's different because most other files are changed atomically as a
> unit (for example, saving text in a text editor), but frequently
> multiple programs want to change different metadata keys for the same
> file, and this will often 
> 
> > Now it'd be one thing if we stored all the metadata in a single
> > file, but if we are storing metadata as a per-file thing on the
> > filesystem anyway...
> 
> It's actually stored as a per-directory thing now, although you could
> change that choice.

Ian and I are planning to change that for the GnomeVFS metadata
implementation, for a variety of reasons.

> > 
> > Now I can understand a potential problem with locking metadata across a
> > single process, as program using GnomeVFS async might conceivably be
> > performing a few operations on the file at the same time.
> > 
> 
> The real actual non-theoretical bug we ran into in Nautilus happened
> when the main Nautilus binary and one of the sidebar panels (notes)
> both tried to change the metadata file at similar times. The delayed
> writeback of metadata meant that one set of changes often got lost,
> leading to loss of real user data (the note about the file).
> 
> This was what convinced us that we needed a metadata server. What
> would your solution to this problem be?

Honestly I don't see how this is substantially different from the
theoretical case where, say AbiWord, had an out-of-process component
that made small modifications to the file you were currently working on.
How would you implement this? Would you suggest having a per-user daemon
that ensured clean locking for all files through GnomeVFS?

This general problem is not of pressing importance because, in general,
multiple programs tend not to end up accessing the same file at the same
time. If they do, we have traditionally foisted responsibility for the
ensuing problems on the user (i.e. if they open a file in two programs
at the same time and overwrite changes from one with the other, well we
let that happen, though we're supposed to warn the file has changed on
disk). I'm not really defending this view, because I think its broken
personally, but I don't see how metadata is substantially different.

Now on the other hand, different parts of the same program will often
access the same file, since they're all the parts of a document-based
program that is centered on that particular document. The Nautilus
problem occurs because Nautilus works like a single program, but in
actual implementation it has multiple processes. Thus I consider this
primarily a Nautilus engineering problem, unless we're interested in
solving the locking problem in the general case for file locking (which
I am, just not at the immediate moment).

I would suggest Nautilus use its current implementation, namely a CORBA
metadata server, but that it use GnomeVFS APIs to actually write the
metadata out to disk. That will give the desired locking between
Nautilus processes.

> > But the problem of multiple processes interacting over a single file
> > seems, to me, to be the same as the problem (and liklihood) of having
> > two programs modifying Document.abw or File.txt or whatever at the same
> > time.
> 
> However, when you save file.txt you typically expect to get those
> exact contents in the file, you don't expect changes to be merged. But
> it is reasonably to expect different metadata keys for the same file
> to be changeable independently without losing either change.
> 
> The metadata for a file is more like a directory than like a file. If
> two different programs add an entry to a directory at the same time,
> it would clearly be wrong if only one of the files actually appeared
> in the directory, right?
> 
> I see setting metadata keys the same way.

This is part of the reason I've briefly considered implementing metadata
as a hidden directory per file, with a file per key inside that
subdirectory. Its sort of nifty, except in the practical world in
consumes a lot of space on standard filesystems (which, other than
ReiserFS, don't store small file efficiently). The finer the granularity
we drop to wrt to # of keys or objects of metadata per actual file
stored, obviously we reduce risk of having contention for that resource.

-Seth