Re: About metadata (long!)

[Kevin Littlejohn <darius connect com au>]

(I'm snipping a whole heap of stuff because this is getting too long.
I'm trying to restrict down to these points only at this stage...)

> 
> > I have two questions I'd like answered re: extended attributes:
> > 
> > 1) How do you deal with many people wanting different values for the same
> >    attribute on the same file?  (eg. different viewers for the same particular
> >    pictures).
> 
> This is all stored in the users' preferences database.

So if a file has an attribute 'viewer', and every user on my system wants
to use a different 'viewer' for a particular file, we _won't_ store the
'viewer' metadata in the extended attribute?

What _are_ we storing in the extended attributes?

> 
> > 2) How do you deal with trying to assign attributes to files you only have
> >    read access over?
> 
> It depends on the attribute.  If you have the ability to override the
> attribute, then it is stored in your local database.  If not, then you
> can't change it, just like you can't change the text in the file.
> Example: if the file has a copyright attribute, you can't change it.

So, what I'm trying to figure out is, exactly what are you supposed to be
gaining from changing the data on disk?  It seems like this has narrowed
down to 'if we can tinker with this structure on disk, then we will, and
we'll store something in it - but not if someone else has already stored
something in there, or if we don't have write access to the file, in those
cases we'll fall-back to the main system.

If I'm getting this right, you're suggesting extensions to the linux
ext2 filesystem, which I respectfully suggest is _not_ a GNOME project,
but a linux/ext2 project.

> 
> > My proposal is still (for reference :) :
> > 
> > A system-wide, and a more specific person-wide database (both of the same
> 
> person-wide?

user-wide:)  Sorry, laziness rears it's head :(

> 
> We need:  A system-wide "hints" or "class" database.  mime.types, eg.
>           A system-wide "attributes" database (or store it in the fs)

This is what annoys me - "_or_ store it in the fs".  That's plain _wrong_ -
if you wanna store stuff in the fs, be my guest, but _don't_ suggest that
storing stuff in the fs should in any way, shape, or form enter into the
design of the database.  Storing stuff in the fs is a _complete_ extra,
something that can be considered way down the track, probably as a standard
method for files that match certain prereqs - if file->attr has 'extended',
and query is for 'view', use 'file->extattr->view'.  That sort of thing
should be supportable by a solid, _generic_ database - without putting
"or's" into the design suggestions.

>           A user-level "attributes" database (to override defaults)
> 
> #2 is the hard one, because you need to essentially duplicate the
> functionality of the kernel, plus attribute attributes.  It will also have
> to be SUID/SGID so that any user can modify it for their own files.  A
> better solution may be the 'xv' style database so that users can own their
> own attribute db, but that doesn't work when you have mixed UIDs in a
> single directory, so can't really work.
> 
> > format), that can return methods for queries on files.  The database itself
> > should be, probably, a .db file, or integrated into Corba, or something
> 
> Hmm.  If it is a standalone file, then .db is probably okay, but for
> integration directly into the fs, it should be text.  Something like:
> 
> Signature: GNOME-Metadata
> Reference.file: /usr/local/share/example.file
> Reference.size: 43218 bytes
> Reference.time: (utime), (mtime), (ctime), &c
> Reference.owner: (uid), (gid), (perms)
> Comment.1: This an a comment that must follow this file, it cannot change
> Comment.1: Additional comments may not be added.
> Comment.1.perm: FmL  {Force-on-copy, !Malleable, Locked}
> Method.1: View
> View.1: xv
> View.2: display
> Method.2: Edit
> Edit.1: gimp
> 
> 
> I'm sure this is hardly ideal, but it should be a fair starting point.
> Keeping the backreference at the beginning of the file should help in
> reassociating the data if it were ever lost.  Hostname, IP, ethernet
> address, something to indicate the machine it originated from would be
> helpful as well...
> 
> There's a lot of stuff this doesn't deal with (regexp based defaults -
> mime.types) or probably a myriad of other things, but I did it quickly and
> with a pounding headache now, thanks alot guys.  :)

That stuff is pretty much what we should be looking for, to my mind - I
think the PRELOAD stuff and the ext2 stuff is smokescreen over the actual
question, and I personally would like to see them thrown out until we have
a solid database to start with.  Also, (as per following comment), if we
design the database carefully, these things should be easy adds later on...
I keep picturing something similar, along the lines of:

gfdb_query(filename,'view') checks through a db file for anything that
matches filename (either by name/regexp or by hash, or whatever), and
returns the method 'view' associated.  I think if you can manage to
create some sort of session management, you can add cacheing of the responses,
which is good for performance but again really pretty irrelevant to the
design, in a lot of ways.

[snip] 

> Good ideas, but I don't want to think about the right now.

The trick is to design the database in a generic enough idea that, while
not necessarily designing _for_ this sort of added functionality, we leave
the door open enough for extensions to later have someone come along and
say 'hey, check this - if I set this here, then I can do this!' :)  The sign
of a well-designed system - extensible, flexible, _and_ meeting the
original requirements :)

I'm wondering if maybe we're aiming toward subtly different things - from what
I'm understanding, you're aiming toward the files being objects, keeping
as much information about themselves as possible - so if you move a file
from your system to mine, it'll keep it's info.  This seems really good
for things like author, copyright, creation time, version, source, etc.,
but really bad for things like editor, viewer, copy_program, etc.
I _like_ your idea for the first (author etc) - although I think you're
fighting a very big battle, and fighting it in the wrong place - I
believe I've seen discussions about using ext2 attributes before on
linux-kernel mailing lists, and the fact that they've not gone anywhere
despite having dedicated people involved suggests to me that the issue
may be larger than I'm aware of...  I _don't_ like your idea for the
second, as I'm suggesting a way of me personalising my desktop in a way
that is consistant, and portable for me - so I can determine that I
use this viewer for that type of file, and modify the behaviour of
my interface to the computer as I see fit - and then carry that environment
either via networked database engines (ala ACAP, as pointed out by someone
else), or via floppy-copy of the relevant .db files, or whatever.  I don't
want other people's preferences re: viewers and editors and so forth to
extend to my desktop, thanks :)

I'm hoping that the suggestions I've made can be carried out to a system
that will cover your requirements too - gnome-copy programs that export
the relevant meta-data to whatever media you're pushing the file to, so
you _can_ shuffle things around _with metadata intact_, so long as you
use the gnome-tools.  I think that's reasonable - I hope so, anyway :)

KevinL