[Summary] Meta-data/filesystem-encapsulation

[David Jeske <jeske home chat net>]

I think a summary of what is being discussed will be useful
here. Please comment where you feel appropriate and I'll include it in
a future summary.

** Issues at hand:

1) a way to store resource data for items in the filesystem
(i.e. applications, shlibs, etc) so that items can be moved around
without breaking relationships.

2) a way for user's to setup item specific configuration such as:
    - application to open, edit, print, etc an item type
    - an icon for that item or item type


** Previous solutions to the problems:

   * MacOS stores resource forks in the file and provides an API to access
     them.
     + in the beginning apps were just one 'item'
     + apps could be moved around
     + type/creator information allowed application preferences for a file
     - Mac resource are limited
     - later on, apps were no longer one item because resource are ill
       suited to hold 'files'.
     - type/creator system was fixed to four character keys
    
   * NeXT uses a 'wrapper' directory
     + arbitarary resource data could be stored and accessed via
       relative paths from the app lanuch location
     + apps could be moved around
     + type system kept track of application preferences for a file type
     + worked with the underlying UNIX filesystem unchanged
   +/- used extensions to determine filetype (depending on who you
       ask this is good or bad)
     + old-school UNIX programs could move apps around without breaking them
     + because wrappers are basically directorys, apps are truly atomic,
        people just stuffed whatever they needed in them, and apps access
        that data through a _relative_ path.
     - launching apps from the command line was different than from the
       GUI.

   * OS/2, NT(?) had some kind of file attributes, not used much

   * BeOS provide arbitrary attributes on files, the system uses it heavily
     + use MIME types to define filetypes
     + type system keeps track of application preferences for a file type
     + can override the 'default application' for this mime type on
       a per-file basis
     - attributes are named with strings, collisions are immenent
     - attributes have no owners, so there is no easy way to add/remove
       them across the system
     - not all application datafiles get stuffed into attributes, so
       apps are still not atomic.

** Possible solution routes (in order of decreasing complexity):

A) make a new filesystem (ala BeOS) and change the filesystem/kernel API

B) graft a meta/encapsulation system into the existing system without
changing it. Come up with backward compatibility so old apps don't
break associations.

--------------------------------------

Below I'm going to summarize existing solutions, if you have an
additional solution, post it, or email it to me, and I'll include
it. If you have comments on one of these solutions, please email them
to the person who proposed the solution.

** Solution Proposed by David Jeske (jeske@chat.net):

I think we should start with a system much like NeXT used, because
they had a similar goal. That is, they wanted to put an
encapsulation/resource system on top of UNIX with minimum distruption
and maximum gain. Another support for this is that GnuStep already
does it (although this isn't enough by itself).

If you don't undestand the explanation below, please email questions
to me directly, I'll explain them to you and include the description
in a future summary.

1) use extensions for type information (of both files and directories)
2) encapsulate all 'multi-file' items on the system into a directory.
3) require apps and code to use relative paths!
4) make a type/icon/app server which will keep track of things like
   applications to launch for a type, extension -> MIME type mappings,
   BUT...
5) have apps/wrappers passively export type information (i.e. no
   configuration files which have to be hacked when an app is installed)
   In other words, the 'type/icon/app server' should just be a _cache_ of
   information which is pulled from every app-wrapper. 
6) create a STANDARD 'app lanucher' for UNIX shells which will use the
   type/icon/app server to launch an app regardless of it's location
7) modify the UNIX "#!<app>" shell convention handling to allow 
   redirection to the 'type/icon/app server'.
8) use an improved version of NeXT 'services' so apps can perform services 
   other than just 'open'.

The tradeoffs of this are
 - we can't easily do BeOS style 'app preference per file'
 - types are defined by extensions (some don't like this)
 + types are defined by extensions (this is how it is on the internet)
 + apps can be moved around
 + old UNIX programs (like cp, mv, etc) can still deal with the wrappers,
   because they are just directories
 + apps can be truly atomic, because they can store any files they want in
   their directory
 + we can provide a way to allow old school UNIX to launch new apps
   without stuffing them all in the PATH.
 + shell scripts can be installed and run without hacking their "#!" lines.
 + an app can be tarred up, copied to another machine, untarred, and it'll
   'just work' as if it was installed there. 


Examples/Information about the points above:

3) by requiring apps to use relative paths to get to their components,
   we allow administrators to have control over where apps are installed.
   In addition to other issues, this allows things like:
    - user installing apps in their own directories
    - multiple versions of the same app
    - pre-compiled binaries to be installed anywhere

4/5) The type/icon/app server should:
    - scan a pre-set (by the admin) set of paths for wrappers. If it
      finds a wrapper, it can look inside for information about what
      types that app handles, or what kind of wrapper it is. This prevents
      problems like the NT registry, or the current UNIX config file system,
      where the type information can get out of data with respect to the
      applications
    - store the user's preferences about what app to open for each item
      type, and what icon to display for each item type.

6/7) by having UNIX shell 'app launching' go through this new system, we
     can solve problems which have existed forever with launch apps on old
     UNIX. For example:

    - we can create a way to unify the different mechanisms to configure
      app launching. Currently in old school unix, there are at least
      three ways to decide what apps to launch:
        - environment variables (EDITOR)
        - configuration files (custom for every app)
        - hard coded (i.e. scripts with #!/usr/local/bin/perl)

      We would be able to both improve/replace all of these (for GUI apps)
      and remain backward compatible with them. For example, we could
      map "#!<name>" to our app launcher, where the admin/user could
      configure what app was launched. The app server would figure out
      where that app was located. We would be able to do things like
      "set EDITOR=///EDITOR" and have "///EDITOR" trapped and redireted
      to launch the user's preferred editor. 

      More importantly, we could do away with custom configuration files
      for every single app to configure what it would launch for different
      operations and filetypes.


** Solution Proposed by Leareth <leareth@geocities.com>:

[NOTE: I don't understand this proposal. In fact, I don't understand
many of the sentences, I'm just pasting it from an email Leareth
posted. - jeske@chat.net]

So, a multi user os, with a multi threading file system, that doesn't
support any sort of extended data beond what it was originaly designed
for, and your not able to depend upon but the options available from
chmod, as all other's a sys spec.
 
So, you either make a central database & library interface to it,
which may or may not alow for segragated user data, and if files are
manipulated outside of the library, links could keep going dead
continuisly causeing endless amounts of truble. Or you keep user
metadata databases, belonging to the user, kept in there home
directory, and manipulated through a library. still looses track of
file that are moved/renamed outside of the library.
 
Neither option really works. Here's one that does.
 
Someone, awhile ago, wrote a library called libvfs, that alowed any
app linked to libc, to open things as mc(vfs orig) did, without
recompileing or relinking anything.
 
Now here's we we get creative. We build both central & user databases,
the central with a list of files that contain the name(s) of the users
that need to be updated when the corisponding file is changed. Each
user, keeps a database of metadat on all files that have been
extended, includeing files from remote sites. This database will be
updated by a library that set & get's the data, and is updated by
libvfs when a file is changed through normal means such as mv. This
will keep all meta data updtaed, and user spec.
 
Complex files, suck as opendoc file, that know about how to
view/print/edit them selves, comtain there own meta data, makeing them
universal(same for all users, reguardless of who makes the
changes). The user database just redirects all setting to be set in
the file it self.
 
Comments please? 

-- 
David Jeske (N9LCA) + http://www.chat.net/~jeske/ + jeske@chat.net