Re: Writing a GNOME mail client.

[Warren Young <tangent cyberport com>]

Michael Lausch wrote:
> 
> >>>>> "sw" == Scott Wimer <scottw@dev.cgibuilder.com>
> >>>>> wrote the following on Sat, 17 Apr 1999 18:45:55 -0700 (PDT)
> 
> sw> Matthias,
> sw> The problem with the file system approach is that it only will cover
> sw> a simple model of the end user environment.  Granted, it would also
> sw> be potentially easier to implement.  However, I fear that the time
> sw> spent implementing it would endup being tossed because the model just
> sw> doesn't fit a wide enough range of environments.
> 
> And it already exists. MH uses one file for one message and directories as it's
> folders. The whole mail system consists of unix programs which are
> combinable via pipes or parameter substitution.

Let's also not forget reliability: programs crash.  If as a result of a
crash this fancy database gets corrupted, you'll likely lose access to
all of your email and news.  If instead we use the filesystem, you'll
probably only lose a few messages in a crash.

Distilled, my view is that there are two models to choose from: the
Windows Registry and CVS.  Me, I'd prefer the CVS approach.

Both systems try to hide their data from the end user.  With the Windows
registry, you have to use the regedit tool or the Reg* Win32 functions
to access it.  With CVS, you have to use the cvs tool to access the
data.  But, if things go badly, CVS at least stores the data in a way
that still allows access with cd and vi.

Consider this layout (fixed-width font required):

gmail                    -- GNOME mail/news directory
|-- mail                 -- mail folders
|   |-- Gnome-L          -- Mail from the gnome-list, filtered from
Inbox
|   |   |-- 05927923832  -- A mail message (files in dir not shown)
|   |   `-- 32874932729  -- Another mail message
|   |       |-- control  -- Msg metatata file, like in Debian packages
|   |       |-- foo1.jpg -- An attachment, named in the control file
|   |       |-- foo2.tar.gz -- Another attachment
|   |       |-- links    -- Another control file, perhaps used by a
module
|   |       `-- msg      -- The text of the main message
|   `-- Inbox            -- The main mail inbox
`-- news                 -- Contains newsgroups
    `-- alt.hackers      -- A newsgroup
        `-- 75082790232  -- A message in that newsgroup
            |-- control  -- Similar in format to mail/*/*/control
            |-- links    -- Ditto
            `-- msg      -- Ditto

The only "crowded" directories would be those with many messages. 
Perhaps to combat this the system could add another directory layer with
numbered directories and partition a set of messages to each directory. 
E.g. mail/Inbox/001/573922/... and mail/Inbox/002/497509/...

The control files would be text files, fairly easily parseable, but also
human-readable and modifiable at need.  I envision a file closer to
Debian's package control file than to the CVS control files that live in
the CVS/ subdirectories.  The latter sacrifice readability for easy
parsing.

Each message is split into parts, parsed along MIME lines.  The control
file shows how to stitch these pieces back together so the user can do
things like forwarding the message.  If the user elects to delete one of
the attachments to save space, or move it somewhere, the mail system can
be resilient and simply indicate that the attachment is no longer
available.

The msg file probably only contains the text of the message, with the
headers in a separate file, or parsed out into the control file.  Or,
perhaps a bit of both.  The original headers could be split out into a
hdr file, with a few of those headers extracted and cleaned up to go
into the control file.  For example, you don't need the
"X-ImpeachClinton: vote yes" header in the control file, but you'd want
to preserve it.  The To: header would definitely go into the control
file.

An interesting feature of this layout is that each message directory can
have files in it that the main mail program didn't create.  In my
example above, the links file is generated by a plugin that lets you
create a "network" of messages that link messages to each other in ways
that make sense to the user.  

I think this idea alone is quite powerful, but the system allows for any
number of plugins.  Perhaps encryption would be an optional plugin that
saves its data in a "key" file in each message directory.  The gmail API
would expose a way of walking the above directory tree, and then
attaching a special control file to the message.

I don't think this would be too hard to implement.  It's simple,
straightforward and robust.

There's one other strike against using a relational database for this:
it's hard to map hierarchical data to a relational schema.  A
filesystem, on the other hand, is already hierarchical.  Now if we can
find a free hierarchical database system _maybe_ that would be a good
storage engine.  But I don't see that it would have a great amount of
advantage over my idea.

Comments?
-- 
= Warren -- http://www.cyberport.com/~tangent/
= ICBM Address: 36.8274040 N, 108.0204086 W, alt. 1714m
= Chain tagline - Stolen 378 times - Add 1 when stolen.