Re: [Evolution-hackers] Folder Compression (redux)

[Not Zed <notzed ximian com>]

On Tue, 2005-01-18 at 18:17 +1100, Michael Pearson wrote:
> I'd like to implement compressed folders in evolution.
>
> I've searched lists.ximian.org and read the "compression for folders?"
> thread of May 04, and I've looked at bug 23621. My notes and own
> proposal (which is quite similar to that made by Todd Fries, of whom I
> am trying to contact) is below.
>
> Note that, based on what I've seen so far of how evo works, I'm making
> gross assumptions about camel, evo, and evo-data-server in this email.
> Please correct me if anything that I'm suggesting is unworkable.
Ok, will do.
> COMPRESSION VS ARCHIVAL
> I agree, to an extent, that the only folders which will need to be
> compressed are those that would traditionally be marked as "archived",
> as well. As a systems administrator with several gig of old cron
> output, this is certainly the case in my situation. It should,
> however, be possible to write to archived, compressed folders, but it
> needs to be made clear to the user that this will take quite a bit
> longer than usual for large operations.
>
> If it makes you feel any better, substitute "archive" instead of
> "compress" throughout the rest of this email.

> TRANSPARENT COMPRESSION
> By "transparent" I mean transparent to the calling code, not the user.
> The compression folder backend should not require any real change on
> the part of non-compression code.
>
> The user, on the other hand, should know that the folder is
> compressed, and what that means.
Well yes and no, only that compression is an option, I guess.
> USER INTERFACE
> I'm not sure here. A checkbox in the folder property window?
>
> Actually.. a checkbox is a bit too minimalist - it hides the fact that
> there's a fairly intensive operation about to happen when "apply" is
> clicked. Instead, I suggest a "Compress this folder" button, and a
> "decompress this folder" button when the folder is already compressed.
> The "Compress this folder" button may show a warning to the user
> (which can be disabled), pointing out that while the compressed folder
> will save space
>
> It should be possible to select a range of folders and compress them
> all, or specify that all folders under this one will be created
> compressed. If we feel like going overboard, compressed folders can be
> shown with an icon next to them, or in a different colour, NTFS style
I'd rather not extend the local folders specifically to add compression to them.  I'd rather another 'compressed storage' mechanism, i.e. a new store (like imap or maildir are 'new stores'), not being able to applied to arbitrary folders (i see you suggest this later on, but probably misunderstand how this maps to the main ui).
> DISK IS CHEAP / THIS IS A USELESS FEATURE
> Yes... and no. It's not cheap when you have no money, which is my
> situation. I have many gig of ancient mail and cron output kept for
> sentimental value, like a heap of bad records stuffed at the back of
> the shed. I'm also out of disk space. Besides, if implimented in a
> properly modularised fashion, transparent compression would not add
> any maintenance overhead to other sections of code.
>
> IMPLIMENTATION
> Looking at the list discussion, I propose that messages be compressed
> in blocks of 100-500k. Compressing the whole folder and
> un/recompressing the whole folder every time a message is seen or
> modified is not workable. Neither is compressing individual messages -
> too much compression / cpu time is wasted.
The whole folder would be ok - provided you can seek reasonably efficiently on read, and open for append.  Messages are never altered once stored, only state about them is, and that can be stored separately.  But if you can't seek or can't append, then probably groups of say 100 messages or something might make sense.
> The only problem here is if a particular function specifies "do simple
> thing $a to messages $b[0-1000]", where messages 0-1000 are mostly in
> the same compression "block". Caching or "commit" calls are a possible
> solution, but risks adding overwhelming complexity. This may already
> be solved in the way that evo interacts with its folder backends, I'm
> not sure.
Well as above, messages are only ever appended to a folder, and removed from a folder (at expunge time).  They cannot be altered - only some meta-state about them can be, and you store that separately (we already do).  So this greatly simplifies what you need to support.
> I agree with the list that compressed folders should be implemented as
> a separate local folder backend. I however feel that they should be
> presented as something that can be done to a folder or an option on
> the folder rather than a separate folder type.
This is not possible.  The "local folders" branch is one storage (as above, like an individual imap branch is one storage), and it enforces a single folder type for all folders in that storage.  This was changed 1.4->2.0 to clean up some significant code issues.

--

Michael Zucchi <notzed ximian com> "Ride, Work, Sleep. Beer." Novell's Evolution and Free Software Developer