[Evolution-hackers] Folder Compression (redux)

[Michael Pearson <mipearson gmail com>]

I'd like to implement compressed folders in evolution.

I've searched lists.ximian.org and read the "compression for folders?"
thread of May 04, and I've looked at bug 23621. My notes and own
proposal (which is quite similar to that made by Todd Fries, of whom I
am trying to contact) is below.

Note that, based on what I've seen so far of how evo works, I'm making
gross assumptions about camel, evo, and evo-data-server in this email.
Please correct me if anything that I'm suggesting is unworkable.

COMPRESSION VS ARCHIVAL
I agree, to an extent, that the only folders which will need to be
compressed are those that would traditionally be marked as "archived",
as well. As a systems administrator with several gig of old cron
output, this is certainly the case in my situation. It should,
however, be possible to write to archived, compressed folders, but it
needs to be made clear to the user that this will take quite a bit
longer than usual for large operations.

If it makes you feel any better, substitute "archive" instead of
"compress" throughout the rest of this email.

TRANSPARENT COMPRESSION
By "transparent" I mean transparent to the calling code, not the user.
The compression folder backend should not require any real change on
the part of non-compression code.

The user, on the other hand, should know that the folder is
compressed, and what that means.

USER INTERFACE
I'm not sure here. A checkbox in the folder property window?

Actually.. a checkbox is a bit too minimalist - it hides the fact that
there's a fairly intensive operation about to happen when "apply" is
clicked. Instead, I suggest a "Compress this folder" button, and a
"decompress this folder" button when the folder is already compressed.
The "Compress this folder" button may show a warning to the user
(which can be disabled), pointing out that while the compressed folder
will save space

It should be possible to select a range of folders and compress them
all, or specify that all folders under this one will be created
compressed. If we feel like going overboard, compressed folders can be
shown with an icon next to them, or in a different colour, NTFS style.

DISK IS CHEAP / THIS IS A USELESS FEATURE
Yes... and no. It's not cheap when you have no money, which is my
situation. I have many gig of ancient mail and cron output kept for
sentimental value, like a heap of bad records stuffed at the back of
the shed. I'm also out of disk space. Besides, if implimented in a
properly modularised fashion, transparent compression would not add
any maintenance overhead to other sections of code.

IMPLIMENTATION
Looking at the list discussion, I propose that messages be compressed
in blocks of 100-500k. Compressing the whole folder and
un/recompressing the whole folder every time a message is seen or
modified is not workable. Neither is compressing individual messages -
too much compression / cpu time is wasted.

The only problem here is if a particular function specifies "do simple
thing $a to messages $b[0-1000]", where messages 0-1000 are mostly in
the same compression "block". Caching or "commit" calls are a possible
solution, but risks adding overwhelming complexity. This may already
be solved in the way that evo interacts with its folder backends, I'm
not sure.

I agree with the list that compressed folders should be implemented as
a separate local folder backend. I however feel that they should be
presented as something that can be done to a folder or an option on
the folder rather than a separate folder type.

Regards,

--
Michael Pearson