Re: [Evolution-hackers] compression for folders?

[Jeffrey Stedfast <fejj ximian com>]

On Tue, 2004-05-11 at 15:35 -0500, Todd T. Fries wrote:
> I find your arguments amusing.
>
> On one hand, you diss my ideas in general because of a specific case
> that compressing an mbox is too cpu and io intensive of a process to be
> a useful mechanism.  Then you turn around and argue for this very thing
> because it is simple.

uh, you are totally misinterpreting what I said.

> Let me be clear.
>
> I have experienced the very mbox compression madness with mutt.
>
> It is true that you can append gzip streams one ontop of another, and
> decompression never knows the difference from my perspective.
>
> Aka:
>
>    echo hi | gzip > foo.gz
>    echo ho | gzip >> foo.gz
>    echo fun| gzip >> foo.gz
>    zcat foo.gz
>    hi
>    ho
>    fun
>
> .. works just fine.

but it's slower than if hi, ho, and fun were all gzipped in one stream. to find the end of the gzipped 'hi' stream, you have to continually test of EOS by comparing 8 bytes (crc32 and isize). If they match, then you've found the end of the gzip stream, else continue.

I never said that catting gzip streams back to back doesn't work. I just said that handling that is a lot more expensive.

> Mutt utilizes this ad nauseum.

no wonder you've been complaining that it's too slow :-)

>   To save to a folder with a '.gz'
> extension, it simply does 'gzip -9 < /tmp/tempmessage >> folder.gz'.  To
> read a folder with a '.gz' extension, it decompresses the whole thing to
> a temp dir, does a normal mbox open on it, and recompresses the whole
> thing before writing it back to the original folder name if you make any
> modifications.

right, because there's no other way.

now just imagine having a massive mailbox and having to do that all the time. SUCK!!!!

hence why I don't want to allow users to compress arbitrary folders. it just isn't worth it.

if you're gonna start inflating everything to /tmp anyway, wtf even bother to compress it? just seems silly to me.

implementing seek support for a gzip stream is likely the way I'd go about reading a gzipped mailbox, trying to avoid decompressing it to disk (or memory) as much as I reasonably could. Obviously appending or modifying the mailbox would not allow that, and I *would* have to inflate it. But in the normal operation of just needing to be able to read from the gzipped mailbox, I wouldn't ever bother inflating the whole thing. Don't need to. Sure, accessing messages won't be as fast, but because the folders would all be achives, the user would find this acceptable.

Anyways, it's your time. You can spend it any way you want.

Jeff

-- 
Jeffrey Stedfast
Evolution Hacker - Novell, Inc.
fejj ximian com  - www.novell.com