Re: [Evolution-hackers] A Camel API to get the filename of the cache, also a proposal to have one format to rule them all

[Srinivasa Ragavan <sragavan novell com>]

Hey Philip,

[Im lagging in my mail-replies, still a lot to go, due to my 3 week
vacation.]

On Fri, 2009-01-02 at 13:25 +0100, Philip Van Hoof wrote:
> Hi there evos,
> 
> For an EPlugin that I'm working on I will need a Camel API to get the
> filename of the cache.

Sure and the patch seems fine to me, but the Exchange portion of the
patch is missing. It should be similar/simple.
> 
> I will attach a patch that adds this API. The EPlugin that I'm developing is
> available at Bug# 565091 and more information about it can be found at
> 
> http://live.gnome.org/Evolution/Metadata.
> 
> 
> I added a bug for tracking this request:
> 
> http://bugzilla.gnome.org/show_bug.cgi?id=566279
> 
> I know that for maildir (cur, tmp, new) and mbox (seek position) it's a
> little bit controversial to return a filename. For maildir I always use
> the cur-file one and for mbox I added "/!seek_pos" to the end of the
> returned filename. 
> 
> The reason why I need this is that for indexing already cached E-mails,
> Tracker will MIME parse what we can MIME parse. For example filenames
> and Exif data of attached images is stolen out of the cached items, to
> be made searchable.
> 
> We don't want to require Evolution to eat all the code involved in
> indexing massive amounts of file formats. Best thing we can do right now
> is to simply pass the filenames over IPC.
> 
> We STRONGLY recommend to the Evolution team to:
> 
> a) migrate away the IMAP specific data cache (see c to store separate parts)
I thought we already store parts separate. Is is just about the encoding
or more than that? I seriously don't have an idea on this. May be Fejj,
Sankar, Matt can add on it.

> b) migrate away the mbox data cache (the all-in-one file crap)
I'm all for it. Once I thought of doing this, but the options were like
Maildir or a format of one mbox file per mail in a distributed folder
[CamelDataCache sort of format, like imap4/GW/Exchange]. But IIRC Fejj,
had some concern like, Local still might be good to be held in a
'standards' way. I know it hurts us on expunge/mailbox rewrite etc.

> 
> And to
> 
> c) invent a better storage format that doesn't store the attachments in
> server's (usually) Base64 encoding. The one format to rule them all.
> 
> Instead store the encoded attachments in decoded format (original file
> format). This will reduce diskspace (encoding increases diskspace usage)
> and will make it more easy to scan the original file for XMP and Exif
> information. Don't try to gzip or whatever anything. None of that makes
> any sense (original files are usually compressed ideally already).
> 
> For example: devices that want to compress have filesystems that do this
> for you. Don't be silly trying to do this yourself.
> 
> By storing the encoded version the only thing you currently gain is that
> the feature "view E-mail source" doesn't need to recode the attachments.
> 
> This ain't a much-used feature. It doesn't have to be fast, at all.
> 
> No it doesn't. Really it doesn't.
Is thatz it? I need some other opinions, I don't have much thoughts
here. Sankar, Matt, Fejj?
> 
> For Maildir I recommend wasting diskspace by storing both the original
> Maildir format and in parallel store the attachments separately.
> 
> Maildir ain't accessible by current Evolution's UI, by the way.
> 
> For MBox I recommend TO STOP USING THIS BROKEN FORMAT. It's insane with
> today's mailboxes that easily grow to 3 gigabytes in size per user.
I second your thoughts for MBox stuff. 

> 
> 
> Once all start using the CamelDataCache API, implementing that new
> format and implementing converters wont be very hard. 
> 
> For existing CamelDataCache users it's just one format to convert. For
> IMAP, mbox, Maildir and mh it's indeed a few extra formats to handle
> using a conversion. Wont kill you to implement that, and,  I'll help.

Thatz so nice of you to help us :-)

-Srini