Re: [Evolution-hackers] A Camel API to get the filename of the cache, also a proposal to have one format to rule them all

[Jeffrey Stedfast <fejj novell com>]

Srinivasa Ragavan wrote:
> Hey Philip,
>
> [Im lagging in my mail-replies, still a lot to go, due to my 3 week
> vacation.]
>
> On Fri, 2009-01-02 at 13:25 +0100, Philip Van Hoof wrote:
>   
>> Hi there evos,
>>
>> For an EPlugin that I'm working on I will need a Camel API to get the
>> filename of the cache.
>>     
>
> Sure and the patch seems fine to me, but the Exchange portion of the
> patch is missing. It should be similar/simple.
>   
>> I will attach a patch that adds this API. The EPlugin that I'm developing is
>> available at Bug# 565091 and more information about it can be found at
>>
>> http://live.gnome.org/Evolution/Metadata.
>>
>>
>> I added a bug for tracking this request:
>>
>> http://bugzilla.gnome.org/show_bug.cgi?id=566279
>>
>> I know that for maildir (cur, tmp, new) and mbox (seek position) it's a
>> little bit controversial to return a filename. For maildir I always use
>> the cur-file one and for mbox I added "/!seek_pos" to the end of the
>> returned filename. 
>>
>> The reason why I need this is that for indexing already cached E-mails,
>> Tracker will MIME parse what we can MIME parse. For example filenames
>> and Exif data of attached images is stolen out of the cached items, to
>> be made searchable.
>>
>> We don't want to require Evolution to eat all the code involved in
>> indexing massive amounts of file formats. Best thing we can do right now
>> is to simply pass the filenames over IPC.
>>
>> We STRONGLY recommend to the Evolution team to:
>>
>> a) migrate away the IMAP specific data cache (see c to store separate parts)
>>     
> I thought we already store parts separate. Is is just about the encoding
> or more than that? I seriously don't have an idea on this. May be Fejj,
> Sankar, Matt can add on it.
>   

migrating away from the IMAP specific data cache would be good.

>   
>> b) migrate away the mbox data cache (the all-in-one file crap)
>>     
> I'm all for it. Once I thought of doing this, but the options were like
> Maildir or a format of one mbox file per mail in a distributed folder
> [CamelDataCache sort of format, like imap4/GW/Exchange]. But IIRC Fejj,
> had some concern like, Local still might be good to be held in a
> 'standards' way. I know it hurts us on expunge/mailbox rewrite etc.
>   

what mbox data cache? CamelDataCache would probably be the best cache to
use for IMAP.

>   
>> And to
>>
>> c) invent a better storage format that doesn't store the attachments in
>> server's (usually) Base64 encoding. The one format to rule them all.
>>
>> Instead store the encoded attachments in decoded format (original file
>> format). This will reduce diskspace (encoding increases diskspace usage)
>> and will make it more easy to scan the original file for XMP and Exif
>> information. Don't try to gzip or whatever anything. None of that makes
>> any sense (original files are usually compressed ideally already).
>>
>> For example: devices that want to compress have filesystems that do this
>> for you. Don't be silly trying to do this yourself.
>>
>> By storing the encoded version the only thing you currently gain is that
>> the feature "view E-mail source" doesn't need to recode the attachments.
>>
>> This ain't a much-used feature. It doesn't have to be fast, at all.
>>
>> No it doesn't. Really it doesn't.
>>     
> Is thatz it? I need some other opinions, I don't have much thoughts
> here. Sankar, Matt, Fejj?
>   

this can cause problems if you need to verify signed parts because
re-encoding them might not result in the same output.

>> For Maildir I recommend wasting diskspace by storing both the original
>> Maildir format and in parallel store the attachments separately.
>>
>> Maildir ain't accessible by current Evolution's UI, by the way.
>>
>> For MBox I recommend TO STOP USING THIS BROKEN FORMAT. It's insane with
>> today's mailboxes that easily grow to 3 gigabytes in size per user.
>>     
> I second your thoughts for MBox stuff. 
>   

Eh, I think mbox works fine but I can understand wanting to move to
Maildir which is also fine :-)

>   
>> Once all start using the CamelDataCache API, implementing that new
>> format and implementing converters wont be very hard. 
>>
>> For existing CamelDataCache users it's just one format to convert. For
>> IMAP, mbox, Maildir and mh it's indeed a few extra formats to handle
>> using a conversion. Wont kill you to implement that, and,  I'll help.
>>     
>
> Thatz so nice of you to help us :-)
>
> -Srini
>
>
>   

Jeff