Re: [Tracker] more issues with indexer-split

[Carlos Garnacho <carlos imendio com>]

Hi!,

On mar, 2008-08-12 at 14:18 -0400, Jamie McCracken wrote:

<snip>

> that sounds inefficient - trunk only ever checked for existing deleted
> or junk emails at startup because iterating through all emails in the
> summary files is expensive. 

> From what I've read in trunk code, you still iterate through all the
mails in the summary in check_summary_file(), and you will have to
iterate over them again later to index new messages, etc...

As far as I know, it's quite unavoidable to parse again summaries, since
under some circumstances Message IDs could be reused, which would leave
you with inconsistent data in the DBs. Even if it isn't, expunging a
folder would render any stored offset for the summary file useless (even
dangerous).

Besides, when testing summary parsing, I remember it was pretty fast
(like 2-3 seconds for a ~6500 emails summary), of course without
inserting to DBs nor doing message body or attachments sniffing, which
is more or less what should happen if the junk/deleted flag is set.

> the use of a separate junk email table meant
> lookups were confined to that table and not the services table so was
> faster when number of emails was high

You mean the JunkMails table in email-meta.db? As far as I see, this
table is just looked up to make sure there aren't duplicates when
inserting. And in the end, you still have to lookup/modify the Services
table, even if the junk mail wasn't there.

> we should also avoid doing this whenever the summary file changes which
> is why we stored an offset in trunk so we skip over messages to get to
> the new ones only when summary files change or do nothing if no new ones
> are present

As said above, I think there are pretty good reasons to avoid this.

> the trunk way is faster so i would prefer that restored

If you bear with me, I'd prefer to try a few optimizations before having
to add special cases.

Regards,
   Carlos

> thanks
>
> jamie
-- 
Carlos Garnacho
Imendio AB - Expert solutions in GTK+
http://www.imendio.com