Re: [Evolution-hackers] Spam filtering thoughts

[Not Zed <notzed ximian com>]

On Fri, 2003-08-08 at 16:19, Ettore Perazzoli wrote:
> I think we need to have some built-in anti-spam functionality in Evo
> 2.0.


>       * Support black lists and white lists.  (This could be optional.)

So we need to write a gui editor for a spamassisn config file and crap
like that ?

> For the actual spam detection, I think we should just use Spamassassin. 
> It works great, is actively maintained, and is very simple to interface
> to.  It also does a great job without any training, although it does
> support Bayesan filtering.
> 
> (If at some point some better solution shows up, it should be fairly
> easy to either switch to the new solution, or make the interfaces
> generic enough to make the system fully pluggable.  However, I don't
> think pluggability is a must; let's focus on the user experience first.)

i think pluggability is more important but whatever ... it would be
cheap with a decent design anyway (as opposed to a nasty quick-hack).

> >From the mail side of things, we could do something like this:
> 
>       * We add a way for Camel to mark a message as spam or not.  It
>         should probably have to be a bit in the summary, the same way we
>         handle things like colors/labels.

Nothing needs to be done to camel to support this (other than the
detection, which can also be done without camel changes).

>       * We invoke Spamassassin in spam detection mode every time a new
>         message goes through the mailer, and set the corresponding bit
>         in Camel according to what it tells us about the message.

it should really only be new inbox mail.  if people do server-side
filtering they should also do server-side spam detection.  otherwise
we'd basically have to do a spam check every time you open a message, or
have 2 bits one for 'spam checked' and one for 'spam status', yuck.

>       * We give the user a way to say what happens to messages detected
>         as spam; i.e. whether they should stay in the folder, or moved
>         to a "Junk" folder or deleted.  (This should probably be
>         separate from the general filter dialog, because this choice
>         should be available without the user having to understand
>         filters.)

Not sure I agree here.  Another ui for the same thing?  It also has to
integrate with filters for when filtering is requested.

>       * We put a button in the mail toolbar to mark a message as spam or
>         not spam.  When a message gets marked by the user as spam or not
>         spam, Evolution sends it to Spamassassin to train the filter
>         accordingly.

a lot of toolbar buttons already ...

>       * As an additional aid to the user, messages detected as spam
>         could be displayed with a header saying "this message is spam,
>         if it's not click here to mark it as non spam", like in Apple
>         Mail:
> 
>                 http://www.apple.com/macosx/jaguar/images/mainmailbox.jpg
>                 
>         This button would behave exactly like the "not spam!" toolbar
>         button.

if we do that i dont know whether we should bother with a toolbar item

>       * We add a command to delete all the messages marked as spam in
>         the current folder.
> 
> Now the possible issues:
> 
>       * In the case of IMAP mail, when would the spam value get
>         measured?  We have to download the message before we decide
>         whether it's spam or not.  I guess if we make Evolution download
>         all messages by default in the background for offline then we
>         can just hook the spam detection to that.  (But if the user
>         turns that off, then detection wouldn't happen until she opens
>         the message.)

We could also just disable it.  Otherwise we'd have to download all
messages in all folders to perform spam checking on it, esp if we use a
vfolder for spam.

>       * We would probably want it to automatically recognize messages
>         that went through a spam filter in the server already.  I.e.
>         Evolution could recognize the X-Spam headers and turn the "this
>         is spam" bit on automatically when appropriate?

Yeah, easy.  Although then we run the risk of people who aren't using
spam filters being duped by spam adding that header and saying it isn't
spam.  So it would probably have to be an explicit option for
environments where the x-spam-status can be trusted.

>       * Should the "Junk" folder be implemented as a vfolder, like the
>         Trash?  Then messages wouldn't actually move back and forth
>         between folders, and if there is a false positive the user
>         marking the message as "not spam" would automatically result in
>         the message disappearing from the vfolder and appearing in its
>         original folder.

Sounds reasonable to me.

Although then (actually with any junk folder), you have to do spam
checking all the time.  If you didn't have any junk folder, or it was
just part of a normal filter rule (i.e.  message is junk rule), then
you'd only have to do it on displayed messages, and not need to record
it in camel or anything.