Beagle roadmap.

[Nat Friedman <nat novell com>]

Hi guys,

Our little search project is growing, so I'm going to hand out some
tasks and goals to those Novell people working on Beagle.  I decided to
send this to the public list so that everyone knows what we're up to.
Some of this is just informational, so there's a global view as to what
we're doing.

Novell people, please focus on these things, so that we can avoid
dissipation.

The work is broken into two phases.  I'd like everyone to work on phase
I, and then when that's done, everyone starts to work on phase II.
People should not jump ahead from phase I to phase II.  As long as your
comrades in arms have phase I work left, so do you :-).  This obviously
does not apply to non-Novell employees ;-).

Phase I
=======

Beagle Core Infrastructure
--------------------------

        Team: Jon Trowbridge, Dave Camp and Joe Shaw
        
        Our mission is to make Beagle usable on a day-to-day basis.  The
        main things that we currently lack for a usable Beagle are:
        
        a) Not crashing or failing, most of the time.
                
                We need to be more fault tolerant and to present
                meaningful errors and debug logs to users in the event
                of failures, so that they can file reasonable bug
                reports.  Things like Raja's error messages in Best are
                helpful to users here too.
                
                Dave's recent log work is a good first step; we should
                make sure we deploy errors everywhere.
                
        b) Run all day.
                
                The daemon needs to be able to run for a long time
                without horking your system.  We need to figure out how
                to 'nice' the daemon's activities so that it doesn't
                take down the rest of the system.  We also need a
                mechanism for periodically restarting the daemon,
                because the Mono GC is not precise enough to allow us to
                run forever without eating all your RAM.
                
        c) Index mail.
                
                A large portion of people's information is in their
                mail, so we can't claim to be a usable search solution
                until we can search mail.  The primary goal will be
                Evolution mail integration; patches for other mail
                clients will, of course, be accepted.
                
                We need to be able to do a serviceable job of searching
                mail without requiring major changes in Evolution or
                building out of a separate patchset.  So, any patches
                required should be (i) built by the Beagle team, and
                (ii) included into Evolution 2.2 (feature freeze
                January).
                
                For disconnected IMAP users, this is relatively
                straightforward, since Evolution caches IMAP MIME parts
                in separate files on the disk; we can use inotify to
                notice new email getting cached here.  The downside, of
                course, is that we require users to cache their IMAP
                mail locally in order to search it.  This doesn't seem
                so bad to me, though, since I'd rather cache my mail
                locally than fire off network queries to my IMAP server
                every time I do a search in Beagle.
                
                For local mail users this is more difficult.  Ideally we
                would use Camel's index directly and write a Camel query
                driver, but it seems that Camel does not provide the
                locking we need.  So we're going to need to actually
                index local mail ourselves.  
                
                Indexing local mail is made hard by the fact that mbox
                files and ev-summaries can get very large.  So, this
                will be a challenge.
                
                I put this in Phase I because I think that searching
                mail is so important, it cannot be put off till later.
                This will mean we need to get patches into the Evo dev
                branch and Beagle users will have to run Evo snaps.
                
        These three items will be our focus in Beagle proper, until we
        can declare ourselves "reasonably usable."

Filters
-------

        Team: Veerapuram Varadhan
        
        The mission is to make sure Beagle can read the most important
        files off your disk.
        
        Jon and Varadhan wrote up a task list for this.  Essentially,
        the following filters are important and need work, in
        approximate priority order:  MS Office, PDF, PostScript,
        Monodoc, RTF, Texi, Man, ID3 tags, RPM.
        
        Varadhan will be making his way through this list as best he
        can.  Anyone else who wants to work on these is obviously
        welcome; filters are one of the most parallelizable pieces of
        work in Beagle, so everyone is encouraged to join.
        
Snapshot Packages
-----------------

        Team: Chris Lahey, Duncan Mak
        
        To keep up with the fast pace of change in the various parts of
        the architecture, we're going to need regular snapshot packages
        of the various bits of the Mono Desktop Extensions set.
        
        The following packages will be snapshotted many times a day by a
        cron job:
        
        dbus, evolution-sharp, libgalago, libgalago-gtk, galago-daemon,
        galago-sharp, galago-gtk-sharp, tomboy, f-spot, timeline,
        beagle, dashboard.
        
        [ Someone shout out if I forgot something important. ]
        
        Chris will build these for SUSE 9.0/9.1/9.2 and NLD.  If it's
        easy, maybe also for FC3.
        
        Let's put these somewhere easy to get at, maybe an open carpet
        server that replaces nat.org.  Chris and Jon to determine where
        these go (mdesnaps.org? iheartmde.org?)
        
        Evo snaps will come from the usual place (Rodney to chime in
        here).
        
inotify / kernel assistance
---------------------------

        Team: Robert Love
        
        Robert is going to finish his inotify work, so that beagle can
        find out when things change on the file system.  Robert's on top
        of this.
        
UI/Interaction Design
---------------------

        Team: Tuomas Kuosmanen
        
        Tuomas will be our go-to guy for search UI.  This encompasses:
        Best, the tray search tool, dashboard and the information
        browser.
        
        Tasks:
        
        a) Collect all mockups/designs.
        
                Garrett did a bunch of designs a few months ago.  We
                should put all of these on a web page so that they're
                accessible/viewable by everyone.  New mockups should go
                to the web page also.
                
                This should be done on beaglewiki.org, for easy
                editability.
                
        b) Tile design.
        
                We need updated tile designs, implemented in HTML, that
                include easy access to operations like sharing,
                printing, presence, etc.
                
        c) Tray UI design.
        
                We have the beginnings of a tray applet in CVS.
                However, the UI is awful.  We need a mockup of what this
                should look like.
                
        d) Best / information browser / Dashboard.
        
        
File Selector Integration
-------------------------

        Team: Federico Mena
        
        We need to be able to search from the file selector.  Federico
        is generalizing the filesel API to make this cleaner; he is also
        in charge of implementing filesel integration with Beagle.
        

Presence in People Tiles
------------------------

        Team: Robert Love
        
        People tiles should show presence, using Galago.  Christian
        Hammond has created nice Galago# bindings for us, including some
        widgets that we could embed into People tiles.
        
Tray UI
-------

        Team: Chris Lahey
        
        Implement the UI design that Tuomas creates.  Start with what we
        have now, and make it usable.
        
Web Page
--------

        Team: Garrett Lesage
        
        Alex set a new benchmark for open source project web site design
        with the tomboy website: 
        
        http://www.beatniksoftware.com/tomboy
        
        We should try to reach this level of cleanliness and information
        content for Beagle.
        
        Garrett should create a new web site for Beagle to replace the
        one we have.  Garrett, the address for the Beagle web site is
        www.gnome.org/projects/beagle; go ahead and just check in a new
        web site to GNOME CVS.
        
        Also, we need a T-shirt design.
        
Phase II
========

This stuff is a bit less concrete than Phase I, but we will be able to
flesh it out better when we declare Phase I done.

Metadata System
---------------

        Team: Jon Trowbridge, Robert Love, Joe Shaw

        We need a system for storing explicit relationships between
        things in your life.
        
        For example:
                
                "You downloaded /home/blah/foo.c from
                http://nat.org/stuff";
                
                "funny.jpg came from the mail 'Dude, check this out'
                from rml novell com"
                
                "Some parts of the file MyStrategy.doc were cut and
                pasted from RobertsStrategy.doc"
                
                "Objects a, b, c and d are part of the Beagle Team's
                team space."
        
        Which can be represented as subject/verb/object triplets:
        
                <file:///home/blah/foo.c, 
                downloaded-from,
                http://nat.org/stuff>
                
                <file:///full/path/funny.jpg,
                attachment-from,
                mail:///local/folder/uid?9238u4982374>
                
                <file:///home/nat/MyStrategy.doc,
                cut-and-paste,
                ifolder:///NatAndRobertsSharedArea/RobertsStrategy.doc>
                
                <a,explicit-relationship,team:///BeagleTeam>
                
        Note that all of these are explicit relationships; the implicit
        relationships, like "this web page is similar to this document"
        are discoverable, and don't need to be stored.
        
        Also note that we need our metadata system to cache some handy
        relationships for us that might be discoverable.  For example:
        
                <http://nat.org/,
                created-by,
                person://NatFriedman>
                
                <mail:///local/folder/uid?9238u4982374,
                sender,
                mailto:rml novell com>
        
        This will make these relationships easy to search.
        
        We need a simple metadata store for explicit relationships and
        cached implicit relationships in Beagle.  This should have a
        nice API on it for storing stuff.
        
        Once we have this, we can start instrumenting applications and
        making them store metadata in Beagle.  We can also do things
        like create 'watchers' which store metadata as you use your
        system; imagine if timeline were pimped out to store
        relationsihps between files, mails or web pages you view at the
        same time.
        
        Various thoughts:
        
                * The implementation should focus on getting something
                to work soon, not solving all the world's problems.
                
                * The backing store will be Lucene; perhaps EAs will be
                involved in Linux on the filesystem.
                
                * We need a simple API that can be used from C to store
                metadata.  
                
                * We should have a simple RDF-like interface for
                metadata query results.
                
                * We need a documented universal URI scheme.
                
                * Team definitions should live in the addressbook.
                
                * The implementation should focus on getting something
                to work soon, not solving all the world's problems. :-)
                
        Once we have this thing, and the team notion, we can build the
        information browser, which will really be fun.

Nautilus Search Integration
---------------------------

        Team: Dave Camp
        
        We need a simple file search interface in Nautilus.  This should
        include emblem searching.


Dashboard
---------

        Team: Joe Shaw
        
        Dashboard needs to use Beagle tiles, needs cluechaining for
        addressbook contacts, needs to track focus properly, needs an
        a11y-based observer, and needs specific frontend instrumentation
        for Evolution, OpenOffice, Firefox.
        
Information Browser
-------------------

        Team: Jon Trowbridge
        
        This is Jon's baby, to be explained through mockups, etc, as we
        go.
        
Tiles
-----

        Team: Federico Mena
        
        We need drag-and-droppable tiles, a better tile widget system,
        etc.

Monodoc Integration
-------------------

        Team: Veerapuram Varadhan
        
        Monodoc should use Beagle to do a full keyword search of its
        contents.