Beagle 0.0.12

[Jon Trowbridge <trow novell com>]

I'm pleased to announce the release of Beagle 0.0.12.

This version of Beagle contains a number of exciting new features, including:
* libbeagle, a C library for querying the beagle daemon.
* Tools for building stand-alone, static indexes that can be searched
  by the beagle daemon.
* Many improvements to the configuration system.
* A much-needed revamp of the filtering system that (among other things)
  has allowed up to starting indexing e-mail attachments.

We have also continued to focus on fixing bugs.  This is the most stable
version of Beagle yet, and should be suitable for everyday use.

Please note that Beagle 0.0.12 uses inotify 0.23 when available.  Earlier
versions of inotify will fail in mysterious ways, and will generally
sap your will to live.

Also note that as of this version, Beagle requires Gmime 2.1.15.

For the best possible experience, you should run Beagle with Mono 1.1.8.2 or
better.


OUR MANY URLS
-------------

To download the 0.0.12 tarball or learn more, visit the Beagle wiki at:
http://www.beagle-project.org

Joe Gasiorek writes a Beagle newsletter.  You can read it at:
http://www.beagle-project.org/Newsletter

The latest gossip is available at:
http://www.planetbeagle.org

Nat Friedman made some cool movies that demonstrate Beagle in action:
http://nat.org/demos

We still talk about Beagle on the dashboard-hackers mailing list:
http://mail.gnome.org/mailman/listinfo/dashboard-hackers

Fireflies are not flies.  They are beetles:
http://iris.biosci.ohio-state.edu/projects/FFiles


WHAT IS BEAGLE?
---------------
 
Beagle is a tool for indexing and searching your data.  It is in an early
stage of development and should be considered experimental.  Beagle is
improving rapidly on many fronts, and should work well enough for
everyday use.
 
The Beagle daemon transparently monitors your data and updates the index
to reflect any changes.  On an inotify-enabled system, these updates happen
more-or-less in real time.  So for example,
 
* Files are immediately indexed when they are created, are re-indexed
  when they are modified, and are dropped from the index upon
  deletion.
* E-mails are indexed upon arrival.
* IM conversations are indexed as you chat, a line at a time.
 
Beagle uses the Lucene indexing system from the prodigious Doug
Cutting.

Best is a graphical tool for searching the index that the daemon creates.
Best doesn't query the index directly; it passes the search terms to the
daemon and the daemon sends any matches back to Best.  Best then renders the
results and allows you to perform useful actions on the matching objects.

Indexing your data requires a fair amount of computing power, but the Beagle
daemon tries to be as unobtrusive as possible.  It contains a scheduler that
works to prioritize tasks and control CPU usage, based on whether or not
you are actively using your workstation.


DEPENDENCY HECK
---------------

Beagle has many dependencies, and thus can be difficult to compile.
It requires:
* The full Mono stack (We all use 1.1.x, and you probably
  should too, but 1.0.6 will also work.  1.0.5 and earlier will NOT work.)
* gtk-sharp 1.0.8 (1.9.x will NOT work)
* Gecko-sharp 0.6
* Gmime 2.1.15
* Libexif 0.5 or better

For the best possible Beagle experience, you should also have:
* Mono 1.1.8 or better
* Evolution-sharp 0.6
* wv 1.0.3
* An inotify 0.23-enabled kernel
* A tall, cool glass of lemonade


CHANGES SINCE 0.0.11.1
----------------------

Daemon/Infrastructure:
* Add the libbeagle C API (Imendio, Joe Shaw)
* Big changes to the filtering code to make it more flexible (Joe, Fredrik
  Hedberg)
* Match multiple filters per indexable, use the highest ranking
  match (Fredrik)
* Match filters with a combination of Uri, Extension and/or
  MimeType (Fredrik)
* Allow FilterFactory to fall throught the filter list in the case of
  an error (Fredrik)
* Introducted events / callbacks when indexables are filtered (Fredrik)
* Mark files in the file system backend with the Filter used (Fredrik)
* Re-index files when a newer filter is available (Fredrik)
* Don't instantiate a new LuceneDriver for every remote indexing
  request (Fredrik)
* Allow us to disable Lucene locking per directory (Fredrik)
* Handle more failure conditions in the message passing code (Joe, Jon
  Trowbridge)
* Add support for running without extended attributes, and speed up the
  sqlite fallback by using transactions (Joe, Fredrik)
* Use UTC timestamps in Lucene so that we're not boxing integers
  unnecessarily (Joe)
* Add throttling to the scheduler when many immediate priority tasks are
  added, like when untarring a tarball or running rm -r on a directory
  (Joe)
* Set the default QueryDomain to both Local and System, so that per-user
  and the new systemwide indexes are both searched by default (Joe)
* Fix a bug whereby updates would cause hits to show up multiple times
  by issuing a subtract event before the add event (Joe, Daniel Drake)
* Friendlier inotify error messages (Daniel)
* libbeagle: Try BEAGLE_HOME and check that the bus socket exists (Daniel)
* Don't try and discover the file mime-type twice (Daniel)
* Small fixes to libbeagle (Lukas Lipka)
* Better built-in heap-buddy support (Jon)
* Avoid unnecessary boxing in the message passing code (Jon)
* Sqlite FileAttributeStore clean-ups (Jon)
* Added the ability to change an Inotify.Watch's event mask (Jon)

Backends:
* Indexing of mail attachments (Joe)
* Add a mail filter, giving us Maildir support, and port the Evolution
  mail backend to use it (Joe)
* Use IndexableGenerators for the Blam and Liferea backends so that it
  scales and performs better (Joe)
* Configuration option to ignore files by path, pattern etc. (Fredrik)
* Configuration option to ignore mail folders (Fredrik)
* Added StaticQueryable for searched canned indexes (Fredrik)
* Disabled the LauncherQueryable in favor of the system indexing
  features (Fredrik)
* Fix the Evolution mail backend to not reschedule a reindex if it's
  already in the middle of indexing itself, which would otherwise
  interrupt it (Joe)
* Fix a bug in the addressbook backend with older (pre-2.2) versions of
  Evolution in which contacts wouldn't be indexed after the addressbook
  index was first created (Joe)
* Fix handling of gaim logs with non-ascii characters represented in
  names (Daniel)
* Update the properties in some backends to the be correct (Lukas)
* Explicitly disable filtering on gaim log indexables, since the stream
  we produce is already filtered (Jon)
* To ensure that beagle-index-info returns correct information, get
  the item count from the IIndexer, not the driver we are using for
  querying (Jon)
* Fix a problem where we were losing mails from the index if an IMAP
  summary was updated later (Joe)

Filters:
* Added .desktop filter (Fredrik)
* Added .directory filter (Fredrik)
* Added Monodoc filter (Fredrik)
* Added Docbook filter (Fredrik)
* Filter text/x-chdr (C header files) using the C source code filter (Joe)
* Image filter abstraction. Created a FilterImage which all the image
  filters use. This lets us index F-Spot metadata information (Lukas)
* Added F-Spot metadata support which we now index (Lukas)

UI/Tools:
* Added beagle-build-index for building static indexes (Fredrik)
* Added beagle-manage-index for low-level Lucene index management (Fredrik)
* Configuration UI (Fredrik)
* Fix a problem with Best whereby the window wasn't being placed in the
  same place when F12 was pressed (Joe)
* Add a "Reply" action back to the mail tile, as it's available in Evo
  2.2 (Joe)
* Don't show "stop words" in the snippet highlights (Joe)
* Fix ImLogViewer to work with gaim's character escaping style (Daniel)
* Remember selected log in the IM timeline tree, improve searching,
  other random optimizations to the IM viewer (Lukas)
* Galago support and integration. Added Galago support to the Mail, IM
  log and Contact tile (Lukas)
* Fixed a crasher in TileMailMessage.cs and some random cleanups.  Fix a
  bug when the tile didnt show the subject when 'Who' was null. This was
  related to the email attachments (Lukas)
* Added an author field to the blog template (Lukas)
* Render mail attachments nicely (Lukas)

Web Services:
* Support for cascaded Beagle Networking (KN Vijay)
* Configuration of Beagle Web-services and Beagle networking via
  'beagle-config', with corresponding handlers for real-time update of
  configuration changes (Vijay)
* Added on-demand Prefetching and Caching of snippets for network hits
  (Vijay)
* Updates to Web interface wrt handling Actions for external accesses
  (Vijay)

Everything Else:
* Support either sqlite2 or sqlite3 (Daniel)
* Build fixes (Jon)
* Newsletter (Joe Gasiorek)
* Added Polish translation (Michal Kastelik)
* Added Turkish translation (Enver Altin)
* Updated Canadian English translation (Adam Weinberger)
* Updated Chinese translation (Chao-Hsiung Liao)
* Updated German translation (Hendrik Brandt)
* Updated Hungarian translation (Gabor Kelemen)
* Updated Norwegian translation (Terance Sola)
* Updated Spanish translation (Francisco Javier F. Serrador)
* Updated Swedish translation (Fredrik)
* All the stuff I forgot (All of the people I forgot)


KNOWN ISSUES
------------
 
It doesn't take that much ingenuity to confuse the file system backend.  In
particular, the right thing doesn't always happen if a file's name changes
very rapidly.  (i.e. "mv foo bar; mv bar baz; mv baz foo")

The amount of memory consumed by the beagle daemon tends to grows over time.
All of the major leaks have been fixed, but it might need to be manually
killed and restarted after running for long periods of time.

Due to a bug in mono, the beagle daemon will occasionally lock up when
starting the helper process.  As far as we can tell, this is pretty rare and
only happens on SMP systems.

There are a few other relatively rare deadlocks that also seem to be mono
bugs.  Occasionally beagled will deadlock on start-up.  Sometimes a query will
throw an exception and cause the beagled to deadlock until the querying
process is killed.  These are hard to trigger, but have been observed on
non-SMP systems.

On non-inotify systems, sometimes beagle will get confused and repeatedly
re-index the same directory.

At this point in development, we cannot commit to stable APIs or file formats.
You will almost certainly need to delete your indexes and start again at some
point in the future.