[Tracker] [Fwd: {ANNOUNCE] Tracker 0.0.4 "Now indexing at Warp speed"]

[Jamie McCracken <jamiemcc blueyonder co uk>]

The next vastly improved version of the Tracker Search engine, Indexer
framework and first class object/metadata/tag database is now available at:

http://www.gnome.org/~jamiemcc/tracker/

Tarball : http://www.gnome.org/~jamiemcc/tracker/tracker-0.0.4.tar.gz
Deb : http://www.gnome.org/~jamiemcc/tracker/DEB/tracker_0.0.4_i386.deb
RPM (FC5): http://www.gnome.org/~jamiemcc/tracker/RPM/

All discussion related to tracker happens on the tracker mailing list
(http://mail.gnome.org/mailman/listinfo/tracker-list)


INTRODUCTION

Tracker is a powerful desktop-neutral first class object database,
tag/metadata database, search tool and indexer.

Tracker is also extremely fast and super efficient with your systems
memory when compared with some other competing frameworks and is by far
the fastest and most memory efficient Nautilus search and Deskbar
backends currently availble.

It consists of a common object database that allows entities to have an
almost infinte number of properties, metadata (both embedded/harvested
as well as user definable), a comprehensive database of keywords/tags
and links to other entities.

It provides additional features for file based objects including context
linking and audit trails for a file object.

It has the ability to index, store, harvest metadata. retrieve and
search all types of files and other first class objects.

First class object support includes : Files, Documents, Music, Images,
Videos, Playlists*, Notes*, Applications*, Contacts*, Emails*,
Conversations*, Appointments*, Tasks*, Bookmarks*, History* and Projects*

(* these services are not currently indexed at the moment but will be in
later versions)


More infomation on Tracker can also be found at
http://freedesktop.org/wiki/Software/Tracker


USE CASES

Tracker is the most powerful open source metadata database and indexer
framework currently available and because it is built around a
combination indexer and sql database and not a dedicated indexer it has
much more powerful use cases:

* Provide search and indexing facilities similiar to those on other systems

* Common database storage for all first class objects (EG a common
music/photo/contacts/email/bookmarks/history database) complete with
additional metadata and tags/keywords

* Comprehensive one stop solution for all applications needing an object
database, powerful search (via RDF Query), first class methods, related
metadata and user definable metadata/tags

* Can provide a full semantic desktop with metadata everywhere

* Can provide powerful criteria based searching suitable for creating
smart file dialogs and vfolder systems

* Can provide a more intelligent desktop using statistical metadata



NEW CHANGES

* Hugely optimised indexing when many files are waiting to be indexed
(especailly when you first run trackerd)

* Mass queueing of files no longer blocks the main thread meaning super
fast searches can still be performed during heavy indexing

* Eliminated cpu bottlenecks and improved thread synchronisations so
Tracker now hits the ground running when indexing (approx 500+ files
indexed per minute on inotify enabled systems)

* Extended metadata support for more Exif fields

* Added more service types

* Imporved build and support for FC5 (includes RPMs)

* Redesigned Database around version 5 of the *embedded* in-process
mysql database library

* Now uses the auto repair facility provided by mysql to automatically
repair damaged database files so you need never worry about losing your
precious data.

* Moved virtually all DB logic into stored procedures which provides a
clean seperation of DB logic and application logic

* Added support for parsing dates in various formats including
conversion to/from ISO 8601 format

* Fixed MsWord filter to prevent looping (WvText causes inotify to
report file write change causing endless looping)

* Redesigned DB structure to be more generic and service orientated

* Added support for service types (first class objects) to DB

* Cleaned up code warnings and fixed potential crasher (thanks to patch
from Nate Nielsen)

* Fixed issues with argv handling (thanks to patch from  Dan Nicolaescu)

* Fixed build issue for Fedora Core 5 (thanks to patch from  Dan Nicolaescu)



FEATURES

* Desktop-neutral design (its a freedesktop product built around other
freedesktop technologies like DBus and XDGMime but contains no GNOME
specific dependencies)

* Very memory efficient and non-leaking (typical RAM usage 4 - 6 MB).
Unlike some other indexers, tracker is designed and built to run well on
lower memory systems with typically 128MB or 256MB memory. It should
even be efficient enough to use on some mobile devices.

* Non-bloated and written in C for maximum efficiency.

* Small size and minimal dependencies makes it easy to bundle into
various distro's including live cds.

* Fast indexing and unobtrusive - no need to index stuff overnight.
Tracker runs at nice+10 so it should have a minimal impact on your system.

* Implements the freedesktop specification for metadata
http://freedesktop.org/wiki/Standards/shared-filemetadata-spec

* Extracts embedded File, Image, Document and Audio type metadata from
files.

* Extracts embedded metadata from HTML, PDF, PS, OLE2 (DOC, XLS, PPT),
OpenOffice (sxw), StarOffice (sdw), DVI, MAN, MP3 (ID3v1 and ID3v2),
OGG, WAV, EXIV2, JPEG, GIF, PNG, TIFF, DEB, RPM, TAR(.GZ), ZIP, ELF,
REAL, RIFF (AVI), MPEG, QT and ASF files

* Supports the WC3's RDF Query syntax for querying metadata

* Provides support for both free text search (like Beagle/Google) as
well as structured searches using RDF Query

* Respond in real time to file system changes to keep its metadata
database up to date and in synch

* Fully extendable with custom metadata - you can store, retrieve,
register and search via RDF Query all your own custom metadata

* Can extract a file's contents as plain text and index them

* Provides text filters for PDF, MS Office, OpenOffice (all versions),
HTML and PS files.

* Can provide thumbnailing on the fly



INSTALLATION (from source):


The following build dependency is needed:

The embedded mysql library ibmysqld.a version 5.0.19 or higher is
required complete with corresponding header files and the mysql_config
program. The libmysqld.a library is always statically linked so it is
not a runtime dependency. It can be found in Debian/Ubuntu package
libmysqlclient15-dev. Other distros should check their mysql client libs
to see if it is present.

If not present you can compile libmysqld.a by downloading source tarball
for an appropriate version (5.0.19+) from the bottom of page
http://dev.mysql.com/downloads/mysql/5.0.html

The following configure flags are reccommended if building mysqld.a from
source: --without-server --with-embedded-server --enable-assembler
--with-mysqld-ldflags --with-client-ldflags


Run time dependencies (also needed for build) :

libdbus (0.50 +)

dbus-glib bndings (0.50 +)

glib (2.6+)

zlib

libvorbisfile3 (1.1+)


Additional recommended packages:

libextractor (0.5.9+) (tracker has a streamlined version inlined)
wv (1.0.2+)
poppler (pdftotext)



COMPILATION

To compile Tracker uses the following commands :

./configure --prefix=/usr --enable-static --with-pic
CFLAGS=-I/usr/include/exiv2
make
sudo make install

If you install using any other prefix then you might have problems with
files not being installed correctly.

(You may need to copy and amend the dbus service file to the correct
directory and/or might need to update ld_conf if you install into non
standard directories.)




Compile Options

Tracker has several compiler options to enable/disable certain features.
The following is available (all options should be passed as command line
parameters to autogen.sh EG ./autogen.sh --disable-fam)

--disable-fam : this option omits support of FAM/GAMIN with tracker

--disable-inotify : this option omits support for inotify with tracker

--enable-libextractor  : this option forces the use of the prepackaged
streamlined version of libextractor in tracker



RUNNING TRACKER


To run tracker, you need to manually start the tracker daemon trackerd.
By default trackerd will index your entire home directory.

You can also pass a directory root to be indexed as a command line
parameter if you dont want your entire home directory indexed. EG

"trackerd /home/jamie/Documents"

You can make sure that tracker only indexes a subset of your home
directory and also specify folders not in your home directory by editing
the tracker.cfg file in ~/.Tracker (which is created when you first run
trackerd) and setting WatchDirectoryRoots to a semicolon delimited list
of directories (full path required!)

EG:

WatchDirectoryRoots=directory1;directory2;directory3

On the first run, Tracker will automatically create a new database and
start populating it with metadata by browsing through the user's home
directory (or the root folder(s) specified).

On subsequent runs, Tracker will start up much much faster and will only
ever incrementally index files (IE files that have changed since last
index).

If installed correctly, the tracker daemon (trackerd) can also be
started automatically via Dbus activation.


Tracker And Nautilus Search

Once you have installed Tracker and have some indexed contents, you
should now compile Nautilus (ver 2.13.4 or higher) which should auto
detect that tracker is installed and automatically compile in tracker
support. You are now ready to appreciate a powerful and super efficient
c based indexer in all its glory... happy hunting!

To make sure trackerd always start when you login to Gnome, you will
need to add it to Gnome-session (select sessions from preferences menu,
select startup program tab and then add /usr/bin/trackerd). For
non-gnome installations, see the desktop docs for how to auto start an
application for your particular desktop.

Tracker and Deskbar applet

Tracker is also integrated in GNOME's deskbar applet. Please see that
applet for more info.



COMMAND LINE TOOLS

Tracker comes with a number of command line apps that you can use:

"tracker-extract FILE" - this extracts embedded metadata from FILE and
prints to stdout

"tracker-search SEARCHTERM" - this perfoms a google like search using
SEARCHTERM to retrieve all matching files where SEARCHTERM appears in
any searchable metadata

"tracker-query" - this reads from STDIN an RDF Query that specifies the
search criteria for various fields. It prints to STDOUT all matching
files. You can see some example queries in the RDF-Query-examples
folder. You can run the examples as "tracker-query < RDFFILE"


--
Mr Jamie McCracken
http://jamiemcc.livejournal.com/


--
Mr Jamie McCracken
http://jamiemcc.livejournal.com/