Re: Finding and Reminding, tech issues, 3.0 and beyond

[Owen Taylor <otaylor redhat com>]

(Replying selectively - lots of stuff snipped that I agree with)

On Tue, 2010-04-13 at 21:18 -0500, Federico Mena Quintero wrote:
> On Fri, 2010-04-09 at 18:09 -0400, Owen Taylor wrote:
> > I've attempted below to extract out some of the technical bits from
> > http://live.gnome.org/GnomeShell/Design/Whiteboards/FindingAndReminding
> 
> This is great stuff.  I feel kind of bad commenting from the sidelines,
> given that I have done practically no work on gnome-activity-journal or
> the Zeitgeist engine, but anyway, here goes.
> 
> Basically, the FindingAndReminding page has come to conclusions that are
> very similar to what gnome-activity-journal tries to be.  In this mail I
> want to convince you that g-a-j plus Zeitgeist are the right way to go
> for GNOME, and that reimplementing them would be a waste of resources.

[...]
  
> > "User defined tags"
> > 
> >   A completely flat view of all documents doesn't handle all users
> >   or use cases. "Frequent filers" will want to be able to identify
> >   projects and other subsets of files.
> 
> Yeah, we need to show tags as something more than "this file has such
> and such tags" and "this is a search query for tags".
> 
> When I was thinking about the Journal two GUADECs ago, I imagined that
> you could be able to "circle" a bunch of files and tag them visually.
> Imagine taking a red pencil, circling a few events in the journal, and
> giving them a purple tag that says "Research Paper on Monkeys".  The
> purple tag appears somewhere prominent ("current projects"?  "recent
> tags"?) so you can access those items again easily without scrolling
> back in the timeline.
> 
> Firefox's "recent tags" command is incredibly useful- this is more or
> less the same.

One of the things on the Wiki view that I didn't comment on here -
because I didn't have much of an idea of how we would do it - was
 
 "What if similar or related items were automatically stacked 
  together so they don't clutter the Desktop?"

In some cases, we should be able to figure out relatedness without the
user being involved - if I view a bunch of images in a directory in a
row, we may want what we show the directory rather than the images as
the toplevel history item.

But at other times, we need user input, and that user input is usually
after the fact - asking users to come up with tags as they save
documents isn't enough.

[ Though I think what would be very cool from a file save dialog with 
  tagging support is that if you entered a new tag, it gave you the 
  ability also to add that tag to some of your recent files. ]

Lots of UI work to figure this part out, hopefully largely independent
of the backend, except for the low-level details of tag storage.
  
> > "Timeline view of files"
> 
> Clearly we agree on this :)
> 
> One thing I like about the mockup in the FindingAndReminding page is
> that the items are laid out in a grid.  Imagine this:

>   [       ]
>   [ thumb ] File with a long descriptive name.odt
>   [       ]
> 
>   [thumb] [thumb] [thumb] [...]   (imported 234 photos)
> 
>   [       ]
>   [ thumb ] Another file with a meaningful name.pdf
>   [       ]
> 
> I.e. an irregular grid.  You don't want to show meaningless names like
> "dsc02345.jpg" for photos, but you do want document titles or
> descriptive filenames when they are available.  You may want bigger
> thumbnails of PDFs than of photos so that they are easier to recognize.

Sounds neat. And hard.

[...]

> > "Adding non-files to Desktop"
> 
> Good.  Zeitgeist supports this, and is one of the more interesting
> features.
> 
> I'd add these to the timeline:
> 
> - Web pages, or those that you bookmark.
> - Mails that I sent.
> - Mails that I received that have attachments.
> - Attachments that I saved.
> - IM conversations that I had.
> - Handwritten notes ("Tomboy in the journal")
> - Files or items that I can drop into future dates.
> - Git pushes.
> - With help from Greasemonkey or whatever, google docs that I edited.

I think this is something a little different than what I was talking
about. The Desktop is the set of things that is immediately available -
current and future items - and is separate from the timeline view of
past items.

Some of the items you list above (web page bookmarks, notes, emails you
received, etc), do make sense on the desktop, but I don't think you'd
want, to say, drag a Git push to the desktop.

[...]

> >    The only think I can think of in the current mockups
> >    that requires a Zeitgeist-like approach is the
> >    "Frequent" selector. Without a longitudinal view
> >    of usage, it's hard to answer "what are the most frequently 
> >    used documents in the last 30 days".
> 
> One really interesting aspect of Zeitgeist is the data-mining algorithms
> it uses to present "items that are frequently used together" - the
> Apriori algorithm and such.  I wouldn't want to lose this.  One thing
> that would make the timeline really useful would be to split the screen
> in two, with the timeline on one side and the related items in the other
> side.

I'm going to be up-front and express some skepticism about this area. If
it worked well, it would be great. If it worked badly, then it would be
odd and confusing to the user.

I don't think it's been demonstrated yet that it works, and for 3.0 I'd
like to stick to things that we're pretty sure will work... we already
have enough variables to juggle here.

This does relate to how to go from finding a file to a "nearby" file -
see my response to Johannes:

http://mail.gnome.org/archives/desktop-devel-list/2010-April/msg00089.html

But for that I think you typically want something more deterministic -
what files did I actually work with at the same time as this file.

> >  * To a much greater extent than tracker, Zeitgeist is
> >    is designed to require applications to be modified to
> >    push events to it.
> 
> Yes and no.  Apps that create files should already be storing that info
> in ~/.recently-used, which Zeitgeist feeds from.
> 
> The Zeitgeist API is mostly for when you want to log non-file items.
> Mails that you got with attachments, web bookmarks, etc.

Ah. My understanding was that Zeitgeist really wanted to know when files
were closed, not just opened, something not available from piggybacking
from .recently-used.xbel. But I guess that is most important for
"relatedness" in any case.

[...]

   
> >  * Zeitgeist is designed to be standalone and independent
> >    from Tracker, but also used in conjunction. This, at
> >    times, makes things not as good as they could be. For
> >    example, Tracker has a pretty sophisticated system to
> >    assign a UID to each file and track files as they
> >    move around the file system, but Zeitgeist, which
> >    identifies file by file paths will lose a file as
> >    soon as it is moved - it doesn't piggyback off the
> >    work that Tracker is doing.
> 
> Hmm, couldn't Zeitgeist be modified to support this?  Although the
> Zeitgeist team has worked hard on finalizing the database format and
> APIs, nothing released "in the wild" yet, so changes are feasible. 

We don't really want Zeitgeist duplicating the work that Tracker is
doing. My understanding is that Tracker has some ungodly combination of
gio hooks and inotify to detect renames - one process doing that is
likely enough.

Zeitgeist could piggyback off the work Tracker is doing - either by
writing out its log in terms of Tracker URN's, or by listening to
Tracker change notification. (Writing the log in terms of Tracker URN's
is simpler and more reliable, but probably prohibitively expensive for
queries if the log isn't actually written into the Tracker store.)

Another thing that Tracker handles is detecting when files are deleted -
this is actually a big problem for gnome-shell currently. We don't want
to show files in the history that no longer exist, but it's a big
performance hit to stat all the files in the history when starting up,
and also a big performance problem if we added inotify watches on all of
them.

[...]

> Now, about the graphical part...
> 
> Gnome-activity-journal has substantial work put into it.  The user
> interface is not perfect, of course, but it's there and it works.
> 
> Gnome-shell is going to have to accomodate the journal somehow.
> Rewriting a timeline interface and all the associated widgetry, by hand,
> using gnome-shell's not-really-a-widget-system approach, sounds painful.
> However, gnome-shell already includes a window manager - couldn't it run
> gnome-activity-journal as a separate process and just manage its window
> in a special way?  (This is the kind of extensibility in the window
> manager that I was advocating two GUADECs ago - remember the tabs that
> pull out of the edges of the screen?)

Certainly embedding the pygtk application is possible to an extent. But
you can't really just run a normal GTK+ application inside the overview
context: the appearance will be wrong, the user interface ideas won't
exactly match, and it will get very odd if the application opens menus
or dialogs.

> One idea that floated around was the following.  Gnome-shell has a big
> "Activities" button that shows you all the stuff that you have open.  It
> could very well have a "Journal" button next to that one, that would
> show you the journal in a large part of the screen --- it is All Your
> Work, so it deserves sufficient screen real estate.

The basic idea of the Activities Overview is that it's not a destination
of its own, but it's a "one stop shopping" place you go to get from one
task to another task.

That means that it's very fast to get to - with the hot corner or logo
key. That the most likely things you want to get to are all one click
once you get there - both switching to existing tasks and opening new
tasks. That if you start typing you search across all the different
categories of destination. 

So for that reason, we've been hesitant to start piling up additional
buttons next to the Activities button. Whether a Journal button or a
People button, or...

Also, once we move from Activities Overview and Main View to more
possibilities, it starts getting less clear why your contacts list (say)
is accessed by button at the top of the screen, but your web browser is
an application. We've added another level of nesting to the interface.

> The summary of all of this is:
> 
> - Don't reinvent Zeitgeist.  Make it better; solve the issues with
> Tracker.
> 
> - Don't reinvent the journal.  Make it work with gnome-shell.

I don't think there is any question of reinventing Zeitgeist. We
probably could get pretty far without having event storage, and just
going off of file timestamps, but the ideal thing is that Zeitgeist is
there, integrates well with Tracker, and we can get all the information
we want simply and efficiently.

> - Thanks for the detailed analysis of the literature.

All the credit for that goes to Jon McCann. I'm just picking it up at
the "how do we implement it" level.

- Owen