Re: Online Desktop, Tomboy, and user storage

[Havoc Pennington <hp redhat com>]

Hi,

John Stowers wrote:
> I agree with Sandys asessment on the need for this. When I looked into
> writing some kind of .Mac equivilent for Conduit to use as a central
> store to sync stuff with I inevitably came back to the same point, why
> write my own backend anything when things like WebDav and gnomevfs
> support for it already exist.
>
> According to http://live.gnome.org/OnlineDesktop we will basically
> store account info for people and the services they use. I dont think
> reinventing the wheel and writing a server side backend to store
> generic use data is all that effective use of time (putting aside
> concerns like bandwidth, storage space and security for the moment).
> Why not use apaches WebDav support and have it authenticate against
> online.gnome.org.

To be clear, I don't see the possibilities here as _only_ syncing my 
private stuff between systems. The pure sync is already mostly working 
in Tomboy afaik, though you do have to set up your own DAV server so for 
now it's a "highly motivated technologists only" sort of feature.

To see what else we could do, Sandy blogged some mockups for example:
   http://automorphic.blogspot.com/2007/08/tomboy-online-mockup.html

Ideally I can "publish" a note to the world, share it only with some 
people, browse/search other people's public notes, and access my own 
private notes from a browser.

For the stuff beyond sync, DAV is not the major part of the solution. It 
gives you one mechanism you need, namely:

 GET - get the XML file
 PUT - put the XML file

that's WebDAV. (OK, I'm exaggerating *slightly* - you can also list 
directories.)

So we need to get and put the XML files, but we also need to do a lot of 
other things.

First, just to get WebDAV working smoothly and without clunkily 
configuring an URL, username, and password in every app that uses it, we 
have to solve some issues.

WebDAV autoconfig and auth issues:

 - gnome-vfs, let alone FUSE, do not have your browser cookies.
   This means even though you're logged on to online.gnome.org,
   you are not logged on to dav://online.gnome.org/dav

 - last night I tried adding HTTP auth to the webdav support
   that the server on online.gnome.org already has. Then you
   can put your username (which is an ugly GUID) and a password
   in the config dialog for Tomboy webdav.

   It still doesn't work for me for reasons I haven't debugged yet.

   There's at least one limitation with http
   auth vs. cookies: you can't have anything which *optionally* involves
   login.
   That is, clients only send the http auth credentials after
   they get bounced with 401 UNAUTHORIZED.

   So for example, say you want to make a file or directory public, you
   can't also make it look different to different users.

   As an example where this matters for DAV, say you want to allow
   marking notes as shared with certain people but not others, or some
   notes public and others not; you can't mix public notes into
   a directory with any private or limited-audience notes, since the
   owner or intended audience won't get 401 so won't be logged in.

   Cookies are better in this respect, but we have to get the cookies
   into gnome-vfs, or horrors, FUSE.

   This may not really matter for Tomboy, private-only DAV might be
   fine, and any public note sharing would be via regular http/web-ui.

 - it should be possible to globally say "put my stuff on
   online.gnome.org" or "put my stuff on myserver.mydomain" instead of
   configuring each app and logging in separately for each app.

Anyway, these config and auth issues need solving even for plain sync to 
work "out of the box" and nicely.

Second, to get the cooler features beyond sync, we have to go beyond 
getting DAV working smoothly.

 - the web UI. I think we should be able to have a Publish button
   in Tomboy to make a note public for sharing, and also
   you should be able to browse your own notes online.

   again,
    http://automorphic.blogspot.com/2007/08/tomboy-online-mockup.html

   This involves the server understanding how webdav will be used,
   at minimum. If you look at the Tomboy sync directory layout that
   Sandy posted, we would not want the web UI to be listing that
   cruft as a raw directory listing. It isn't a "human readable"
   directory layout.

   In your homedir, a directory like this would be a dotfile, is
   one way to think of it, to keep it hidden.

   This means at minimum a "dedicated" DAV share for Tomboy,
   rather than having just a generic WebDAV server that a variety of
   different apps use, some for "dotfile" type stuff and some for
   human-readable stuff. The DAV share for Tomboy should be dedicated
   so it isn't mixed with your human-readable DAV share, and so the
   server can understand its format and list/display Tomboy notes.

   I think I could refactor the current DAV support to have such a
   dedicated DAV share, like tomboy-dav/users/GUID

   Then, the server has to be able to interpret the stuff on the
   special Tomboy DAV share, and do something with it.

 - Given a dedicated DAV share with a format the server understands,
   there is really no difference between current DAV approach:

     GET online.gnome.org/tomboy-dav/manifest.xml
         (gets per-note revision numbers)
     GET online.gnome.org/tomboy-dav/<revision>/<note name>
     PUT online.gnome.org/tomboy-dav/<revision>/<note name>

   and a typical http API like:

     GET online.gnome.org/tomboy_manifest
           (returns notes and revisions)
     GET online.gnome.org/tomboy_note?name=Foo
     PUT online.gnome.org/tomboy_note?name=Bar&revision=N
           (upload XML data)

   With the DAV approach we can make POSIX filesystem calls that
   eventually become the GETs and PUTs via FUSE, but the server is not
   looking at this as a filesystem; to do anything beyond sync,
   the server must understand what's _on_ the filesystem enough to
   do the http API as in the second case.

   Conceptually, the server has to be able to map that http API onto the
   DAV file tree and vice versa.

   The point isn't that we shouldn't use DAV; it doesn't matter. The
   point is that the server can't be "just a DAV server," it has to
   understand the data, if we get beyond sync. DAV is an
   implementation detail of a custom Tomboy server, as soon as we do
   more than sync.

   Thus, we can't use just the Apache DAV. Fortunately, I already coded
   a simple DAV implementation months ago.

 - collaboration features add another angle where we might want
   live notification - "here are the people looking at this note"
   type of stuff. If we got fancy and did that, we might want to
   map some of the Tomboy state onto the XMPP protocol we have.

   I would envision that the available Tomboy note names and their
   revisions are in the Online Desktop Engine data model, and if
   the revision changes Tomboy would know to http/dav GET a new version.
   We would then also have a property of the note in the data model
   which was people looking at it or whatever.

All this said I have to figure out where to start hacking first ;-) so I 
was thinking of doing the custom Tomboy store on the server, plus a 
public/private flag for each note stored in the database, and a simple 
web page that lists your notes (filtering out private ones if someone 
other than you views it).

Then the first API to the custom Tomboy store on the server could be 
"upload note with a given name," and would be used by a Tomboy add-in 
that adds a Publish menu item and just uploads the note. (This isn't 
sync - it just shoves a single note on the server, and never downloads 
notes.)

The second server API could be a dedicated DAV root that implements the 
Tomboy sync layout, and doesn't allow anything other than Tomboy notes 
in that layout to be uploaded or downloaded.

When a Tomboy note is uploaded to the DAV share, we would also create 
the Tomboy note metadata in the database with e.g. the public/private 
flag or a list of users who would have access rights to view it.

If this is how it works, I do wonder if it is easier to write the 
note-list-to-filesystem-tree code on the server side, or to write a 
different sync plugin for the client side that does not convert to a 
filesystem tree or mess with FUSE as an intermediate step but instead 
just talks to the server in terms of note names and revisions.

I have to read more code to figure out all the details, I'm sure there's 
much to be revised in the above notes.

Havoc