Re: [Opensync-devel] Ubuntu UDS-Boston, Syncing solutions

[John Carr <john carr unrouted co uk>]

Hello again

The problem I see is that there are two IMO great sync implementations.
OpenSync has some really nice features for the PIM sync case
(capabilities) and because of the care that has been taken with design
and scope its low level and portable. Conduit has some really nice
features for media sync, like dynamic conversion to the correct video or
audio format, and nice (though could be nicer!) HAL stuff. And it
exports interfaces over dbus, so "You have connected X" type wizards are
really straightforward to deliver.

Right now opensync especially, but conduit as well, are established
enough that they are not going to abandon the work they've done, but
also don't want to fight and duplicate work. I think we can agree that
opensync is lowlevel. And that conduit is quite.. immoral.. with its
desires to take over your desktop :) 

> >     --> Focus more on online services
> We might conflict here - since we would provide our own set of OpenSync 
> plugins. Currently we limited our self to PIM data .. since OpenSync is 
> awesome for syncing content-related data in little size - like PIM data ;)
> 
> Example: google-calendar plugin (Could need more love - we need support for 
> the latest GCal API features, like batch processing and the calendar 
> ressource API)

A big chunk of our dataproviders currently target getting photos from
F-spot, EOG and Cheese online to Flickr, SmugMug, Box.net and others. We
also do file sync to box.net and hopefully S3.

Our goal is that we become a blessed dependency of GNOME and get f-spot
to use us for exports instead of its own plugins. We are also
implementing f-spot code to let us put data back in. We also want to get
to the point where tomboy switches to conduit instead of its own sync
engine. We implemented the dbus interface for this. I think we already
more than match their supported sync targets. 

I think the fact that we are putting code back into those projects (that
a pure opensync project could benefit from too) is a good example of us
being integrated and "good" players in this game.

> >     --> Focus on a attractive file sync use case (not rsync level, but
> > workable for hundreds of files at a time)
> 
> Hmm, this might conflict with csync ... I'm in contact with the author of 
> csync. He evaluated OpenSync before i started with csync implementation. 
> Actually it is his diploma thesis, so the quality should be quite high. It's 
> like OpenSync+rsync - it doesn't touch the content like OpenSync could to 
> converter between different formats, it supports bidirectional 
> synchronization, rsync only supports uni-directional which isn't acceptable 
> for a default-desktop-user.

I'm not sure how the implementations stack up as csync is quite low
level and the last time i saw it was tied to samba(?). The work John S
has done here is at a GnomeVFS level so can sync to quite a few
different places, and to sites like S3 and Box.net. Writing a KioSlave
DP would still allow syncing to box.net and S3, obviously.

This could also pokes my AutoSync code in all the right places so will
sync files as they change (with an optional delay to stop spamming).

> 
> > 2) I am also trying to actively make our Conduit core more
> > integratable with opensync plugins
> >     --> Support both hashes and mtimes
> >     --> Be smart with object comparison and rebuilding relationships
> >     --> Remove all gnome/gtk dependencies in core
> >     --> Reduce memory consumption
> >     --> Etc
> 
> This sound like completely hijacking the OpenSync synchronization logic :)
> Or maybe i missed something...

Integrating with opensync plugins is phase 1, and very mean :-) The hash
refactoring allows us to use the opensync dataproviders as though they
were native to conduit. Only in theory, i've only tested it with the
Evo2 plugin! Unfortunately i can't escape needing to shim the opensync
plugins yet :(

The object comparison/rebuilding relationships stuff does/will take a
different approach to opensync. In this case, python will allow rapid
prototyping of something you might want to backport into opensync :) But
John S is still working on the finer points in his head.

> I know about the idea of J2ck to have a solution to provide different 
> synchronization logic - not quite sure about that yet if this should really 
> done in this way. Since we could waste a lot of time with double effort ... 
> and for sure currently conduit can compare with the synchronization logic of 
> OpenSync - actually I'm not aware of any powerful synchronization logic like 
> the one OpenSync has - even the propertaries ones.

My thoughts are that the code of conduit is reasonably well split up and
that if in the same way Conduit can use OpenSync dataproviders it should
work in reverse. I can proxy calls by opensync onto the conduit
dataproviders so opensync can use our plugins. This can then be
controlled via conduits dbus interface. 

Putting conduit's dbus in charge is important as it allows GUI's to
respond to HAL, avahi and NetworkManager changes without re-inventing
any wheels. It also means that opensync, conduit, csync, and other(?)
partnerships can be managed from one interface. And a considerable
amount of logic can be shared between Kubuntu/Ubuntu GUIs (I presume a
GTK gui would be out of the question Martin? What about Xubuntu?).
Finally, mulitple clients can be active at once. For example, an
application can manage syncing internally.. This ties in to work for
online desktop where Conduit is used to build an app connected to a
database with an online backend that it keeps in sync with. There are
much better uses cases, but i'm already bordering on a stupid length
e-mail.

The main motivation to keep conduit sync logic around is that its a lot
more agile than opensync right now. We are not sure how easily some
things would be to implement in opensync, or if they would be accepted.
Arguably we should push to opensync, and i'd agree. But right now its a
time where we can't afford to lose those features / face massive
regressions. Allow me to explain:

The GNOME Online Desktop people are implementing sync left right and
centre. They are very much for tomboy having its own sync and recently
brought us gconf-sync-daemon. If we can't do tomboy sync and gconf sync
AND have AutoSync that matches theirs then we don't have any hope of
even holding a conversation about removing yet more duplication from the
free desktop. 

There are numerous other sync projects popping up. Bright side: At least
opensync and conduit are doing something to work together, eh? We need
to focus on getting sync out of the box in all distros (and Conduit has
contacts on Foresight and at least one other distro to that end as well
as trying to build platform glue in to the GNOME platform). Otherwise
we'll meet the downside. It will be too late and GNOME will be full of
mini-sync engines and you'll have to install x different sync clients
and involve services like scheduleworld to make everything work.

> >
> > Furthur to that the following is a list of areas that I have avoided
> > development work on so far. This is both because of time issues, and
> > because I wanted to agree on how to procede together first.
> 
> >    --> How can we use HAL to encapsulate the sync capabilities of a
> > device. Such as have fields in FDI files that describe a mobilie
> > device as having "Notes", "Contact", etc data to sync
> That would be indeed very intersting. But i hope you agree that we don't want 
> to create tons of separated FDI files to list all the device capabilities. 
> Only because of the  reason it's easy...
> 
> We should kindly ask the HAL developers to add support for certain protocols 
> and probe them if a device got attached. Examples:
<snip>

I think it would be crazy to ask HAL to speak anything to do with
Windows Mobile. Anyway, this is something we can do ourselves and i had
planned for Windows Mobile. I believe HAL supports callout binaries that
can populate HAL with more info. So in the WM case, I will be populating
all the the info that can be scraped by libsynce into the HAL tree.

I think it's more appropriate to have a bunch of FDI files that say
which plugin is needed to "speak" to a given device. This is what I have
been experimenting with in the SynCE case and in Conduit.

> > At this point I would (1) 
> > communicate with the daemon from conduit over DBus. I would also (2)
> > rewrite the opensync syncml plugin(s) to communicate with these
> > daemons over dbus. Obviously I have not yet considered how much work
> > (2) is, and even if its possible. While this still has some code
> > duplication, its at least an effective middle ground (through
> > libsyncml). I would appreciate your thoughts on this.
> Sorry, this is really not needed. You should have a look on the OBEX Spec.
> The commandline tool obexftp has a simple reference implementation how to 
> request the x-obex/capabilties. "obexftp -X"
> 
> If  the HAL developers don't fully disagree with my idea of implementation 
> there is no need to have yet another system daemon.
> But i see your point in have same process/daemon running which takes care  all 
> the time about triggering a sync or reporting there is a new and unknown 
> device which could be synced.

I don't like the idea of another daemon, and if HAL can be populated
with HAL callouts then I don't think there is a need for this. I would
rather have python-libsyncml than another daemon.

> I already had an idea about a (D-Bus) session daemon (which is desktop 
> independent), which could handle mobile devices related stuff. Like 
> triggering a sync if your local PIM environment added a new bunch of new 
> contact entries... so periodic sync like every 5 Minutes is not needed. I 
> would suggested to only do syncs if the device appear and is after 5 Minutes 
> still available. I called this "MobileStation" - anyway thats quite offtopic 
> now ;)

I'd encourage you to work with Conduit to realise your version of
MobileStation. If we split our source code how I see it your
MobileStation is one of the pieces that is left, but with a much more
interesting scope. It is used for desktop-to-desktop sync, responding to
internet availability and integrating with GNOME Online Desktop rather
than just mobile devices.

Perhaps this is something for Sync Fest? There are other things I would
like to discuss there, including a convergence of our plugin API.
Basically, the current API doesn't really cater for non-sync cases. But
I have a few cases where I want to uses conduit dataproviders directly. 

Moving forward from UDS, I'm really interested in trying to get
interested parties together at Linux Conf AU as both myself and John S
will be there. I'm also pretty available within the confines of the UK.

John