Always up to date

[John Carr <john carr unrouted co uk>]

Hi all

So i've been thinking about how exactly "always up to date" should work.

Our initial infrastructure was to just fire a signal when a changed
happens and let the ordinary sync engine figure out what to do. This
seems a little wasteful and spammy.

Attached is a patch which captures change events so that a later refresh
isn't needed. A tweakable timeout is used so that updates are batched.
This is mainly to reduce how much "syncing spam" we might create if we
have chatty applications/services.

Buffering changes introduces a side effect. What if you add and edit and
delete all before a sync? Those kind of situations are filtered out in
the handle_added/modified/deleted functions. I've also provided a test
case that makes sure this logic is behaving as expected (because it
looks *butt ugly*).

Finally, there is a _get_changes method which returns just the changes
we know about, without needing a refresh. On large datasets, thats a
nice win for us.

I still need to work out when to use the AutoSync _get_changes and when
to use DeltaProviders get_changes. When I decide on this behaviour it'll
be implemented in such a way that child dp's do not need to consider it.
(Wrapping like i'm a decorator or something).

As i'll provide an alternative get_changes, the ordinary sync is used.
So the changed trigger would behave the same as ever.

The patch updates Tomboy and Folders to use this class. I'll post an
updated version in the morning that "blesses" gconf sync.

You may be asking why I've implemented this in a separate class. I'm
really keen to isolate this code. If someone else comes along to look at
this code they will know its all in one file, instead of sprayed over
the DataProvider class. DataProviders that don't support
always-up-to-date also don't have to be weighed down by it.

Thoughts are sought.

John

Index: test/python-tests/TestUp2Date.py
===================================================================
--- test/python-tests/TestUp2Date.py	(revision 0)
+++ test/python-tests/TestUp2Date.py	(revision 0)
@@ -0,0 +1,73 @@
+#common sets up the conduit environment
+from common import *
+
+import conduit.dataproviders.AutoSync as AutoSync
+
+a = AutoSync.AutoSync()
+a.handle_added("1")
+ok("ADD", len(a.as_added)==1 and len(a.as_modified)==0 and len(a.as_deleted)==0)
+
+a = AutoSync.AutoSync()
+a.handle_modified("1")
+ok("MOD", len(a.as_added)==0 and len(a.as_modified)==1 and len(a.as_deleted)==0)
+
+a = AutoSync.AutoSync()
+a.handle_deleted("1")
+ok("DEL", len(a.as_added)==0 and len(a.as_modified)==0 and len(a.as_deleted)==1)
+
+# If we get two adds, only one should appear in get_changes!
+a = AutoSync.AutoSync()
+a.handle_added("1")
+a.handle_added("1")
+ok("ADD ADD", len(a.as_added)==1 and len(a.as_modified)==0 and len(a.as_deleted)==0)
+
+# If we get an add and an edit, ignore the edit!
+a = AutoSync.AutoSync()
+a.handle_added("1")
+a.handle_modified("1")
+ok("ADD MOD", len(a.as_added)==1 and len(a.as_modified)==0 and len(a.as_deleted)==0)
+
+# If we get and add and a delete then other side never even needs to see it!
+a = AutoSync.AutoSync()
+a.handle_added("1")
+a.handle_deleted("1")
+ok("ADD DEL", len(a.as_added)==0 and len(a.as_modified)==0 and len(a.as_deleted)==0)
+
+# If we get a mod and an add, ignore the add!
+a = AutoSync.AutoSync()
+a.handle_modified("1")
+a.handle_added("1")
+ok("MOD ADD", len(a.as_added)==0 and len(a.as_modified)==1 and len(a.as_deleted)==0)
+
+# If we get two mods, only one should appear in get_changes!
+a = AutoSync.AutoSync()
+a.handle_modified("1")
+a.handle_modified("1")
+ok("MOD MOD", len(a.as_added)==0 and len(a.as_modified)==1 and len(a.as_deleted)==0)
+
+# If we get a mod and a delete, only delete should appear in get_changes!
+a = AutoSync.AutoSync()
+a.handle_modified("1")
+a.handle_deleted("1")
+ok("MOD DEL", len(a.as_added)==0 and len(a.as_modified)==0 and len(a.as_deleted)==1)
+
+# If we get a del and then add, treat as modified in get_changes!
+a = AutoSync.AutoSync()
+a.handle_deleted("1")
+a.handle_added("1")
+ok("DEL ADD", len(a.as_added)==0 and len(a.as_modified)==1 and len(a.as_deleted)==0)
+
+# If we get a del and then a mod, the universe will fall apart. ignore the mod??
+a = AutoSync.AutoSync()
+a.handle_deleted("1")
+a.handle_modified("1")
+ok("DEL MOD", len(a.as_added)==0 and len(a.as_modified)==0 and len(a.as_deleted)==1)
+
+# If we get two dels, only one should appear in get_changes!
+a = AutoSync.AutoSync()
+a.handle_deleted("1")
+a.handle_deleted("1")
+ok("DEL DEL", len(a.as_added)==0 and len(a.as_modified)==0 and len(a.as_deleted)==1)
+
+finished()
+
Index: conduit/modules/FileModule/FileModule.py
===================================================================
--- conduit/modules/FileModule/FileModule.py	(revision 969)
+++ conduit/modules/FileModule/FileModule.py	(working copy)
@@ -5,6 +5,7 @@
 import conduit
 import conduit.dataproviders.DataProvider as DataProvider
 import conduit.dataproviders.File as FileDataProvider
+import conduit.dataproviders.AutoSync as AutoSync
 import conduit.Utils as Utils
 
 MODULES = {
@@ -55,7 +56,7 @@
     def get_UID(self):
         return Utils.get_user_string()
 
-class FolderTwoWay(FileDataProvider.FolderTwoWay):
+class FolderTwoWay(FileDataProvider.FolderTwoWay, AutoSync.AutoSync):
     """
     TwoWay dataprovider for synchronizing a folder
     """
@@ -75,6 +76,7 @@
                 FolderTwoWay.DEFAULT_HIDDEN,
                 FolderTwoWay.DEFAULT_COMPARE_IGNORE_MTIME
                 )
+        AutoSync.AutoSync.__init__(self)
         self.need_configuration(True)
 
         self._monitor_folder_id = None
@@ -117,7 +119,6 @@
         if self._monitor_folder_id != None:
             gnomevfs.monitor_cancel(self._monitor_folder_id)
             self._monitor_folder_id = None
-
         try:
             self._monitor_folder_id = gnomevfs.monitor_add(self.folder, gnomevfs.MONITOR_DIRECTORY, self._monitor_folder_cb)
         except gnomevfs.NotSupportedError:
@@ -129,8 +130,12 @@
         Called when a file in the current folder is changed, added or deleted
         """
         # supported events = CHANGED, DELETED, STARTEXECUTING, STOPEXECUTING, CREATED, METADATA_CHANGED
-        if event in (gnomevfs.MONITOR_EVENT_CREATED, gnomevfs.MONITOR_EVENT_CHANGED, gnomevfs.MONITOR_EVENT_DELETED):
-            self.emit_change_detected()
+        if event == gnomevfs.MONITOR_EVENT_CREATED:
+            self.handle_added(event_uri)
+        elif event == gnomevfs.MONITOR_EVENT_CHANGED:
+            self.handle_modified(event_uri)
+        elif event == gnomevfs.MONITOR_EVENT_DELETED:
+            self.handle_deleted(event_uri)
 
 class USBFactory(DataProvider.DataProviderFactory):
     def __init__(self, **kwargs):
@@ -175,5 +180,3 @@
 
         del self.usb[udi]
 
-
-
Index: conduit/modules/TomboyModule.py
===================================================================
--- conduit/modules/TomboyModule.py	(revision 969)
+++ conduit/modules/TomboyModule.py	(working copy)
@@ -11,6 +11,7 @@
 import conduit
 from conduit import log,logd,logw
 import conduit.dataproviders.DataProvider as DataProvider
+import conduit.dataproviders.AutoSync as AutoSync
 import conduit.Exceptions as Exceptions
 import conduit.datatypes.Note as Note
 import conduit.datatypes.File as File
@@ -30,7 +31,7 @@
 	"TomboyNoteTwoWay" :    { "type": "dataprovider" }
 }
 
-class TomboyNoteTwoWay(DataProvider.TwoWay):
+class TomboyNoteTwoWay(DataProvider.TwoWay, AutoSync.AutoSync):
     """
     LUID is the tomboy uid string
     """
@@ -45,8 +46,13 @@
 
     def __init__(self, *args):
         DataProvider.TwoWay.__init__(self)
+        AutoSync.AutoSync.__init__(self)
         self.notes = []
         self.bus = dbus.SessionBus()
+        if self._check_tomboy_version():
+            self.remoteTomboy.connect_to_signal("NoteAdded", lambda uid: self.handle_added(str(uid)))
+            self.remoteTomboy.connect_to_signal("NoteSaved", lambda uid: self.handle_modified(str(uid)))
+            self.remoteTomboy.connect_to_signal("NoteDeleted", lambda uid, x: self.handle_deleted(str(uid)))
 
     def _check_tomboy_version(self):
         if Utils.dbus_service_available(self.bus,TOMBOY_DBUS_IFACE):
Index: conduit/dataproviders/AutoSync.py
===================================================================
--- conduit/dataproviders/AutoSync.py	(revision 0)
+++ conduit/dataproviders/AutoSync.py	(revision 0)
@@ -0,0 +1,54 @@
+import gobject
+
+class AutoSync(object):
+
+    def __init__(self):
+        self.as_added = []
+        self.as_modified = []
+        self.as_deleted = []
+        self.timeout = 5
+        self._timeout_id = 0
+
+    def handle_added(self, uid):
+        if not uid in self.as_added and not uid in self.as_modified:
+            if uid in self.as_deleted:
+                self.as_deleted.remove(uid)
+                self.as_modified.append(uid)
+            else:    
+                self.as_added.append(uid)
+            self._handle_change()
+
+    def handle_modified(self, uid):
+        if not uid in self.as_modified and not uid in self.as_deleted and not uid in self.as_added:
+            self.as_modified.append(uid)
+            self._handle_change()
+
+    def handle_deleted(self, uid):
+        if not uid in self.as_deleted:
+            if uid in self.as_added:
+                self.as_added.remove(uid)
+            else:
+                if uid in self.as_modified:
+                    self.as_modified.remove(uid)
+                self.as_deleted.append(uid)
+            self._handle_change()
+
+    def _handle_change(self):
+        # reset timer..
+        if self._timeout_id > 0:
+            gobject.source_remove(self._timeout_id)
+            self._timeout_id = 0
+
+        # add a new one, or trigger sync immediately
+        if self.timeout > 0:
+            self._timeout_id = gobject.timeout_add(self.timeout * 1000, self._handle_sync)
+        else:
+            self.emit_change_detected()
+
+    def _handle_sync(self):
+        self._timeout_id = 0
+        self.emit_change_detected()
+        print self._get_changes()
+
+    def _get_changes(self):
+        return self.as_added, self.as_modified, self.as_deleted