Re: GSoC 2011: Synchronizing F-Spot data

[Simon Lindgren <simon n lindgren gmail com>]

lör 2011-04-02 klockan 02:44 +0200 skrev Simon Lindgren:
> Hi Sven,
> 
> fre 2011-04-01 klockan 22:27 +0200 skrev Sven Utcke:
> > Hello Simon,
> > 
> > > > Now I need to find a mentor.
> > 
> > If you are looking for one on this list, you might want to clarify
> > what the requirements for / expectations of a mentor are...
> 
> I thought I hade a good idea about this... but when trying to write it
> down I realized that I do not. The book on mentoring:
> http://www.booki.cc/gsoc-mentoring/
> provides some insight into this, but it seems to be oriented towards
> mentoring _organizations_ more than individual mentors.

Ugh, minutes after sending I found this:
http://people.gnome.org/~federico/docs/summer-of-code-mentoring-howto/index.html

> 
> What I believe I would need the most help with is:
>  * Some F-Spot issues. I am new to the code base and I will probably
> fail to notice some possible ways of implementing things.
>  * Protocols; If this ends up being a web service I will need some
> feedback on the api. I have previously created protocols (much lower
> level than HTTP though) and failed miserably... hopefully I have learned
> a few things, but you never know.
> 
> > 
> > >  * Some notes on why I consider Flickr, Amazon S3 and Conduit
> > >    problematic for implementing this particular task.
> > 
> > I'll add my comments on that below
> > 
> > > In my particular case, I want to browse and edit my images on my
> > > main computer, which is a desktop. When traveling I only have my
> > > laptop with me and thus I do not currently have access to my F-Spot
> > > database. A workaround is to copy the database between the
> > > computers, but that is tedious and error prone. This project aims to
> > > solve this problem by introducing a synchronization solution capable
> > > of synchronizing the F-Spot data model in its entirety across
> > > multiple F-Spot installations.
> > 
> > So in your usecase, no two people will work with the same data
> > concurrently.  Doesn't that mean you could simply sync the DB,
> > provided the photos are already stored offline?  So all you need is
> > some conflict resolution on the DB (fairly easy, as the DB can be
> > converted to text and back to Sqlite3 format without loss of
> > information), and possibly some local caching, right?
> > 
> 
> My usecase is single-user, that is correct. It is also fairly rare to
> sync, since that is only needed when going on a trip. But I can also
> imagine a case where there are multiple users (especially if the catalog
> concept below is implemented) or more frequent syncing. Also, there is
> the need for file locking even in the case of only a single user since
> that user could keep F-Spot running on two computers simultaneously,
> with potentially conflicting edits. Relying on the user to not sync
> multiple computers at the same time could potentially lead to weird
> issues, especially if the synchronization is done in the background.
> 
> > > It would also be nice if it could
> > >  * support additional data that might be added by addins
> > >  * support lightweight transfer of on-file metadata changes
> > > It might be too much to implement those last two things this summer though.
> > 
> > Well, the latter, at least, ought to be quite easy.  Although it
> > either requires a complete download (and, worse, upload) of the file,
> > or a machine running some server-software, which I sort of dislike,
> > see below...
> > 
> > > I have looked at some different approach to the problem, including
> > > using a web based storage service like Amazon S3 or a Flickr Pro
> > > account, but those kinds of approaches will probably make it hard to
> > > handle the F-Spot-specific parts of the metadata, like versions and
> > > the tag tree.
> > 
> > I really sort-a like the idea of using flickr or any other
> > webstorage.  The problem, of course, is that this would not allow for
> > partial updates of the files (never mind whether we are talking
> > metadata or photos).  Still, it would be quite nice.
> > 
> 
> Yes, it definitely would be nice.
> 
> > > Therefore, my current thinking is that I will implement this as a
> > > remote repository of the F-Spot data that support pushing and
> > > pulling data to/from the repository and also some extra
> > > functionality to simplify synchronization.  To communicate with the
> > > server an XML-based REST api will be used. XML makes it easy to
> > > integrate XMP metadata, which I believe will cover much of the
> > > per-image data. An addin to F-Spot keeps the local F-Spot database
> > > up to date with the server so that F-Spot can continue to use the
> > > already existing queries, and load images quickly from the local
> > > harddrive. This addin will also track changes to the data and manage
> > > synchronizing the changes back to the server.
> > 
> > In my estimate, the by far easiest way to achieve your particular goal
> > under those particular constraints (no concurrent access) would be to
> > find a server running unison and be done (but I did mention that,
> > didn't I :-)
> 
> Yes, and I looked at it briefly. It handles concurrent access too, but
> that is only true for files that are actually synced by Unison. If
> storing metadata as .xmp sidecars and then some other file format for
> the tags, the import rolls, the versioning and so on, then Unison could
> be a workable solution for all of the synchronization.
> 
> > 
> > So if you aren't going that way, it might be worth adding some extra
> > functionality justifying the overhead.
> > 
> > >  * mhutch suggested that a concept like catalogs in Lightroom might
> > >    be useful.  Basically, this would be a way to only sync selected
> > >    pictures.  It could be implemented as a tag query, to tie in to
> > >    existing F-Spot concepts.
> > 
> > Makes sense.  So there's some extra functionality at least.
> > 
> > >  * Partial local cache. Perhaps this overlaps a bit with the above,
> > >    but it transform the local pictures folder into a cache
> > >    containing the most recently used files. Reduces initial
> > >    bandwidth requirements and used storage space.
> > 
> > Did I mention I'm all for that?  Although, so far I'm not clear
> > whether your original plan was to store all photos _only_ offline, o
> > whether to store copies of all photos on the server and all clients (I
> > had guessed the latter, but maybe erroneously).
> > 
> 
> You guessed correctly, I initially meant for the pictures to be stored
> on all clients and on the server as well.
> 
> Also, I would like to add that neither of the extra features necessarily
> exclude Unison as a possible implementation strategy, at least from what
> I have learned so far.
> 
> I mainly see one issue with Unison: It requires the server and the
> client to be running the same version of the program. I think a natural
> step for a thing like this is to expand it to also include a web service
> so people don't need to host their own servers. And with Unison this
> might raise problems with versioning. 
> 
> Still, Unison would be an elegant solution... Maybe those potential
> issues can be solved, maybe they don't even need to be solved.
> 


-- 
Simon Lindgren