Re: GSoC 2011: Synchronizing F-Spot data

[Simon Lindgren <simon n lindgren gmail com>]

Hi Sven,

fre 2011-04-01 klockan 22:27 +0200 skrev Sven Utcke:
> Hello Simon,
> 
> > > Now I need to find a mentor.
> 
> If you are looking for one on this list, you might want to clarify
> what the requirements for / expectations of a mentor are...

I thought I hade a good idea about this... but when trying to write it
down I realized that I do not. The book on mentoring:
http://www.booki.cc/gsoc-mentoring/
provides some insight into this, but it seems to be oriented towards
mentoring _organizations_ more than individual mentors.

What I believe I would need the most help with is:
 * Some F-Spot issues. I am new to the code base and I will probably
fail to notice some possible ways of implementing things.
 * Protocols; If this ends up being a web service I will need some
feedback on the api. I have previously created protocols (much lower
level than HTTP though) and failed miserably... hopefully I have learned
a few things, but you never know.

> 
> >  * Some notes on why I consider Flickr, Amazon S3 and Conduit
> >    problematic for implementing this particular task.
> 
> I'll add my comments on that below
> 
> > In my particular case, I want to browse and edit my images on my
> > main computer, which is a desktop. When traveling I only have my
> > laptop with me and thus I do not currently have access to my F-Spot
> > database. A workaround is to copy the database between the
> > computers, but that is tedious and error prone. This project aims to
> > solve this problem by introducing a synchronization solution capable
> > of synchronizing the F-Spot data model in its entirety across
> > multiple F-Spot installations.
> 
> So in your usecase, no two people will work with the same data
> concurrently.  Doesn't that mean you could simply sync the DB,
> provided the photos are already stored offline?  So all you need is
> some conflict resolution on the DB (fairly easy, as the DB can be
> converted to text and back to Sqlite3 format without loss of
> information), and possibly some local caching, right?
> 

My usecase is single-user, that is correct. It is also fairly rare to
sync, since that is only needed when going on a trip. But I can also
imagine a case where there are multiple users (especially if the catalog
concept below is implemented) or more frequent syncing. Also, there is
the need for file locking even in the case of only a single user since
that user could keep F-Spot running on two computers simultaneously,
with potentially conflicting edits. Relying on the user to not sync
multiple computers at the same time could potentially lead to weird
issues, especially if the synchronization is done in the background.

> > It would also be nice if it could
> >  * support additional data that might be added by addins
> >  * support lightweight transfer of on-file metadata changes
> > It might be too much to implement those last two things this summer though.
> 
> Well, the latter, at least, ought to be quite easy.  Although it
> either requires a complete download (and, worse, upload) of the file,
> or a machine running some server-software, which I sort of dislike,
> see below...
> 
> > I have looked at some different approach to the problem, including
> > using a web based storage service like Amazon S3 or a Flickr Pro
> > account, but those kinds of approaches will probably make it hard to
> > handle the F-Spot-specific parts of the metadata, like versions and
> > the tag tree.
> 
> I really sort-a like the idea of using flickr or any other
> webstorage.  The problem, of course, is that this would not allow for
> partial updates of the files (never mind whether we are talking
> metadata or photos).  Still, it would be quite nice.
> 

Yes, it definitely would be nice.

> > Therefore, my current thinking is that I will implement this as a
> > remote repository of the F-Spot data that support pushing and
> > pulling data to/from the repository and also some extra
> > functionality to simplify synchronization.  To communicate with the
> > server an XML-based REST api will be used. XML makes it easy to
> > integrate XMP metadata, which I believe will cover much of the
> > per-image data. An addin to F-Spot keeps the local F-Spot database
> > up to date with the server so that F-Spot can continue to use the
> > already existing queries, and load images quickly from the local
> > harddrive. This addin will also track changes to the data and manage
> > synchronizing the changes back to the server.
> 
> In my estimate, the by far easiest way to achieve your particular goal
> under those particular constraints (no concurrent access) would be to
> find a server running unison and be done (but I did mention that,
> didn't I :-)

Yes, and I looked at it briefly. It handles concurrent access too, but
that is only true for files that are actually synced by Unison. If
storing metadata as .xmp sidecars and then some other file format for
the tags, the import rolls, the versioning and so on, then Unison could
be a workable solution for all of the synchronization.

> 
> So if you aren't going that way, it might be worth adding some extra
> functionality justifying the overhead.
> 
> >  * mhutch suggested that a concept like catalogs in Lightroom might
> >    be useful.  Basically, this would be a way to only sync selected
> >    pictures.  It could be implemented as a tag query, to tie in to
> >    existing F-Spot concepts.
> 
> Makes sense.  So there's some extra functionality at least.
> 
> >  * Partial local cache. Perhaps this overlaps a bit with the above,
> >    but it transform the local pictures folder into a cache
> >    containing the most recently used files. Reduces initial
> >    bandwidth requirements and used storage space.
> 
> Did I mention I'm all for that?  Although, so far I'm not clear
> whether your original plan was to store all photos _only_ offline, o
> whether to store copies of all photos on the server and all clients (I
> had guessed the latter, but maybe erroneously).
> 

You guessed correctly, I initially meant for the pictures to be stored
on all clients and on the server as well.

Also, I would like to add that neither of the extra features necessarily
exclude Unison as a possible implementation strategy, at least from what
I have learned so far.

I mainly see one issue with Unison: It requires the server and the
client to be running the same version of the program. I think a natural
step for a thing like this is to expand it to also include a web service
so people don't need to host their own servers. And with Unison this
might raise problems with versioning. 

Still, Unison would be an elegant solution... Maybe those potential
issues can be solved, maybe they don't even need to be solved.

-- 
Simon Lindgren