[final report] Raw processing in f-spot

[Ruben Vermeersch <ruben savanne be>]

Hi all,

The pencils down date for the summer of code is coming up next monday,
but due to unavoidable social obligations, I'm sending this in
early. Don't worry, I'm going to continue the work afterwards.

In this mail:
  * Quick recap
  * A bit of background
  * Realizations
  * Code changes
  * Testing my work
  * Known issues
  * Going further
  * In closing

== Quick recap ==

  As a quick recap, my GSoC project is about RAW processing. Support
  for RAW files in F-Spot was in a bit of a dire state. You could
  import and view the files (well, the small preview within the
  files), but you couldn't do much else with them, unless you
  processed each of them with a tool like ufraw.

  If we want to be a serious photographers tool, we'll need proper RAW
  support. This is a huge task, but in my GSoC project, I'll put down
  the foundations and make the first steps towards getting there.


== A bit of background ==

  Photos generally come in two varieties: JPEG files, which are
  pre-processed files, basically big maps of pixels, that can be
  displayed directly and (on higher end cameras such as DSLRs) RAW
  files, which contain the recorded sensor data in it's purest
  untouched version.

  To display a RAW file, a number of steps have to be taken: the RAW
  file needs to be unpacked (each manufacturer has it's own format),
  the sensor data needs to be transformed to pixels, demosaiced,
  delinearized, gamma corrected, color corrected and converted into an
  8 bit color space.

  Why go through all this trouble? Because tweaking the parameters of
  the RAW processing as opposed to changing the end result allows
  photo adjustments to happen in much higher bitdepths, leading to
  better quality. And photographers are generally quality whores when
  it comes to images.

  The de-facto standard RAW processing tool is dcraw. Multiple
  libraries have been based on this tool, including libraw, which I
  used for RAW processing.


== Realizations ==

  Over the course of summer, I have worked on the following:

  * New loading infrastructure: RAW loading is slow, due to all the
    processing involved. In order to maintain responsiveness of the UI
    during slow loads, a progressive loader was needed. I've created a
    new loading infrastructure that has a specific API for requesting
    different image sizes, both synchronously and asynchronously. All
    of F-Spot has been ported to use the new loaders. One future
    change which I'll build on this is loader reuse: currently image
    files are loaded multiple times: for the imageview, the histogram,
    ... This can be avoided, leading to better performance and less
    memory use.

  * A RAW file loader based on libraw. I have written a native library
    wrapping libraw in a GObject API: libfspotraw. Then I have created
    a C# bindings library for this library: libfspotraw-sharp. With
    these libraries I've built a libraw based image loader that hooks
    in the previously mentioned loader infrastructure. This means
    F-Spot can now load all RAW image formats supported by dcraw,
    which basically means all of them.

  * Improved RAW handling: things like using the F-Spot editing tools
    from the edit sidebar or exporting a RAW file to flickr were not
    possible. All of this works in my branch. When a user selects a
    RAW file and chooses to upload it to Flickr, F-Spot will try
    default processing parameters instead of just failing. This allows
    users to use RAW files as if they were JPEG files, which basically
    means that you can safely shoot everything in RAW and not suffer
    the downsides of them not being very usable in F-Spot. The shots
    that do need fixing are still available in their full glory
    though.

  * Repeatable editing: I've added the initial bits of
    non-destructive, repeatable editing. This is a big deal: it means
    that rather than every action you do being destructive and lossy,
    it will store a set of transformation parameters and always apply
    this to the original.  Better image processing quality, but also
    tweakability.  You can do a color adjustment and tweak it at a
    later point in time, similar to what e.g.  Adobe Lightroom offers.
    This is implemented for the color adjustment tools (sepia,
    desaturate and color adjust). Converting the other editors
    will be a bit harder, but I've made the code such that the old and
    new ones can live together. Adding a curves tool should be trivial
    here (except the curves widget itself).

  What's missing:

  * 16 bit image processing: This is something I wanted to add, but
    had to let go off. Turns out 16 bit image processing is impossible
    with GdkPixbufs, dcraw is horribly designed and thus unusable
    unless completely rewritten and GEGL is way to slow. Fixing this
    will take more than one summer, but it's certainly something I
    want to go after when there is no fixed deadline :-)

  So the sum is: RAW loading, handling and non-destructive: in, 16 bit
  processing: out. In the end, it means that we are a whole lot closer
  to removing the need for ufraw and becoming a nicely integrated RAW
  photography solution.


== Code changes ==

  This section can safely be skipped if you're not an F-Spot
  developer. In my branch I made the following changes (this list
  might be incomplete):

  * Added libfspotraw, a libraw embedded library.
   
  * Added libfspotraw-sharp, C# bindings to libfspotraw.
   
  * I broke Imaging code out into an FSpot.Imaging.dll assembly
   
  * Added a new loader infrastructure in Loaders/, which lives in 
    FSpot.Loaders.dll. There are two loaders: A reworked
    GdkImageLoader based on sde's ImageLoader and a libraw based
    LibrawImageLoader.

  * Converted all of F-Spot to use IImageLoader.
   
  * Fixed quite a lot of RAW handling bugs scattered around the place.
  
  * Changed PhotoVersion to have different types: Simple, Processable
    and Hidden. Check the comments in PhotoVersion.cs for their
    descriptions.

  * Added a RepeatableEditor editor type, which makes it possible to
    implement non-destructive editors.

  * This works by hooking into a processing pipeline which lives in
    Editors/Processing/. This framework transforms images from
    original to fully processed. In the future, when it gains 16 bit
    support, this can be used to do RAW processing. For now there is
    only one step in the pipeline, but it is built with extensibility
    in mind. Once extra steps are added, it'll have to become smarter
    about performance and caching, but we'll tackle that when we get
    there.

  * Added a settings store, which stores processing settings. This is
    stored in the processing_settings table.

  * Rewrote bits of Editor to make previewing suck less, even though
    parts of it still do.


== Testing my work ==

  My code can be found on gitorious:

   git clone git://gitorious.org/+f-spot-developers/f-spot/rubenvs-clone.git
   cd rubenvs-clone
   git checkout libfspotraw
   ./autogen.sh
   make

  Backup your database first, as this branch introduces some changes
  to the schema. Feedback is highly appreciated! The branch is known
  to fix these bugs:
  http://bugzilla.gnome.org/showdependencytree.cgi?id=590150


== Known issues ==

  There are a couple of things that need fixing (or accepting) before
  merging this back to master:

   * Difference between embedded JPEG previews and processed version.
     When dcraw processes a file, it looks different from the embedded
     jpeg preview (for some files). This means that the color of a
     photo will suddenly change when the full size is loaded, often to
     a less-saturated version.  As camera manufacturers don't publish
     the way they process their files in-camera, it will be hard to
     avoid this phenomenon. I'm not sure what the best way is to go
     about this. Other applications build up a large cache, maybe
     we'll have to do that too.

   * Memory usage. For some reason memory usage keeps growing, despite
     disposing all loaders (I checked with debug statements, they do
     get freed) and not leaking code in the native library (there's a
     valgrind test to confirm that). So what's wrong? I don't know,
     I've searched like crazy, but I still haven't tracked it down.

   * Metadata handling is mostly non-existant. This is a problem in
     all of F-Spot which needs to be fixed. Related to RAW files, this
     mostly means that metadata is lost when editing or exporting.

   * And probably some breakage here and there which I have yet to
     discover.


== Going further ==

  I'd like to get this merged at some point, but to do that the issues
  above need to have some sort of resolution. Also, I don't want to
  merge this unless it gets a bit of more testing. It works on my
  system, but it might burn yours ;-). I fully intend to keep
  maintaining this after the GSoC, F-Spot is still my main hacking
  project and will stay that way.


== In closing ==

  Congratulations if you've read all the way until the end. Do check
  out my branch if you have time and let me know how well or bad it
  behaves. I'll fix any issues that might pop up if you report them.


-- 
Ruben Vermeersch (rubenv)
http://www.savanne.be/