Re: Import and duplicate images.

[Bill Moseley <moseley hank org>]

On Wed, Aug 27, 2008 at 02:00:36PM -0700, Michael Wayne Goodman wrote:
> > I also just looked and noticed in f-spot I have a number of images
> > duplicated.   Somehow they were imported more than once.
> 
> There is a patch for this: http://bugzilla.gnome.org/show_bug.cgi?id=169646
> The bug report is labeled as "NEW", but it has been around for years.
> Maybe it will be committed soon.

I'm confused by that.  I thought f-spot in the past did not import
photos that were already imported.  Am I mistaken?  It's been a while
since I used f-spot's import so maybe I'm remembering incorrectly.


> > Also, when I delete a photo in f-spot is there any record of it being
> > deleted so that it is not re-imported again?
> 
> This seems like something you'll have to manage yourself. How would
> f-spot know if you actually wanted to reimport the photo? (eg. maybe
> you overwrote the original with an edit, and want the original back).
> Rather, perhaps you will have to start clearing your memory card (at
> least those images you deleted from f-spot).

I guess what I'd do is have a table of recently deleted images (the
uri and a timestamp).  Then when importing if the image does not
already exist in the "photos" table and also does not exist in the
"recent_photos" table then go ahead and import.

Then periodically remove old rows -- or have the option to remove all
rows to re-import something that was deleted.

Yes, I should probably just delete from the memory card, but I often don't
want to do that so I have a backup of my images when I'm traveling.



> > At home I have a Perl script that I use to import my photos -- which
> > doesn't show thumbnails but takes a fraction of the time to import and
> > only brings in new images.  Then when I start up f-spot the thumbnails
> > are created.
> 
> Perhaps you could share?

The script is trivial (and a one-off hack) but I'll attach anyway.
It would be easy to add features (like command line options, but not
sure it's worth the time).

The script isn't very smart -- it just uses the EXIF date (or the file
mtime if no EXIF date) to place the images into directories based on
$year/$month/$day/ and then uses the original file name.  I suppose an
MD5 of the source image might be best name, but I use the image's
original file name.  Any hierarchy on the memory card is lost.

-- 
Bill Moseley
moseley hank org
Sent from my iMutt

#!/usr/local/bin/perl
use strict;
use warnings;

# Script to import photos from memory card, avoid dupicates and
# update f-spot's database.
# Questions: Bill Moseley <moseley hank org>

use Image::ExifTool qw(:Public);
use File::Find;
use File::Path;
use File::Temp 'tempdir';
use File::Copy;
use Data::Dump 'dump';
use File::MimeInfo::Magic;
use Time::Local;
use Class::DBI::Loader;

# $source is where the memory card is mounted
my $source    = '/media/disk';

# $dest_root is the dir where photos are stored
my $dest_root = '/media/store/photos';

# f-spot db file, of course.
my $db_file   = '/home/moseley/.gnome2/f-spot/photos.db';

# place to put backup of the f-spot database and hard-link copy
# of the photos before running script.
$ENV{TMPDIR} = '/media/store/snapshots';


my $total_source;       # count of total source files
my $existing_count;     # count of files already imported from card
my @files_to_move;      # list of source paths to copy
my %day_count;          # to summarize imports by day

# What EXIF fields to look for, and in which order, for the date
my @date_fields = qw/
    CreateDate
    DateTimeOriginal
    ModifyDate
/;



# Process all files on the source media
find( \&find_images, $source );


# Show summary of files ready to import
my $count = scalar @files_to_move;
unless ( $count ) {
    print "nothing to import\n";
    exit;
}



print <<EOF;
Total source files: $total_source
Total to import:    $count
Existing skipped:   $existing_count

Counts by folder:
EOF

dump \%day_count;

print "Continue? : ";
my $in = <STDIN>;
print "\n";

exit unless $in =~ /y/i;

my $tmpdir = tempdir();
print "\nbacking up to $tmpdir\n";

# Backup the database and images
system( "cp $db_file $tmpdir/photos.db" );
system( "cp -al $dest_root $tmpdir" );


# Auto-load the database schema
my $loader = Class::DBI::Loader->new(
    dsn               => "dbi:SQLite:dbname=$db_file",
    options           => { RaiseError => 1, AutoCommit => 1 },
    left_base_classes => qw/Class::DBI::Sweet/,                  # or arrayref
    relationships     => 1,
);

# Load all columns on SELECT
for ( $loader->classes ) {
    $_->columns( Essential => map {"$_"} $_->columns );
}


# Do the job
import_images();




# Find what images need to be imported
sub find_images {
    return unless -f;   # only look at regular files
    return if /^\./;    # skip hidden


    my $file = $_;      # get name


    $total_source++;

    print "$total_source\n" unless $total_source % 50;


    my $mtime;


    # First try and extract out the time from the EXIF info.


    # Create a new Image::ExifTool object
    my $exifTool = new Image::ExifTool;

    # Extract meta information from an image
    $exifTool->ExtractInfo( $file );

    my $dates = $exifTool->GetInfo( @date_fields );


    for my $fld ( @date_fields ) {
        my $datetime = $dates->{$fld} || next;

        my ( $date, $time ) = split /\s+/, $datetime;
        my ( $y, $m,   $d ) = split /:/, $date;
        my ( $h, $min, $s ) = split /:/, $time;

        next if $y < 1970;

        $mtime = timelocal( $s, $min, $h, $d, $m - 1, $y - 1900 );
        last;
    }

    $mtime ||= ( stat $file )[9];


    # Build destination directory and file name.

    my ( $y, $m, $d ) = ( localtime $mtime )[ 5, 4, 3 ];

    my $dir = sprintf( "$dest_root/%04d/%02d/%02d", $y + 1900, $m + 1, $d );

    my $dest_file = lc $file;

    my $dest = "$dir/$dest_file";


    # does this file already exist?
    if ( -f $dest ) {
        $existing_count++;
        return;
    }


    # Save for copy later.
    push @files_to_move, {
        source => $File::Find::name,
        dir    => $dir,
        file   => $dest_file,
        mtime  => $mtime,
    };

    $day_count{$dir}++;

    return;

} ## end sub find_images


sub import_images {

    # Create an import roll.
    my $roll = Rolls->create( {
            time => time,
    } );

    my $cnt;
    for my $image ( @files_to_move ) {

        $cnt++;
        print "imported $cnt\n" unless $cnt % 20;

        my $dir    = $image->{dir};
        my $file   = $image->{file};
        my $source = $image->{source};

        mkpath $dir unless -d $dir;
        my $dest = "$dir/$file";

        if ( -e $dest ) {
            die "Destination file $dest exists, but shouldn't\n";
        }



        copy( $source, $dest ) || die "Failed to copy $source to $dest : $!\n";


        # And insert photo into database.
        # "uri" is for newer f-spot
        # (directory => $dir,  name => $file for older)
        #
        my $photo = Photos->create( {
                time               => $image->{mtime},
                uri                => ( 'file://' . $dir . '/' . $file ),
                roll_id            => $roll,
                default_version_id => 1,
                description        => '',
        } );
    } ## end for my $image ( @files_to_move)

    print "imported $cnt\n";



} ## end sub import_images