Re: Photo import problems using python

[Damon Lynch <damonlynch gmail com>]

On 02/01/2010 04:42 AM, Alexander Larsson wrote:
> On Sat, 2010-01-30 at 13:56 -0600, Damon Lynch wrote:
>    
> > After modifying my code to use gio commands like copy() and
> > enumerate_children(), and hopefully thereby bypassing any potential
> > FUSE issues, I've discovered that my problems still persist with my
> > test hardware (a high end camera, the Canon 5D Mk II). So it seems the
> > problem is not FUSE. These problems include the GUI locking up
> > momentarily, and the download speed being approximately 1/5th that of
> > downloading from a python test file (without a GUI) or from Nautilus.
> > The very same code, when downloading from a memory card reader, has no
> > problems.
> >
> >      
> You could try the gvfs-copy (will use the gio ops, not fuse) command to
> copy a file and see what kind of performance it gets. If thats still
> slow then this seems to not be related to your code but rather some gvfs
> internal issue.
>    

That's interesting. The problem occurs regardless of whether 
gio.File.copy() is called by my code, or whether the python standard 
library executes the copy via FUSE.  The problem only seems to occur 
when a UI is involved. Below I've pasted in some test code which works 
with pretty much exactly the same performance as Nautilus, but this code 
has no UI. I include it to give an indication of the gvfs commands I'm 
using.

As an aside, I have just found a possibly unrelated bug when more than 
one device is plugged in at the same time. To reproduce:

1. plugin in a memory card into a memory card reader
2. plugin in a camera with a memory card in it
3. browse the contents of both devices, and they're identical, because 
the contents of the memory card are shown as being in the camera, 
instead of the camera's content. The memory card's content is displayed 
ok (they're not swapped).

The order is important. If the camera is plugged in first, and it's 
contents browsed, it seems to work ok.

If helpful I'll file a bug / check for duplicates.

> How exactly are you doing the copy? Are you doing a sync or an async
> copy? If sync, are you doing it in a thread? If you're doing a sync copy
> on the main thread (i.e. the ui thread), then the kind of UI lockups
> you're experiencing is expected.
>
>    

I am using a sync copy in a thread. All UI updates are handled within 
the main program thread. All downloads are handled in their own thread 
(it can download from more than one device at once). All downloader 
threads send their UI updates to the main thread using a queue.

If it's relevant this is the code in question: 
http://bazaar.launchpad.net/~dlynch3/rapid/trunk/annotate/head%3A/rapid/rapid.py

(Of course I'm hoping that the problem can be identified without anyone 
needing to examine this code, but I include this link in the off-chance 
it might be helpful).

***

This very basic test code works just great:

#!/usr/bin/python
import gio
import os
from threading import Thread

RAW_FILE_EXTENSIONS = ['arw', 'dcr', 'cr2', 'crw',  'dng', 'mef', 'mos', 
'mrw',
                        'nef', 'orf', 'pef', 'raf', 'raw', 'sr2']

NON_RAW_IMAGE_FILE_EXTENSIONS = ['jpg', 'jpe', 'jpeg', 'tif', 'tiff']

def isImage(fileName):
    ext = os.path.splitext(fileName)[1].lower()[1:]
    return (ext in RAW_FILE_EXTENSIONS) or (ext in 
NON_RAW_IMAGE_FILE_EXTENSIONS)

def gio_scan(path, images, imageSizeSum):
    children = 
path.enumerate_children('standard::name,standard::type,standard::size,time::modified')
    for child in children:
        if child.get_file_type() == gio.FILE_TYPE_DIRECTORY:
            imageSizeSum = gio_scan(path.get_child(child.get_name()), 
images, imageSizeSum)
        elif child.get_file_type() == gio.FILE_TYPE_REGULAR:
            name = child.get_name()
            if isImage(name):
                size = child.get_size()
                images.append((name, path.get_path(), size, 
child.get_modification_time()),)
                imageSizeSum += size
    return imageSizeSum

def progress_callback(self, v):
    pass


class CopyPhotos(Thread):
    def run(self):
        mounts = gio.volume_monitor_get().get_mounts()
        for m in mounts:
            if 'Canon' in m.get_name():

                images = []
                imageSizeSum = 0

                imageSizeSum = gio_scan(m.get_root(), images, imageSizeSum)

                for image, path, size, mtime in images:
                    print image
                    g_dest = 
gio.File(path=os.path.join('/home/damon/tmp', image))
                    g_src = gio.File(path=os.path.join(path, image))
                    if not g_src.copy(g_dest, progress_callback, 
cancellable=gio.Cancellable()):
                        print "copy of %s failed!!!!!" % image

                    else:
                        print "Copied: %s" % image


c = CopyPhotos()
c.run()



--

http://www.damonlynch.net