Re: Mixing XS with Gtk2::TreeStore

["Emmanuel Rodriguez" <emmanuel rodriguez gmail com>]

On Sat, Dec 27, 2008 at 4:14 PM, muppet <scott asofyet org> wrote:
> On Dec 27, 2008, at 6:41 AM, Emmanuel Rodriguez wrote:
>
>
> Stop.  Custom XS should be a last resort, as it ratchets up the difficulty
> of maintaining and deploying your application, and is often unnecessary.
>  Let's do some sanity checks first:

Let me present more deeply the problem. I have a Perl application that
displays an XML document with both source code syntax highlighting and
a DOM tree view. For now the problem I have is showing the DOM tree in
the tree view. For the moment, I'm using a TreeStore because there are
many examples showing how to work with a TreeStore. Populating the
TreeStore with the DOM tree is my current bottleneck. The application
is based on XML::LibXML which in turn is using libxml2. So the there's
already a quite decent data model: XML::LibXML::Document.

The slowness is due to the nature of XML::LibXML where each node and
field access is performed through XS. A small document of 400K can
very easily return over 170_000 DOM nodes. Walking such a tree in Perl
is slow (about 0.5sec) when combining it with the operations of a
TreeStore it takes 1.2 sec. Now imagine loading a 20 Megs document...

I have profiled the application and found some bottlenecks that I have
now fixed, but the application is still slow. Now I'm filling the
TreeStore with the code listed below. This sample code takes 1.2sec in
Perl, while the same code rewritten in C runs in 0.140sec which is
definitely faster.

sub create_model {
        my $model = Gtk2::TreeStore->new(
                'Glib::Scalar', # An instance to XML::LibXML::Element
                'Glib::String', # The name of the Element
        );
}

sub populate {
        my ($treeview, $document, $element) = @_;

        # It's slightly faster to remove the model from the view and to insert it back
        my $model = $treeview->get_model();
        $treeview->set_model(undef);
        $model->clear();
        populate_internal($model, $document, $element, undef, 0);
        $treeview->set_model($model);
}

sub populate_internal {
        my ($model, $document, $node, $parent_iter, $position) = @_;

        my @values = (
                $NODE_ICON => 'gtk-directory',
                $NODE_NAME => $document->get_prefixed_name($node),
                $NODE_DATA => $node,
        );

        # Find out if an attribute is used as an ID
        foreach my $attribute ($node->attributes) {

                # Keep only the attribute
                if ($attribute->isId) {
                
                        # The current node has an ID                    
                        @values = (
                                $NODE_ID_NAME  => $document->get_prefixed_name($attribute),
                                $NODE_ID_VALUE => $attribute->value,
                        );
                
                        # There's only one ID per element
                        last;
                }
        }

        # Set the main data of the node
        my $iter = $model->insert_with_values($parent_iter, $position, @values);
        
        # Add the children to the DOM model
        my $i = 0;
        foreach my $child ($node->childNodes) {
                if (isa_dom_element($child)) {
                        populate_internal($model, $document, $child, $iter, $i++);
                }
        }
}


> b.  Are you mirroring a data structure?  That is, do you have a TreeStore
> that you keep in sync with some other perl data structure?  If so, you are
> wasting a lot of memory and time.  Either use the TreeStore as your native
> data store, or write a custom TreeModel that adapts the TreeModel interface
> to your native data store.

Yes, I'm having an existing model (XML::LibXML::Document). Writing a
custom model crossed my mind as it could same some memory. The problem
is that I will still need to walk the DOM tree each time that a node
is clicked. Although this shouldn't be perceived by user as being so
slow. I will give it a try.

Is there an example on how to implement a custom TreeModel?


> Your stack trace indicates that you *are* adding Perl data types to your
> tree.  Did you create your store with a column type of 'Glib::Scalar'?

Yes the TreeStore has slot for passing the XML::LibXML::Node that's
been displayed in the TreeView. I didn't realized that I had to pass
an SV* there. I was trying to pass a simple C pointer and expected it
to be magically transformed to a Perl type!

Thanks for the help.

-- 
Emmanuel Rodriguez