Re: [Tracker] Unix socket based IPC, experiment for inserting larger amounts of metadata

[Philip Van Hoof <spam pvanhoof be>]

On Wed, 2009-06-03 at 11:25 +0200, Philip Van Hoof wrote:

<snip>

> it goes like this:
>
> > UPDATE {0000000000} {0000000033}\nINSERT { <test0> a nfo:Document }
> > UPDATE {0000000001} {0000000033}\nINSERT { <test1> a nfo:Document }
> < OK:0000000000:{0000000004}:none
> > UPDATE {0000000002} {0000000033}\nINSERT { <test2> a nfo:Document }
> < OK:0000000001:{0000000004}:none
> < OK:0000000002:{0000000004}:none
> > COMMIT {0000000003} {0000000006}\nCOMMIT
> < OK:0000000003:{0000000004}:none

<snip>

> To nonetheless give you a figure (because the difference is impressive):
>
> A test that JÃrg did with the current DBusMessage format, we achieved
> transferring 12,000 statements in about 1.3 seconds. We indeed already
> grouped the statements so that the DBusMessages would be about 4k in
> size each. Don't worry, we too know about this in D-Bus world.
>
> With my test I achieved 10,000 statements in about 0.135 seconds and
> 100,000 statements in 1.6 seconds. Without grouping of statements. A
> Unix Socket is also page-based, so grouping would make this even faster.

I forgot to mention the test that I used to measure the raw throughput:

http://git.gnome.org/cgit/tracker/tree/tests/estress/socket_stress_test.c?h=tracker-store-ipc

Using this test it should be possible for you to reproduce above numbers
(on an average desktop or laptop).

The client API will await the actual INSERT or COMMIT on sqlite before
the callback takes place. Measuring this you wouldn't measure throughput
anymore, but you'd measure actual storage.

Former is obviously going to yield different numbers than latter.

The time for actual storage depends on how busy you are keeping the
database with queries. Queries have priority over stores (apps that need
information are indeed getting absolute priority over apps that store
information). We also yield batches to serve queries upfront.

Meaning that you making a query will always get priority over stores.

This is why the stress-test doesn't await the OKs nor ERs on the socket.

You can safely disconnect your socket, though. It'll implicitly commit
your data. Internally it's a queue.

Hence the wording "throughput" and not "storage performance".


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be