Re: [Tracker] What to do with libtracker-common?

[Martyn Russell <martyn lanedo com>]

On 07/10/14 13:14, Carlos Garnacho wrote:
> Hey,
>
> On Mon, Sep 29, 2014 at 11:55 AM, Martyn Russell <martyn lanedo com
> <mailto:martyn lanedo com>> wrote:
>
> <snip>

Hi all,

Here is just a small update of the work done now.

One thing that wasn't on the original plans was tracker-ontologies.h, 
which is a set of definitions we duplicate a lot and actually are quite 
important. I've moved these to libtracker-sparql since libtracker-data 
uses them a lot and so do extractors.

I also cleaned them up. Carlos, you might want to look over these before 
they become *public*. I've been wondering if they should be public or 
not. See commit:

https://git.gnome.org/browse/tracker/commit/?h=libtracker-common-cleanup&id=07c02a34bccb65d68e33b0d96115300b2540a6cd

>     -> tracker-ioprio.h
>
>         Set the I/O priority to be lower than normal to avoid disk
>         clobbering. Used by tracker-extract and the miners.
>
>         Could actually be in libtracker-miner.
>
>
> I would think this is too specific API, I wouldn't expect any external
> TrackerMiner to thrash the filesystem as massively as tracker-miner-fs does.
>
> If tracker-extract and tracker-miner-fs are kept in the same repo, that
> could be private API between both though.

This is still to do.

> Exactly, I think it deserves the same end...
>
>     -> tracker-locale-gconfdbus.h
>     -> tracker-locale.h
>     -> tracker-meego.h
>
>         Used to notify and keep track of local changes which is needed to
>         re-create database collations because sorting can vary by locale
>         (among other things).
>
>         The -gconfdbus file is an implementation, which we might even be
>         able to remove by now? *Is anyone using still using GConf?*
>
>         The -meego file is use to translate and get the locale on Meego.
>         *Is anyone still using Meego?* Would like to remove this.
>
>
> Please let's remove this outdated stuff, I consider on-the-fly locale
> change handling something unachievable today, no matter how much we tape
> over it

See Aleksander's reply and my response to see the links here, but this 
has all been removed now.

>     -> tracker-os-dependant.h
>
>         The tracker_spawn*() API here is only used by libtracker-data and 2
>         extractors, mplayer and ps (where we use gunzip, which is quite a
>         bad way to do this).
>
>
> I really don't know why the PS extractor isn't using poppler... I would

So, I checked. The PS extractor uses the spawn API as a fallback in case 
it can't use the normal method (which is to use poppler as you say Carlos.

> prefer us to stick to GStreamer and remove the mplayer module, it's
> practically unused in any downstream, after that it can be moved to
> libtracker-data, or just make it use g_spawn_async_with_pipes itself.

It's been removed now, I never use/test or get any bugs about it, so I 
think it's completely unused. This has been here since the early days 
with Jamie.

https://git.gnome.org/browse/tracker/commit/?h=libtracker-common-cleanup&id=0c747da27ea4257ee91cc27995caf1782b2110bf

>         The tracker_memory_setrlimits() is only used by tracker-extract and
>         could be moved there directly or put in libtracker-extract.
>
>
> Please let's just remove this, brings in enough grief.

It's done:

https://git.gnome.org/browse/tracker/commit/?h=libtracker-common-cleanup&id=3a7625fa5cfe795e5b35c3c32b609488ac8ce5f0

I will comment on the bug later, but we should possibly use cgroups 
too?!? At this point, this commit should help us avoid the SIGABRT bugs 
we keep getting.

>     Possible solutions
>     ------------------
>     1. We just make libtracker-common public, perhaps give it a better
>     name (like libtracker-core?) and be done with it. We can then use it
>     anywhere. Advantages: it's quick and easy, Disadvantages: API has to
>     be stable.
>
>     2. We split libtracker-common up and move code to more specific
>     areas, e.g. fundamental type functions (A) go into libtracker-data,
>     etc. Advantages: code is more logically placed, Disadvantages:
>     Linking to larger libraries for small API calls might not be so
>     attractive.
>
>     3. We create newer smaller libraries with logical parts A to D and.
>     Advantages: Smaller concise areas to group API, Disadvantages:
>     Slightly larger maintenance and distribution burden in the beginning.
>
>     4. We copy the code to modules they're needed in. Advantages: Less
>     libraries to link with, Disadvantages: potentially a larger
>     footprint with the same API copied around.
>
>
> 5. Move libtracker-common to a separate git module, set it up on the
> relevant repos through git submodule, compile it statically in these
> repos, and add a configure script to that submodule that fails and
> points people to the right place(s).

What's interesting is now what's left in libtracker-common:

$ git ls-files
.gitignore
COPYING.LIB
Makefile.am
libtracker-common.vapi
stop-words/Makefile.am
stop-words/stopwords.da
stop-words/stopwords.de
stop-words/stopwords.en
stop-words/stopwords.es
stop-words/stopwords.fi
stop-words/stopwords.fr
stop-words/stopwords.hu
stop-words/stopwords.it
stop-words/stopwords.nb
stop-words/stopwords.nl
stop-words/stopwords.pt
stop-words/stopwords.ru
stop-words/stopwords.sv
tracker-common.h
tracker-date-time.c
tracker-date-time.h
tracker-dbus.c
tracker-dbus.h
tracker-enum-types.c.template
tracker-enum-types.h.template
tracker-enums.h
tracker-file-utils.c
tracker-file-utils.h
tracker-ioprio.c
tracker-ioprio.h
tracker-language.c
tracker-language.h
tracker-locale.c
tracker-locale.h
tracker-log.c
tracker-log.h
tracker-sched.c
tracker-sched.h
tracker-storage.c
tracker-storage.h
tracker-type-utils.c
tracker-type-utils.h
tracker-utils.c
tracker-utils.h

There might be some pruning left we can do, but most of the major stuff 
is done now. The sched/ioprio stuff can likely go in libtracker-miner 
(but actually tracker-store is using that API I think), when it probably 
shouldn't (if the miners do already).

The two really hard ones to remove are the log and dbus modules.

Storage can be moved to libtracker-miner, writeback uses it but only to 
listen for mount point changes, which you can do with gio already.

Comments welcome!

--
Regards,
Martyn

Founder & Director @ Lanedo GmbH.
http://www.linkedin.com/in/martynrussell