Re: [Tracker] [systemd-devel] How to use cgroups for Tracker?

[Martyn Russell <martyn lanedo com>]

On 23/10/14 10:34, Lennart Poettering wrote:
> On Thu, 23.10.14 08:15, Aleksander Morgado (aleksander aleksander es) wrote:
>
> > That's a good one indeed; coalescing events in that way in the kernel
> > looks quite a sane approach. Still, one single process in userspace
> > doing all the control of what changed when (like FSEvents does) may
> > actually behave much nicer, as other processes could ask for all
> > changes coalesced since a specific timestamp (e.g. since that process
> > was run). Not thinking in Tracker here, think of a program which runs
> > sporadically, but when it runs it wants to know what changed since the
> > last time it was run (e.g. a backup app).
>
> This is really a file system feature, and btrfs already provides that.

Sadly, we can't rely on one file system. It's not uncommon for people to 
have sshfs, ext{3|4}, nfs and FAT based file systems mounted with their 
content. With that in mind, we can't rely on the feature of one file 
system only - but it can assist us, naturally.

I would, however, like to see something like liboctopus in place which 
simply takes care of "all of this mess" for Tracker. In the older days, 
we really just had FAN and then inotify, now it seems there are more 
combinations of file systems/APIs. If we did use liboctopus, it could 
adapt according to variations in:

- Kernel APIs available (be it inotify, FAM, FANotify, KQueue, Fen,
  Win32, etc). To some extent, GLib handles a large number of these for
  us.

- Modseq/journal/FSEvents approachs, either as a new initiative
  in kernel space or a temporary one in user space until there is some
  FSEvents type of implementation in the kernel.

- Handle all logic and complexity that comes with coalescing events.

- Any privileged work that has to be done so that other libraries in
  the system in user space can depend on it. Bastien has a good link on
  his "wishlist" on kernel things he would like to see and there, the
  article about how OS X does this privileged work and allows only a
  select few (like Spotlight) to make use of that data.

    Wish list:
    https://wiki.gnome.org/BastienNocera/KernelWishlist

    Specific article:
    http://arstechnica.com/apple/2007/10/mac-os-x-10-5/7/


> > Anyway, please remember that being privileged isn't the only reason
> > why Tracker can't use fanotify. It's API being fd-based, it works on
> > existing open files only; e.g. it won't notify file deletes or move
> > events, among other things. If we want some recursirve monitoring
> > approach with all CREATE/UPDATE/DELETE/MOVE events, something new
> > needs to be implemented, or inotify somehow improved to handle that.
>
> I don't think fanotify's interface couldn't be fixed to also generate
> useful events for deletion/moving.

I don't really understand why it was developed as a half complete 
solution if I am honest. It's not as if there are no examples to follow 
out there (FSEvents) and it's not as if we didn't say what our user 
space requirements were at the time.

I know it's a hard problem to solve, but if it's not solved with the 
proposed solutions, the kernel developers shouldn't really be accepting 
them.

--
Regards,
Martyn