Re: [Tracker] [systemd-devel] How to use cgroups for Tracker?

[Philip Van Hoof <philip codeminded be>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 21/10/2014 13:21, Lennart Poettering wrote:

> Well, looking at that bug it appears to me that this is caused
> because you try to use inotify for something it shouldn't be used
> for: to recursively watch an entire directory subtree. If you fake
> recursive fs watching by recursively adding all subdirs to the
> inotify watch then of course you eat a lot of CPU.
>
> The appropriate fix for this is to not make use of inotify this
> way, which might mean fixing the kernel to provide recursive
> subscription to fs events for unpriviliged processes. Sorry if
> that's disappointing, but no amount of cgriups can work around
> that.
>
> Don't try to work around limitations of kernel APIs by
> implementing inherently not scalabale algorithms in userspace. I
> mean, you implemented something that scales O(n) with n the numbers
> of dirs. That's what you need to fix, there's no way around that.
> Just looking for magic wands in cgroups and scheduling facilities
> to make an algorithm that fundamentally scales badly acceptable is
> not going to work.

The problem with allowing unprivileged processes is that a misbehaving
process will cause the kernel to buffer events for it, forcing the
kernel to dynamically allocate memory. Not something kernel or inotify
developers will like a lot.

I guess the kernel could reduce the buffer by not storing the
null-terminated name of the inotify_event, and reconstructing it
on-demand at the read() moment .. but that still means buffering.

I guess that's why we have /proc/sys/fs/inotify/max_queued_events

FSEvents 'solves' this by having a well behaved single process
journalling it to disk, and then providing an API for other consumers
to query the journal.

We could develop a privileged well behaving such process that does
exactly that. We need writable storage in $HOME/.cache/tracker anyway,
tracker-miner-fs could register itself to it to get such logs written
in the user's .cache location.

Maybe that's something for systemd to provide :-)? I wonder how well
that will burn in the plasma-hot flamewars surrounding poor systemd.

/me hides

Kind regards,

Philip


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (MingW32)

iQEcBAEBAgAGBQJUSDVJAAoJEEP2NSGEz4aDPsYH/1y6F1N67RyvBd50QEBHh7fH
m4AezEi4a+nFZVxFW3iXfLyp0Df0kkXSRwZAGgAOTVXbbrvi8aOmuFYSIsAqUuaU
9m98Dh6gTM1Zusa79sWLT1Ktp8bJ9Pa+dNenjXU2cizwkhuqYd58P4SoWAjbUDaA
quQVMrPD50Jyjf6zfCe2IeSUhG0v0leazs6o2XxmKSSNs/EBXf32w17GZd8dh3FX
8QcCxtTz+Lz+LJ92fwYEPPVh094hu0X79pB8st3ES0b+iZLaoDPrPRiOJn2kAETx
S+4V0F5/bQf4lQ8eTRHn7UDjOLLvqZ4PgsBAwSskVjykfCgCu9N+qEBlg5Ho6d4=
=HIJP
-----END PGP SIGNATURE-----