Re: [Tracker] [systemd-devel] How to use cgroups for Tracker?

[Philip Van Hoof <philip codeminded be>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 23/10/2014 8:15, Aleksander Morgado wrote:
> On Thu, Oct 23, 2014 at 1:24 AM, Philip Van Hoof
> <philip codeminded be> wrote:

[cut]

> > > This is fixable, by enforcing a size limit on the queue. As
> > > the limit is hit the algorithm should coalesce queued events
> > > based on subtrees. For example, if one event for /foo/bar/buzz
> > > and one for /foo/bar/waldo is queued, and the queue hits its
> > > limit, both are replaced by an entry for /foo/bar which is
> > > marked with a new flag that some event was lost below this
> > > subtree. For clients this would then mean that when they
> > > receive this they must rescan that specific subtree again, but
> > > not the whole tree.
> > >
> > > It's a simple algorithm, the max number of entries stays
> > > fixed, but perfomance doesn't go completely horrible when the
> > > limit is reached.
> > >
> > > Such a scheme should be implemented in fanotify on the kernel 
> > > side.


[cut]

> > But yes. The coalesce solution in fanotify would be a good idea
> > to allow unprivileged processes. Probably better than what
> > FSEvents does.
>
> That's a good one indeed; coalescing events in that way in the
> kernel looks quite a sane approach. Still, one single process in
> userspace doing all the control of what changed when (like FSEvents
> does) may actually behave much nicer, as other processes could ask
> for all changes coalesced since a specific timestamp (e.g. since
> that process was run). Not thinking in Tracker here, think of a
> program which runs sporadically, but when it runs it wants to know
> what changed since the last time it was run (e.g. a backup app).

I'd design that API to be a lot like how tracker:modified and several
protocol's modseqs works: store a modification sequence in each new
record of the journal and return it after each query by a client.

Clients can then using the modification sequence ask for all changes
since that number.

However. This also means logging forever. So the same problems
tracker-store has when you don't disable its journal: ie. always
growing storage for it and privacy - the 'if I remove/rename/create a
file, I don't want its existence to be nevertheless logged
forever'-use-case. For example if the administrator takes away +rx
from a directory for a user, the user could with the journal still
know what used to be there. So we'd need a method for the admin to
purge that.

I think it's inevitable that such log files will need to be truncated,
rotated and/or also size limited at some point.

The coalesce-only solution doesn't have this problem.

> Anyway, please remember that being privileged isn't the only
> reason why Tracker can't use fanotify. It's API being fd-based, it
> works on existing open files only; e.g. it won't notify file
> deletes or move events, among other things. If we want some
> recursirve monitoring approach with all CREATE/UPDATE/DELETE/MOVE
> events, something new needs to be implemented, or inotify somehow
> improved to handle that.


Correct.

Kind regards,

Philip



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (MingW32)

iQEcBAEBAgAGBQJUSLIdAAoJEEP2NSGEz4aDrEgIAML+jfWgzRrnnTop7jRrLXdv
bRjibU37FaK45rauUxX395ye13hqUWQYUx9FVE4xh6y/yvJVFGCt1SfJFuGIu8+S
XLaud2qY145ePIEfakD4z3iX9BsAWfsWIcvGu6QIz7tpT8Nd6SM+tJYwRx9V1dli
vdspQh8Kf1Xg9DxUjmx2bnVYzV7KFM3dRgyuOGrxlHVjAGOH9HFAn1n0PJlO87Fu
8Hzr2vLgRU+zDnFY8jifavLKo4+a6/2UHeLbthhwRUUNBk9LZixblCs5wgAPJmb3
SZdIk5u+EqOYCAwWs6AgIEl/XQMbiC34DVS+rtckm0DNA8lmvzOTvvPar/ZkuzE=
=Ldnv
-----END PGP SIGNATURE-----