Re: GrlNetWc cache?

["Juan A. Suarez Romero" <jasuarez igalia com>]

On Fri, 2014-02-21 at 19:54 +0100, Bastien Nocera wrote:
> Great, thank you. I'd be happy to implement it if you can tell me what
> you want as an API.
>
> > > I could see myself requiring a persistent cache for certain requests
> > > (checking the "latest" items, but only if it's more than a certain age).
> >
> > Yep. But take in account that GrlNet will not cache per sé, but use
> > libsoup cache feature. So at the end, it will be libsoup the one doing
> > the cache.

I was thinking on something really simple.

Persistent caches would be created with:

grl_net_wc_persistent_new (gchar *cache_id);


This would create a GrlNet and load the content from the cache with that
identifier; if the cache doesn't exist it will be created.

For most cases, the sources would be using their own source_id as the
identifier.

Also, to avoid collisions of two applications using Grilo at the same
time, the cache would be stored/loaded in:

 ~/.cache/grilo/<application_id>/<cache_id>/

I think application_id could be guessed automatically by GrlNet, so it
is not needed to pass it as parameter.

We could add new methods, like grl_net_wc_cache_flush() or
grl_net_wc_cache_empty(). In any case, I think we could add a first
version without them and add them later as they are needed.

> Right. I was wondering how libsoup does its caching, and whether we can
> avoid hitting the web server when we know the data won't have changed
> (because the web server doesn't tell us in the headers, but somewhere in
> the data for example).

Yeah, I've asked to one of the guys working on libsoup about how this
works, and there are several rules to find out if a content can be
cached or not. I don't remember exactly which ones. But yes, it depends
mostly on the server and the HTTP headers. For instance, if the server
tells something can't be cached, or cached for a few minutes when you
know it doesn't change at all for several hours, libsoup won't cache it
or cache for those minutes.

I don't know if we want to implement our own mechanism for caching. We
already have all the pieces for caching content (see GRL_NET_CAPTURE),
but main doubts is how to decide something has expired.


> > Unless I forget something, I think each source uses its own grlnet
> > object. For the case of persistance, I'll add in the documentation that
> > is not recommended at all use the same grlnet object at the same time.
>
> I was more concerned about whether multiple GrlNetWc with the same cache
> directory wouldn't overwrite each other's entries. The soup_cache_dump()
> and soup_cache_load() call make me think that it's a bad idea to use
> separate caches (and thus GrlNetWc) for a single cache directory.

As explained above, the idea is to have a different directory per source
and application, to avoid exactly this collisions.

> So, let me know what sort of API you want, and I can try and implement
> it or we can discuss further in the bug.

As said above, only the grl_net_wc_new_persistent() is the one I have in
mind. If you have any other need, we can add now or later.

        J.A.