Re: XML-based sources (Metadata Factory plugin)

[Bastien Nocera <hadess hadess net>]

On Thu, 2012-06-21 at 12:11 +0200, Juan A. Suarez Romero wrote:
> On Wed, 2012-06-20 at 20:15 +0100, Bastien Nocera wrote:
> > The problem I have with it is that the syntax, due to the fact that it's
> > using XML, is very complicated, probably based on XPath and XSLT. I
> > actually found it more complicated than writing the C code. Now I can
> > only imagine how complicated if one tried to implement a source for a
> > service that uses json (which I see you've done, I certainly wouldn't be
> > able to juggle those 2 so easily).
> 
> 
> I can agree that XML syntax is not the most beautiful one, specially
> when it comes to write it.
> 
> Also, the source I exposed there (BBC news) is one of the most
> complicated, mainly because the regular expressions you need to handle
> to get the URL.
> 
> On the other side, you can take a look at the Nascar Videos source:
> 
> https://gitorious.org/~jasuarez/grilo/jasuarez-grilo-plugins/blobs/media-factory/src/media/media-factory/sources/grl-nascar.xml
> 
> Definitely it is quite simple to write and understand than the BBC one.

I tried to write a source, and I couldn't find a way to debug it. It's
impossible to insert debug of intermediate steps.

> But the point is that sooner or later, you need to deal exactly with
> everything defined in the XML source.

That's not really the point. The point is that it's hard to figure out
how to write it when you know what should be written, it's hard to debug
when something goes wrong (or when a service changes), and it's hard to
adapt sources from other systems (such as MythTV plugins) to it.

> So summing up, this plugin is designed to work mainly with XML; JSON is
> a second class citizen here.
> 
> But everything is defined in that spec is the core of the source, and if
> you write it in C, sooner or later you need to deal with it.

In C, I can add a printf or attach a debugger to see what value a
variable has in the code, or whether that code got hit. It's impossible
with your source.

> > Personally, I would rather have seen:
> > - remove the need for plugins to implement the threading/async'ness, and
> > do it all in the calling code.

> * Add threads. As you well point-out in bgo#667557, this can be done
> with GAsync callbacks/results. The point with threads is that if it is
> not done very very carefully, it could be a can of worms.

Yes, you need to use threads, and just for the fact that you mentioned
(it's hard to do threads right), it should be done by the framework, not
by the plugins.

> > - add helper functions for oft-used features. This might be XML and
> Json
> > helpers for example.
> 
> Regarding the helpers for XML/JSON, I don't have a clue of what
> functions we should provide. Probably you are talking about something
> similar to GrlNet, where we provide a high-level and simple functions to
> retrieve content from Internet.
> 
> So far I've been using libxml2 and json-glib. The former is not very
> friendly, specially for the GNOME world. I think someone is writing a
> wrapper to integrate it with glib. Aren't this libxml2-glib and
> json-glib enough?

Not certain, I'm guessing this would need somebody with an overview of
all the plugins saying what can be factored out.

> > - implement a plugin that uses libpeas or lua to export that
> > functionality to a scripting language
> > 
> 
> Uhm... some time ago I was also exploring the use of libpeas to
> implement the plugins. So it would allow to implement it in any other
> language.
> 
> But I didn't go too much far because one of the key points in using
> libpeas is the introspection, and our API is not very friendly in this
> sense.
> 
> So I stopped it while we do not fix this problem.

I personally think that lua is a good match for this sort of problems,
as it has a very small API of its own, which means we can lock-down the
plugins, and offer a minimal API. It also stops won't trample on
applications, and it's a good match for the work done in quvi.

<snip>
> And there is still another approach: having D-Bus based plugins. This
> was commented in different places, and it is listed in our TODO. The
> main idea was to isolate unstable plugins in their own process, so a
> crash would not imply a break of the full application. But another
> benefit is that the source can be written in any language: it only needs
> to provide the API through D-Bus.

I don't think that using D-Bus will solve Grilo's problems though. The 2
biggest problems are the need for every plugin to handle their own
threading, and the lack of interpreted and easily debuggable scripting.

I want GNOME hackers to feel that adding support for their favourite
video site is an evening's work in front of the TV, not a chore as it
currently is.

Cheers