Re: [Tracker] Request for review, API allowing external software like MTP daemons to get metadata as correct sparql insert

[Martyn Russell <martyn lanedo com>]

On 08/01/13 11:56, Philip Van Hoof wrote:
> On Tue, 2013-01-08 at 11:40 +0000, Martyn Russell wrote:
> > > I guess putting INSERT queries upfront the metadata one needs to be
> > > possible as well as getting a separate query for in case the MTP daemon
> > > wants to do multiple passes. ie. First protocol based metadata insert,
> > > which needs the parent URNs at that point and which doesn't yet need the
> > > file content to be available, then extraction based metadata insert; for
> > > which the file content must be available.
> >
> > Not really. If that insert fails, then you're not guaranteeing the
> > parent is inserted, which we do at the moment.
>
> Why would the insert fail other than something being a bug that must be
> fixed?

In the past, inserts have failed because the extracted metadata is not 
in line with the ontology of the day OR because the database is fscked. 
So it's usually a corner case anyway. Sometimes it's a limitation or 
restraint we have in the DB, like unique nie:url cases - but that's more 
about the code being wrong than something we can't control.

> > Separate queries for the parent are really the way to go here. Once
> > that's done you have to handle failures and retries before the child can
> > be added, etc. This is why the miner-fs has a lot of code in this area
> > to handle this stuff. Ideally, it would be good to make use of that code
> > here so we don't have to fix issues we've already fixed in the miner-fs.
> >
> > > And what this creates must also be compatible with what miner-fs will
> > > otherwise create.
> >
> > Yea I agree.
> >
> > > The idea here is that while a (large) file is being transferred,
> > > protocol based metadata is already in the Nepomuk store available for
> > > applications that are interested in tracker:available-false 'to be
> > > deployed' data. ie. a DivX or DVD being transferred and a movie
> > > application saying: "DVD with title XYZ with actors ABC and written by
> > > DEF is being transferred to your phone (20% completed)". And After that
> > > an extraction to get extra metadata that isn't included in the protocol
> > > based metadata (and all that without needing miner-fs).
> >
> > We did actually have support for this in the past with inotify and half
> > downloaded files, but Nokia decided not to "notify" about half
> > downloaded files after a while. I think this was an area which changed a
> > few times due to requirements. In the end performance was one of the
> > reasons (due to spamming of updates with everything else going on at the
> > same time).
>
> Right. Nokia's requirements might not be everybody else's requirements
> though.

Yea, indeed.

> And with this miner-fs isn't involved. The MTP daemon has better control
> than inotify over the file-transfer to know when to update and when not
> to update.

Sure. Though Nokia IIRC wanted updates to things like nfo:fileSize and 
done so we didn't spam too much but had updates every n seconds, etc.

> BTW. the '20% completed' does not belong in Nepomuk and needs to be
> communicated differently (it's application state information).

Yea, I thought it might.

--
Regards,
Martyn

Founder and CEO of Lanedo GmbH.