Re: [BuildStream] API proposal - enable iterating through all elements

[Douglas Winship <douglas winship codethink co uk>]

On 27/01/2020 23:20, Chandan Singh wrote:
> Hi Douglas,
>
> Thanks for getting back.
>
> Some comments inline.
>
> On Mon, Jan 27, 2020 at 5:28 PM Douglas Winship via buildstream-list
> <buildstream-list gnome org> wrote:
> > [...] I'm looking for something that outputs "foo". [...]
> Sorry, terminology is confusing. I could've been clearer. I think you have rightly decided to go with an 
> example. So, let me try to explain what I meant.
>
> Continuing with the sample "foo:bar/baz.git", I was thinking that the first command to output source fields would give 
> the entire url as-is. And the second command for listing aliases could either print all aliases as key-value pairs, and/or get 
> the value for a given key. This would tell you whether or not "foo" is defined as an alias.
>
> ...
>
> So, while none of the two commands I suggested would output just "foo", I think this approach should still 
> work for what you are trying to do. Let me know what you think.
This would be fine. So long as the necessary information can be accessed 
in an unambiguous, machine-readable form, we'd be able to get what we 
need. I don't see why we couldn't have a command that just spits out 
"foo", but *if* we couldn't, then it's no big deal.
> > > So, even if `translate_url` method becomes a public static function, BuildStream still won't know which URLs 
> > > to translate. Of course, you as the author of this tool, may know which plugins you use and what do they call 
> > > their URL fields.
> > Interesting. If I've understood, you're saying that BuildStream defines a format for specifying the aliases, 
> > and it defines functions that can do the appropriate substitution if you give them a URL to translate, but 
> > BuildStream doesn't actually define any standard method for listing all the URLs that need translating?
> Precisely!

I've actually found a good example of that, just in the last couple of 
days: the 'cargo' plugin, which comes with a default URL written into 
the plugin code.

> > So my rephrased question for the mailing list is this:
> >
> > Can anyone see a good way to let users explore the nodes in an element definition, and run the same kind of 
> > arbitrary processing that you can _already_ include in a plugin, just without enforcing the overhead that's 
> > usually associated with 'bst build'?
> When you say "arbitrary processing", are you also expecting to access any BuildStream API methods, besides 
> getting the data?

Good question. I knew I shouldn't have said "arbitrary processing" like 
that, but I wasn't sure quite how to phrase it. The answer is no, I'm 
not suggesting we open up any other parts of the API. (Or rather, I 
leave that question for other people to discuss, who know more about the 
API than I do.)

What I meant was, I'm just proposing we somehow give programmers access 
to that data, (with proper substitution, composition etc), even if it's 
read-only access. When I said 'arbitrary processing', i meant that a 
developer should be able to get a _copy_ of whatever data they're 
accessing, and then use it as an input for whatever arbitrary processing 
they want to do. eg. making a manifest file, or processing the data with 
pandas, or printing it to PDF for some reason, or using it as a random 
seed for a haiku generator... whatever they feel like doing with it.

I'm not proposing (with this particular suggestion) that we open up any 
part of the API that can leave an effect on the project: nothing that 
would modify any project files, and nothing that would trigger any 
BuildStream functions except the ones needed to access the data. In fact 
getting the data without triggering buildstream functionality is pretty 
much the point, since I'm trying to access this data without causing 
BuildStream to build any elements.

> In other words, would it be sufficient to get just the parsed data (with proper substitution, composition 
> etc) about the element, its dependencies and the project itself? Or [...]
Yes, exactly.
> If it's the former and you just need that data, I think that can be solved entirely by extending the CLI. And this 
> would require merely a "bst show" to get the required output. For example [...]

This could work, and I'll support doing it if it's the quickest/easiest 
way to make the data available. But I'm much more interested in getting 
programmatic access, via the BuildStream python library.

I can't speak for everyone, but I'll certainly be writing my "arbitrary 
processing" in python. It seems like if the data is only available via 
command-line, then I'll have to import subprocess to send command-line 
arguments from my script, and then parse through the command line output 
to get my results (figuring out how to sanitize the input, processing 
escaped characters, etc etc.) It would be much neater if I could access 
the data with appropriately written python functions, which returned 
appropriately structured python objects.