Re: [BuildStream] Changing element files extension requirements

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Phillip,

On Tue, 2018-11-20 at 10:40 +0000, Phillip Smyth wrote:
> Hey Tristan,
>
> Thanks for replying, I just had a couple of follow up questions.
>
> On Wed, Nov 14, 2018 at 06:16:09PM +0900, Tristan Van Berkom wrote:
> > Hi Phillip,
> >
> > Thanks for bringing this up !
> >
> > On Mon, 2018-11-12 at 16:54 +0000, Phillip Smyth via BuildStream-list wrote:
> > > Hi all,
> > >
> > > Following on from a discussion on #buildstream on IRC, there have been 
> > > suggestions of making file extension like `*.bst` mandatory.
> > >
> > > This would currently be of use with the introduction of build all and 
> > > show all commands, as currently it is difficult to know if a file is an 
> > > element or some other kind of file,
> >
> > Yes, it will be unavoidable; the only alternative I could see is to
> > alternatively have a sort of mime-type expression in the beginning of
> > the file content, but this is more expensive both in terms of
> > processing and in terms of the burden of writing .bst files.
>
> Where would be a good place to put this check, and how should we enforce it?
> A simple option seems to be to put a check in the buildstream/_frontend/cli.py
> Which would check the end of the `elements` variable for ".bst"
> and return an error + message if it's not found.

I think we need this in the _loader/loader.py module, as elements are
loaded indirectly as dependencies of their targets.

Probably it would be kinder to issue a warning instead of an error (and
follow the pattern of allowing it to be configured as a fatal warning),
this warning would issue a summary of all encountered elements which do
not have the expected filename extension, explaining that these
inappropriately named elements will not be discoverable by commands
which operate on all of the project elements in one go.

> > >  and the current proposed solution 
> > > isn't fool-proof as it assumes that only element files will exist inside 
> > > of the element-path directory, which may not be the case, and in some 
> > > instances, the element-path directory does not exist.
> >
> > I don't see any issue with this.
> >
> > If elements are outside of the element-path, then they are not
> > accessible in any way, and should not be included in any `--all`
> > commands, they just dont exist.
> >
> > If the element-path directory is specified in project.conf and does not
> > exist, then it is a project that has no elements at all.
>
> In regards to the build all functionality,
> how would you suggest we deal with projects that use junctions?
> As these would seem to cause issues.

How so ?

Every project directory is separate, and a project is only expected to
have elements within it's own element path. Junctions themselves are
also elements and can only be discovered within the element path of
their owning project.

There is a separate question regarding junctions:

   When performing any kind of `--all` command which operates on all
   of the project elements, should we consider all of the elements in
   every subproject, or only the subproject elements which are
   reachable by elements discovered in the toplevel project ?

I rather think that `--all` means all of the elements in "this project"
plus any elements which are reachable in sub projects, but not
inclusive of elements in subprojects which are not at all reachable by
the toplevel project. If "all elements of all subprojects" is desirable
for some reason (is it ?) then it should probably be opt in behavior
with a separate option.

Note again that this question has nothing to do with elements which
exist outside of their owning project's element-path, those elements
are simply impossible and don't exist.

Cheers,
    -Tristan