GNOME.org
 

Re: Project options and other format enhancements (and dropping "variants")

Hi Sander,

On Fri, 2017-09-15 at 12:11 +0000, Sander Striker wrote:

Hi,

On Thu, Sep 14, 2017 at 10:40 PM Tristan Van Berkom <tristan.vanberko
m codethink co uk> wrote:

On Thu, 2017-09-14 at 20:59 +0100, Angelos Evripiotis wrote:

Hi!


Hi Angelos !

Thanks for your feedback...



This plan looks like it nicely addresses some concerns that I

had, which you

listed:

    * Combinatorial explosion of variants needed to express

multiple options,

      e.g.

with{,out}_poll__with{,out}_sqlite__with{,out}_readline__with{,out}
_gdbm

      etc.

    * Global selection of variant options from the CLI, e.g.

'enable all

      asserts everywhere', 'build everything without

optimisation', etc.


I don't think it addresses these ones I also have:

(1) Building particular elements for debugging

i.e. while keeping the rest optimised. In the options world, I

think this means

changing some options selectively from the command-line. Perhaps

something like

this would be workable: 'bst build app1.bst --option

mylib.bst:debug=true'.

Enabling or disabling options for a specific element only is
tricky;
because it's also very important to be able to express option
conditionals in project.conf (because you want to make sweeping
statements which optionally apply to the whole project as well).

That said, the S-Expression parser is very extensible, so it would
be
conceivable to write conditions which say, evaluate to true if a
given
string is found in a list; like say:

  (?):
    condition: (contains debug_elements "mylib.bst")
    value:
      random-key: The value I would apply if mylib is on the debug
list


My biggest concerns with the S expressions proposal is that it has
the potential of being too flexible.  And that it can become too hard
to quickly grasp what a .bst describes.  I liked the clear
declarative syntax of variants and archs.  They are easy to
understand.  Seeing '??' blocks does affect legibility.  I understand
this is a trade-off to become more flexible, but am wondering if we
are overshooting.


Before jumping into details... how flexible must things be and are we
overshooting.

There are some of the cases we've been talking about, like enabling a
debug option for a given build or maintaining builds of 2 versions of a
given ABI in the same project.

Something I would also love to have though is a better ways of tuning a
compiler for a specific board; as currently we only have a single arch
string to test where we just lump everything together. Probably I can
see myself wanting something like:

  (contains feature_list "vfp3")

This proposal doesnt really provide *enough* flexibility to have a
nicely defined and conveniently extensible database of machine names
from which we derive kernel configurations, compiler tunings and qemu
launch commands (these three aspects really go nicely together).




Still, I see your point: making something possible also makes it
possible to abuse, nobody wants to read a spaghetti state machine
riddled with boolean variable branch statements.

I would not have envisioned very intense nesting situations which
become impossible to follow but I can certainly think of situations
where I would find it convenient to just add a nested exception to an
already conditional yaml fragment.

I'm not sure which parts of the proposal specifically you think is
overshooting, but I feel like not having a specific anchor in the
format for expressing them but instead letting them be expressed
anywhere may have the most impact on legibility.

I did since change '??' for (?) because I now know that yaml will eat
that up unquoted, but that doesnt dramatically change things :)

Perhaps a good compromise here is to:
  o Dedicate an 'options' toplevel element (and project) attribute for
    declaring any conditional
  o Perhaps continue to allow nesting of conditionals, but only
    within the toplevel 'options' conditionals.


One of the reasons I originally thought of making (?) something that
can appear almost anywhere, is because it just fits more nicely with
the the new proposed (<), (>) and (=) tokens for expressing how to
apply arrays/lists, which I think is an important thing to address, but
nobody has commented on...


The S expressions in general on the other hand I much rather keep; I
see this purely as a convenience to avoid the user having to write 3 or
4 lines of yaml when you could just use a simple expression.

I dont think there is any escape to this either; as soon as you have
have orthogonal competing options, people are going to need compound
'AND' / 'OR' expressions - if they dont have tools to express the
condition they want; we'll get more nesting and redundant conditionals
to workaround shortcomings in the format.




(2) Allowing elements to specify what they need from their

dependencies


This is so that we can easily tailor to the needs of each

pipeline within a

project. This worked wonderfully with variants, I think the

problem was that

each variant was mutually exclusive.

Might we be able to use options in 'depends' declarations in a

similar way?

e.g.:

    app1.bst:
        ...
        depends:
            - filename: foo.bst
            options: flying-ponies
        ...

    app2.bst:
        ...
        depends:
            - filename: foo.bst
            options: dancing-badgers
        ...


Yes, this is really what I like so much about variants.

The problems I have with moving forward for a full blown variant
solution with competing orthogonal variants, are mostly that:

  o It's going to be very, very tricky to implement.

    We already have an imperfect algorithm for resolving variants
when
    there can only be one variant per element. The solution which
works
    exactly to spec takes several minutes to solve (so already we
need
    work to get it right with a proper constraint solving algorithm
or
    engine, which still wont work in linear time but should be
usable).

  o While it may be easy enough to explain the rules of constraint
    resolution to the user, it will often be difficult for the user
    to easily predict what variant of what element will be chosen.


This second point, Jürg pointed out is already true when variants
are
limited to one per element; because there are some complex cases
where
a variant will be chosen implicitly, simply because it's the only
variant which agrees with an orthogonal element in the graph which
does
have some explicit variant chosen.

The rules seem to be very clear and easy to reason about when there
is
only one variant per element, but I can see this getting
frustrating to
deal with for a user in a system where competing variants could
exist.



Assuming that we can supply both flying ponies and dancing

badgers then this

would be ok. If we can't then I imagine we can use the '!!'

operator to

complain about this in 'foo.bst'.

Perhaps there could be namespacing to separate global options and

element

options, to avoid collision and confusion. I haven't thought that

through yet.


One use-case I'm considering for this is dealing with breaking

changes across

versions of a library. If you imagine a project with 1000's of

flatpak-like

things being generated from BuildStream, from separate pipelines,

it could be

tricky to get all of them to be ok with a breaking change in the

same commit.


Potentially we could use a 'version' option on an element to

allow elements

that depend on the library to require the older or newer version.

If two

versions are requested in a pipeline, we can generate an error

thanks to the

'!!' operator.


Right so with the proposed approach, one could theoretically at
least
make an assertion that the correct version of an API is chosen from
an
element that depends on that API, that element need only have
knowledge
of the name of the option which enables the API.

So one element may build differently depending on an option, and
another element might raise an assertion if the chosen option
doesnt
work for that element, meaning we've at least expressed that a
given
element was incompatible for a given option.

However, what we dont have is the element declaring the value of an
option on an element it depends on; depending on how the element
itself
was configured; this all leads back down the variant path where an
agreement between elements must be reached.


What do you think?


I'm torn, of course I wish variants was going to work perfectly. On
the
other hand I have to concede that the implicit nature of how things
resolved was already a little bit complex for the user to digest,
and
increasing that complexity by making them orthogonal (both to
implement
and to use) seems to be unwise.


I think we should consider whether we need automatic resolution or
whether we need compatibility validation.  That is, if I construct a
pipeline I need to be explicit about the variants (or the default
applies).  If that results in [potentially] conflicting variants, we
would error out.  Similar to what you are eluding to above, with the
additional caveat that an element can assert that the element it
depends on has a certain option enabled (you might just phrase that
as, is of a certain variant).  You could also say an element is able
to explicitly declare compatible variants, and incompatible variants
are a cause for error.


Do we need to have one element decide how it's dependencies are
configured (or have any say in it) ?

Do we need to have one element assert that it's dependencies are
configured a certain way ?

Well, if an element can list the compatible variants of a dependency,
but this is not used to make any automatic resolution; then these must
be decided elsewhere ?

I think that so long as we're not automatically resolving these, even
trying to have options declared on a per element basis instead of just
at the project level is just not worth the trouble.

If the elements cannot work it out amongst themselves, I don't really
want to have to specify the values of various options that individual
project elements invent.

Regarding assertions, there is no reason why a given element cannot
make an assertion to the same effect but based on a project wide option
instead of an element option. It *does* however mean that the project
has some knowledge of the elements in it (as it may declare options
that are specific to a given element).


I'm also curious if this could be punted. Looking at this right
now, is
there a way forward where perhaps some automatic constraint
resolution
could be an added feature to options at a later time, without
breaking
things ?


I don't think we can punt the concept of architectures and variants
if that is what you are saying?  I would consider it key enough to be
part of a 1.0.


No, I rather wonder if there is a way we can:

  A.) Introduce project options, obsoleting the "arches" and
      "variants", and solving the practical problems of today.

  B.) Later, perhaps we can extend the existing options syntaxes
      to support variant like features.



I do agree that this is a big topic that is potentially too meaty to
resolve in a month.


Maybe there is a practical way forward without closing the door
entirely on the variants approach.


I cherish that thought as well...


So that's basically it; variants turned out to have issues, and we have
things that can and need to be solved now - can we have what we need
now and add something that we want later.

If we go forward with project level options now, without any constraint
resolution, then in the very worst case, BuildStream could end up with
a dual standard of specifying options.

The most likely case is that IF we end up doing some automatic
constraint resolutions, it will look like there are project wide static
options and the element specific options are a bit fancy and get
constraint resolution (might even be desirable to have both, if the
variants approach with orthogonal element options ends up looking like
a usable presentable API at all).


Asides from the gritty details of the conditional expressions and
formats, would you agree that implementing project wide options and
conditionals on project wide variables only for now, will give us
something that:

  A.) Will solve our problems today, cause we need conditionals
      to build GNOME and we need a stable 1.0 pronto

  B.) Does not close the door on constraint resolution approaches

  C.) Does not really become an undesirable "wart" if ever constraint
      resolution and element specific options exist one day, but
      rather compliments it well

Cheers,
    -Tristan
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]