Re: Project options and other format enhancements (and dropping "variants")

[Sam Thursfield <sam thursfield codethink co uk>]

Hi Tristan,

In general I agree with your premises, and I think the proposal is 
workable. I don't have anything better to propose.
>     Option Declaration
>     ------------------
>     A project declares which options are valid for the
>     project in the project.conf.
>
>     These options should have some metadata which can be used
>     to declare the defaults, assert valid values of the options,
>     and also a description string which the CLI can use to communicate
>     the meaning of project options to buildstream users (not all users
>     building a project wrote the project.conf).

Are you expecting to support only enum values, or freeform strings and 
integers too?

This sounds quite similar to Meson's option system[1], which is probably 
a good sign.

1. http://mesonbuild.com/Build-options.html


>     Format Enhancements
>     -------------------
>     I would propose we add some special tokens which can be used at any level
>     of a buildstream element.bst file, or also in some specific parts of the
>     project.conf (since project.conf is declaring options, we cannot conditionalize
>     that part)
>
>     Below are my ideas for the '>>', '<<', '==', '??' and '!!' operators.

On one hand this is ugly as sin because YAML describes itself as a "data 
serialization standard", where self-modification shouldn't really be a 
thing. On the other hand, it already contains two special operators ('&' 
and '*', which are effectively "copy" and "paste") and in the interests 
of being "human friendly" there is definitely justification for allowing 
more syntax sugar.

I think the symbols you've chosen are pretty good, I like that they 
don't look anything like normal text so just glancing at a .bst file 
should set off alarm bells of "this isn't just a list of dictionaries, 
there's extra processing being done" and hopefully the reader will head
for the documentation.


It's worth a quick review of existing solutions in this area. There are 
some processing/filtering tools:

* jq -- https://stedolan.github.io/jq/ -- CLI tool for running filters 
on JSON-serialized data, which supports all kinds of manipulations, 
path-based access, and conditionals
* xlst -- https://www.w3.org/standards/xml/transformation -- similar but 
for XML and is about as horrific to use as you would expect

And also formats that support variants / self-modification:

* Ansible Playbooks -- 
https://docs.ansible.com/ansible/latest/playbooks.html -- mixes Jinja2 
templates with YAML, to provide variable substitution and conditionals 
using Jinja's expression syntax
* jsonnet -- http://jsonnet.org/ -- extends JSON to add an expression 
syntax based on JavaScript (although specified independently)

There are probably more things that I'm missing.


>     The '??' expression format
>     ~~~~~~~~~~~~~~~~~~~~~~~~~~
>     So at first I was thinking what this would look like as a pure
>     YAML format, but it looks like it will be way too verbose for
>     expressing simple comparisons.
>
>     Example:
>
>
>         variables:
>         '??':
>           condition:
>             kind: ifeq
>             args:
>               option: debug
>               value: on
>           then:
>             conf-extra: --enable-debug
>           else:
>             conf-extra: --disable-debug

This could be abbreviated to be on one line:

    variables:
    '??':
      condition: { kind: ifeq, args: { option: debug, value: on } }
      then: "conf-extra: --enable-debug"
      else: "conf-extra: --disable-debug"

It's not super readable though.

>     Later I thought maybe we do our own parsing of strings like
>     `ifeq(option, value)`, but that also becomes a little unwieldy, hard
>     to maintain and extend to support compound expressions.
>
>     So what I'm leaning towards now is to create a simple expression
>     format based on S-Expressions, this way the same expression above would
>     just look like:
>
>
>         '??':
>           condition: (ifeq "debug" "on")
>           then:
>             conf-extra: --enable-debug
>           else:
>             conf-extra: --disable-debug
>
>
>     This is especially nice once you want to do anything a bit more
>     complex, the following would be a lot more verbose to express if
>     it were in YAML:
>
>
>         '??':
>           condition: |
>
>             (and (ifeq "logging" "off") (ifeq "debug" "on"))
>
>           then:
>             ... value ...
>           else:
>             ... value ...
>
>
>     The S-Expressions are fairly easy to parse and there is a python
>     library for that (http://sexpdata.readthedocs.io/en/latest/).

I have to admit I was surprised to see Greenspun's 10th rule borne out 
here :-)

This could be workable, but the choice of S-Expressions is risky. I 
think those familiar with Lisp will be unhappy that our implementation 
doesn't match up with their preferred dialect of Lisp, and those 
unfamiliar with Lisp will think "what on earth am I looking at".


One option is to reuse Jinja2 expressions, as they are quite 
Python-like, and are already used in Ansible. The jinja2 library looks 
flexible enough that we could set up an execution environment and just 
evaluate Jinja expressions[2] to produce values, rather than using the 
full templating functionality.

2. http://jinja.pocoo.org/docs/2.9/templates/#expressions

This could get us something like this:

           condition: |
              %{logging} == "off" and %{debug} == "on"
           then:
             ... value ...
           else:
             ... value ...

I would prefer if we could use True and False for booleans rather than 
the strings "off" and "on". That way we could get to this:

           condition: %{logging} and %{debug}
           then:
             ... value ...
           else:
             ... value ...



Some rambling to finish ... does anyone remember BuildJ [3]? It was a 
project to replace Autotools/CMake/etc with a declarative JSON/YAML 
format. Started out promisingly, but faltered before completely figuring 
out conditionals. Hopes for replacing Autotools and CMake then 
gravitated towards Meson, which is pretty much on track for success at 
this point. Rather than using YAML, Meson defines a Python-esque DSL for 
build instructions which is deliberately not Turing complete (no loops 
or functions) and can be parsed in a few 1000 lines of Python.

I used to be disappointed that Meson didn't use a "declarative" approach 
but I actually find it fine to work with now. I like YAML because it's 
always possible to reliably parse it, unlike a Turing-complete 
programming language such as Shell or BitBake. But Meson's language also 
ticks that box. It is also apparently designed to be re-writable so that 
IDEs can make changes to hand-written meson.build files, although I'm 
yet to see how well that works. If it does, I'd be interested in what 
BuildStream would look like if it abandoned YAML for a similar 
Python-like DSL. Of course, this is not at all BuildStream 1.0 
territory, but something to think on :-)



[3] https://wiki.gnome.org/Attic/BuilDj

Sam

--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575