Re: Jinja2 syntax for conditionals (was: Re: Project options and other format enhancements (and dropping "variants"))

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Mon, 2017-09-18 at 16:11 +0100, Sam Thursfield wrote:
> On 18/09/17 12:35, Sam Thursfield wrote:
> > What I'm suggesting is that we avoid creating a new DSL for expressions 
> > in the first place, we try and reuse an existing one instead. Ansible's 
> > use of Jinja proves that this is possible. I'm not entirely sure how it 
> > works though -- if I get time I will try and make a proof of concept 
> > expression parser and we can see if it can beat your 218 lines of code :-)
>
> Attached is a proof of concept that demonstrates what I'm talking about. 
> Also pasted here: https://pastebin.com/hmk0TPQj
>
> It allows expressions that follow the Jinja2 expression syntax, as 
> documented at: http://jinja.pocoo.org/docs/2.9/templates/#expressions

Looking into this a bit deeper now...

> This supports arithmetic expressions (with infix syntax :-), compound 
> conditionals like 'foo or (bar and baz)', string comparisons (case 
> sensitive), list handling such as 'if "feature" in feature_list', all 
> with a very Python-like syntax.
>
> Some examples:
>
>    9 + 5 == 14    (evaluates to True)
>    "foo" in ["foo", "bar", "baz"]   (evaluates to True)
>
> You can pass a dict of key->values into the evaluation function which 
> become usable as variables within the condition expression. Booleans, 
> integers, strings and lists are supported (and possibly more). For 
> example if we pass "myvariable" set to the string "foo", we can use it 
> like this:
>
>    myvariable   (evaluates to True)
>    myvariable == "boris"   (evaluates to False)
>    myvariable[0] == "f"    (evaluates to True)
>
> etc.
>
> The whole thing is only 150 lines, and could be less if we didn't care 
> about the "is defined" / "is undefined" operators (which require some 
> special case hackery).

Right we dont really need that I think, considering that we already
mandate that the project define what these variables are before their
values can ever be controlled by a user (so the proper response for a
reference to something undefined is to simply error out anyway).

> To make it work we just embed the condition expression into a template 
> and then render it through jinja2 to get a result. Of course it feels a 
> bit dirty, but Ansible has been doing this since 2012 to implement 
> 'when' conditions in playbooks and it seems to be working well enough 
> for their tens of thousands of users.

So jinja2 seems to be a pretty huge thing, even includes it's own
execution sandbox (no idea why this is, or if it's a real sandbox or
some kind of python venv or what)...

And the way it is used in your example is to have the jinja2 templating
engine resolve a template in memory, and then evaluate the resulting
generated text to return a boolean value from.


This does seem vastly complex and roundabout in terms of codepaths
reached; for the simple things we need... and then what we get out of
it is... a rigid syntax or set of existing operators that cannot be
easily extended ?


So, technically I dont like the jinja2 approach for a few reasons,
mostly that it's pulling in a fairly huge templating engine and using
it for a fraction of its value (the dependency itself looks like a
liability), and also that it threatens difficulty in the situation that
we need to extend/change it.

That said the jury is still out on the general direction of this; you
seem to be after the exact opposite of what I'm looking for and I'm
trying to understand / balance the reasoning behind this:

  Do we want something fully specified and very cute, with a rigid
  non-extensible syntax that other programmers may recognize because
  they might have used that syntax before ?

Or

  Do we only want a data serialization format that is a bit more
  practical than YAML is for the purpose of serializing optionally
  quoted strings and numbers ?

  The exact same thing can be done with JSON or YAML, it just happens
  to be practical to use S expressions for these one liners.


I tend to like the latter, or, I can say that the latter makes me feel
a lot safer from a maintenance perspective; but it would be good if
other interested parties weighed in here too.

Also, when you say:

  "What I'm suggesting is that we avoid creating a new DSL for
   expressions in the first place"

I'm not sure that this qualifies; i.e. if parsing some serialized data
to be interpreted as nested conditional expressions qualifies as
creating a new DSL, certainly the BuildStream format itself, even
though expressed through YAML, is also it's own DSL.



While looking for alternative serialization formats for expressions, I
did come across this which is interesting:

   http://readable.sourceforge.net/

Which is a thing called "sweet expressions" and basically is just a
slightly more exotic parser around S expressions, making them appear to
be more like fancy programming languages look like.

I wonder what other alternatives we might be out there, both in terms
of simple serialization formats like S expressions, and in terms of
full blown syntaxes that look more like programming languages...
certainly there should be some middle ground here - something simple,
safely extensible if we need, and more esthetically pleasing than S
expressions ? 

Cheers,
    -Tristan