Re: Jinja2 syntax for conditionals

[Sam Thursfield <sam thursfield codethink co uk>]

Thanks for the response here.

To be clear I don't have any particular attachment to Jinja2, I just 
didn't want to propose "please use infix syntax that looks like Python" 
without some kind of implementation to back it up.

I did look for a standalone "Python-like DSL for conditional 
expressions" library and found nothing; I'm sure other projects 
implement such a thing (Meson does, for example) but none of them have 
gone to the extra effort of maintaining it as a separate shared library. 
(Which is perhaps a point in favour of also doing our own thing in 
BuildStream... although that still feels wrong to me).

On 19/09/17 04:30, Tristan Van Berkom wrote:
> > The whole thing is only 150 lines, and could be less if we didn't care
> > about the "is defined" / "is undefined" operators (which require some
> > special case hackery).
>
> Right we dont really need that I think, considering that we already
> mandate that the project define what these variables are before their
> values can ever be controlled by a user (so the proper response for a
> reference to something undefined is to simply error out anyway).

Yeah that makes sense on reflection. I just copied that code path across 
from Ansible anyway.

> > To make it work we just embed the condition expression into a template
> > and then render it through jinja2 to get a result. Of course it feels a
> > bit dirty, but Ansible has been doing this since 2012 to implement
> > 'when' conditions in playbooks and it seems to be working well enough
> > for their tens of thousands of users.
>
> So jinja2 seems to be a pretty huge thing, even includes it's own
> execution sandbox (no idea why this is, or if it's a real sandbox or
> some kind of python venv or what)...

It's 7K lines of Python, with 1 external dependency (MarkupSafe). The 
package is 2.4MB installed, while BuildStream is 2.8MB. So it's not a 
lightweight dep, but I don't think "it's too big" on its own can rule 
out this approach.

The "sandbox" is just a bunch of small mixin classes. I think it's there 
because you can pass arbitrary Python objects as variables into 
templates, and Python objects can do operator overloading so 
theoretically you might end up with 'a == b' formatting your hard drive 
because type(a).__eq__() does something horrible. That wouldn't affect 
us as we'd control everything we pass in.

So it doesn't do anything like BuildStream's bubblewrap sandbox or 
virtualenv, it's just intercepting functional calls that may have side 
effects.

> And the way it is used in your example is to have the jinja2 templating
> engine resolve a template in memory, and then evaluate the resulting
> generated text to return a boolean value from.
>
>
> This does seem vastly complex and roundabout in terms of codepaths
> reached; for the simple things we need... and then what we get out of
> it is... a rigid syntax or set of existing operators that cannot be
> easily extended ?

There's a certain ugliness to it indeed, but on the other hand these 
codepaths have probably hundreds of thousands of users already and are 
maintained by other people.

Extensibility is certainly needed. I don't think there should be any 
need to redefine what "+" or "==" mean beyond the existing 
Jinja2/Python-like interpretations of them, but we'll want to add helper 
functions. Jinja2 expressions can execute callables using the () 
operator, so let's say we want to implement a join_path() method, we 
could write that in Python and pass it in as a global to the template 
environment, and then write conditions like:

    full_path == join_path(prefix, libdir)


> That said the jury is still out on the general direction of this; you
> seem to be after the exact opposite of what I'm looking for and I'm
> trying to understand / balance the reasoning behind this:
>
>    Do we want something fully specified and very cute, with a rigid
>    non-extensible syntax that other programmers may recognize because
>    they might have used that syntax before ?
>
> Or
>
>    Do we only want a data serialization format that is a bit more
>    practical than YAML is for the purpose of serializing optionally
>    quoted strings and numbers ?
> >    The exact same thing can be done with JSON or YAML, it just happens
>    to be practical to use S expressions for these one liners.

I don't see this as the question at all. Either way we're adding logical 
expressions to what was previously a serialization format, and making 
something that is (a) a serialization format with syntax sugar for 
variants, or (b) something else.

Sure you can represent the syntax tree of the S-Expressions as a YAML 
data structure, the same is true of the syntax tree of the Jinja2 
expressions.

For me the questions are:

1) do we want to use S-Expressions, or a more Python-like syntax?

2) do we want to write and maintain the execution engine ourselves, or 
reuse something that exists?


> Also, when you say:
>
>    "What I'm suggesting is that we avoid creating a new DSL for
>     expressions in the first place"
>
> I'm not sure that this qualifies; i.e. if parsing some serialized data
> to be interpreted as nested conditional expressions qualifies as
> creating a new DSL, certainly the BuildStream format itself, even
> though expressed through YAML, is also it's own DSL.

There's no universally agreed place where we draw the line between 
"code" and "data". In my view, once we have self-modifying conditional 
expressions we're crossing line into "code", regardless of whether the 
syntax used is S-Expressions or Jinja2 expressions, or YAML. Where the 
line is doesn't really matter, my point is that (if we ignore 
implementation details) our proposals amount to the same thing.


> While looking for alternative serialization formats for expressions, I
> did come across this which is interesting:
>
>     http://readable.sourceforge.net/
>
> Which is a thing called "sweet expressions" and basically is just a
> slightly more exotic parser around S expressions, making them appear to
> be more like fancy programming languages look like.

That's interesting indeed.

Sam

--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575