Re: What Problem does BuildStream Resolve ?

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Thu, 2018-02-01 at 15:16 +0000, Paul Sherwood wrote:
> Hi Tristan
> thanks for the comprehensive answer. Just one comment to clarify 
> below...
>
> On 2018-02-01 15:00, Tristan Van Berkom wrote:
> <snip>
> >  o construction is deterministic
> >    - for a given set of inputs, we always get the same output
> >
> >    This depends on reproducibility of source code as well, we
> >    have recently run tests which show that gnome builds are mostly
> >    reproducible, and are working on automating tests which buildstream
> >    consumers can use to test the reproducibility of their sources
> >    (similar to reprotest)
> >
> >    - all dependencies are known and controlled
> >
> >    Not sure the context of this, we have source aliasing and have
> >    the framework in place to automate mirroring; mirroring of
> >    sources to ensure you always have what you need is recommended,
> >    but not in scope for automating as a part of buildstream.
>
> We've seen many situations where build outputs are impacted by changes 
> to dependency versions that aren't properly controlled. Simplest example 
> would be where the project assumes it can get $dependency direct from 
> the internet, then upstream moves or disappears entirely.

Yes this is a commonly known occurrence.

> As I understand it buildstream does allow us to specify all dependencies 
> and take them from controlled locations; but obviously if we **choose** 
> to rely on an input which is outside our control, it's at our own risk.

Yes.

A few things to elaborate on this:

  o URLs to where sources can be downloaded from should normally
    be specified with an alias defined in the project.conf

    i.e. instead of "http://farm.com/git/pony.git";, a Source
    in an element.bst file should specify "farm:pony.git" and define
    the "farm" alias separately in project.conf

  o By design and as a matter of policy (policy needed because
    external developers can implement plugins), the unexpanded
    aliased url is considered in cache keys, not the expanded ones.

    This guarantees that you can change the origin from where
    you download external sources without effecting cache keys
    (i.e. we assert what it is, but leave the "where it comes from"
    out of cache key calculations).

  o Because the aliases are already in place; and because BuildStream
    is already capable of mirroring everything one wants to obtain
    from the internet for a local build, and is also capable of
    automatically tracking the latest of all of a project's sources, it
    would be very straight forward to implement a mirroring assistant.

    This is a simple enough task to do, has been discussed before,
    but not yet on the roadmap.

If we were to implement a helper of sorts to maintain a given mirror
(publicly accessible one, or an internal server with great connectivity
to your build farm), we would probably want something like the
following:

  o To do this on a multiple-project basis (existing projects hosted 
    by this mirror define what gets mirrored)

  o To always retain the external origins of upstream sources (projects
    should not have to lose this information)

  o Ability to easily setup a new mirror either from scratch, or from
    another existing mirror.

  o Ability to easily configure your project to try a list of mirrors

Cheers,
    -Tristan