[BuildStream] 1.4 release timeline, targets and breaking changes

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi all,


Release Timeline
~~~~~~~~~~~~~~~~
In the gathering, we have agreed to move our release cycle up a bit,
targeting an eventual December / June 6-month release cycle.

The rationale behind this is that we have regular events which tend to
occur after these dates, and we would prefer if our gatherings usually
occurred at the beginning and mid-cycle; rather than directly near the
end of a cycle.

For the 1.4 release, Agustín and I have boiled down some dates:

  * Feature freeze plan: week 46, Tuesday 13th November.

    On this date, the list of targets below should be agreed upon,
    however the features need not all be implemented by then.

  * Feature/API Freeze: week 1, Wednesday 3rd January

    After this point, no new features or API changes land on master.

    Note: this considers that people will not be working in the
          last weeks of December

  * RC1: week 2, Tuesday 8th Janueary.

  * RC2: week 3, Tuesday 15th January.

  * Release day: week 4, Tuesday 22nd January.


As such, we can keep in mind that the list below is not technically
frozen until November 13th, we might add or remove items from the list
below depending on what we can achieve by then.

In any case, whatever cannot be landed by the actual feature freeze is
punted to 1.6.


Target Features
~~~~~~~~~~~~~~~
Here is a summary of the targets we had agreed on for the 1.4 release
in the BuildStream gathering in Manchester, along with some 
notes on what changes are technically breaking changes, and some
criticism on what is actually achievable in the remaining time frame.

* Debug remotely failed builds - #539

  This allows using the cached build tree and dependency artifacts for
  the reconstruction of the build environment which failed, and allows
  one to shell into it.

  STATUS: This landed yesterday by Jonathan Maw, with some minor loose
          ends to tie up:
          https://gitlab.com/BuildStream/buildstream/issues/740
          https://gitlab.com/BuildStream/buildstream/issues/494#note_113202563

  BREAKAGE: This changes `bst shell --sysroot` semantics to now work with
            a simple sysroot directory instead of our build directory format,
            which no longer litters the user's disk space.

            Overall this is a desirable/positive breakage.

* Download build trees on demand - #494

  This postpones downloading of the build tree portion of the artifact
  to the last minute by default, causing only commands which require
  (or optionally require) a build tree to download the build tree.

  STATUS: In progress by Tom Pollard

* Remote Execution loose ends

  At the gathering, Jürg, Jim and myself had a meeting discussing
  some architectural points about how remote execution fits into
  BuildStream in general, considering some existing use cases.

  A few things which fell out of this:

  - We need to separate the configuration of the remote execution
    CAS from the BuildStream artifact cache CAS, as it is an existing
    use case to push to multiple artifact caches, and it is not
    guaranteed that a project's artifact cache is hosted by the same
    CAS as the remote execution service used to build it (although
    sharing the CAS is usually a good recommendation).

    It probably makes sense to finish splitting out BuildStream's
    internal CasCache from the ArtifactCache first, or as a part
    of the same activity.

    STATUS: Todo (As I understand, Jürg has already made some progress
            on splitting up the internal CAS/ArtifactCache).

  - Execution environment requirements

    Now that we are adding remote execution, it is important to hammer
    out the basic API for a project/element to request an explicit
    execution environment in it's SandboxConfig (or perhaps remain
    neutral for some elements which never execute anything in a
    sandbox)

    This essentially means that an element needs to be able to require
    the OS and machine architecture for the execution environment.

    Since we currently only really support Linux, the OS only needs to
    be implemented as API, and will avoid breaking changes in the
    future.

    STATUS: Todo

  - Sandbox API for command batching - #625

    For sets of commands which run in a sandbox where one failing
    command fails the batch, and no output needs to be inspected
    by the Element plugin in between these commands; batches should
    be delivered to the sandbox.

    This optimizes remote execution payloads.

    STATUS: In progress by Jürg Billeter

* Allow usage of `git describe` - #487

  This is to fix a regression which was introduced in order to deem
  it acceptable to cache build trees.

  STATUS: In proposal phase by Valentin David

* Direct CAS-to-CAS directory import - #574

  Optimization for remote execution

  STATUS: In progress by Jim MacArthur

* Workspace UX reworking - #695

  This was discussed at length on the mailing list, the tail end
  of the discussion starts here:

  https://mail.gnome.org/archives/buildstream-list/2018-September/msg00036.html

  STATUS: In progress by Will Salmon

  BREAKAGE: A fair amount of breakage in the workspace related commands

* bst artifact subgroup

  This was discussed at length on the mailing list, and has not come
  to a conclusion (I am supposed to post a summary to pick this back
  up).

  The thread starts here:

    https://mail.gnome.org/archives/buildstream-list/2018-July/msg00051.html

  This is too large of a scope to hope to land in 1.4, however some
  of the features are easy enough to implement and we expect a
  minimal implementation of:

    o bst artifact log (display a build log of an artifact)
    o bst artifact checkout

  For the checkout command, it cannot be born with a recursive
  checkout as this requires reconstruction of the build graph from
  artifact metadata, and we agreed this would be out of scope for 1.4.

  STATUS: Todo

* bst source-checkout - #672

  The `bst source-bundle` command is to be deprecated and become a
  feature of `bst source-checkout`.

  The `bst checkout` command is intended to grow a `--build` option
  for convenient combination of checking out dependency artifacts
  along with build target source code.

  STATUS: In progress by Chandan Singh

  BREAKAGE: This removes a command, which is an important command to
            retain support for in `bst source-checkout`, but we
            strongly believe is not currently in use, impact minor.

* Remove default implementation of "strip-commands" - #304

  The strip-commands are intensely OS/Platform specific, and as such
  do not belong in BuildStream defaults.

  STATUS: Todo

  BREAKAGE: Currently known projects already provide their own set
            of overrides for strip-commands, so the impact of this
            breakage is considered minor.

* Remove MAKEFLAGS & V environment settings from the "manual" element

  Originally we did not have a bare "make" element, which was later
  added for convenience. As the "manual" element plugin is the base,
  raw BuildElement, it should not have a preference for make.

  STATUS: Todo

  BREAKAGE: We will have to do some babysitting of existing projects
            which assume this works, manual elements which invoke
            make should be replaced with the "make" element.

* Separate core plugins into a new repository

  It was discussed and agreed that as a matter of avoiding showing
  preference to certain plugins which are maintained in the core
  BuildStream repository, we should split out the majority of plugins
  into a separate repository.

  STATUS: Todo

  BREAKAGE: Possibly major - depending on the implementation details,
            this will probably mean that every BuildStream project
            needs to be updated to explicitly require a new plugin
            package and state the used plugins.

  WARNING:

    While I have expressed reservations against this due to the
    breakages it causes to existing projects, we have all agreed that
    it would be for the best if this was restructured.

    I think it is worth exploring the possibility of having an implicit
    set of core plugins in a separate repository, such that only the
    BuildStream installation procedure is changed and BuildStream
    projects need not break for the sake of this.

    Most importantly though, I don't think it is realistic that we can
    safely land this in 1.4 - because the implied work around this
    seems too much to fit into our time frame (especially considering
    the CI aspects):

      o We need to choose which plugins remain core (e.g. import,
        filter, compose, script, use internal BuildStream APIs and
        really are "special").

      o We need to create a new package for this

      o We need to finalize and create a public facing API for
        the testing framework in buildstream/tests/testutils,
        so that this separate package can run CI

      o We need to figure out how to setup CI for both BuildStream
        to test itself against a separately evolving set of plugins,
        and also how to test the plugins against a separately evolving
        BuildStream core.

    For this reason, I think we should abandon this activity for the
    scope of 1.4 and pick it up at the beginning of the 1.6 cycle, this
    would allow us to focus more on things we can realistically land in
    1.4.

Cheers,
    -Tristan