GNOME.org
 

[BuildStream] BuildStream 2.0 planning

Dear BuildStream community,

    Last week a group of us convened to discuss BuildStream 2.0,
we thought it would be useful for us to get on the same page in
a series of working group sessions. We discussed various aspects
around BuildStream 2.0, most notably the scope of the 2.0 release.

Those who participated in our working group, which consisted
of two sessions of two hours each, include Benjamin Schubert,
Chandan Singh, Jürg Billeter, Sander Striker and myself, and I am
writing to you today to report on the results of our sessions.


Timeline and strategy
=====================
We will not release BuildStream 2.0 before it is ready and we are
not setting any specific target dates, however there is consensus
that we want to work towards BuildStream 2.0 now, and we are motivated
to get it ready as soon as we can.

Given our recently announced transition to the Apache Software Foundation,
it is going to be useful that we can demonstrate our ability to do releases.

In order to expedite the release, we focused on the scope of the
work which needs to be done in order to permit a release of 2.0.

The intent here is that we have a clear list of major blockers
to 2.0, such that we do not postpone the release on issues which are
not on that list. Conversely, we do not want to start working on large
initiatives which are tangential to 2.0, risking further delays of
the 2.0 release. As a general rule, we have used the impact on the
API surface (eg. CLI, Plugin API, service API) as the biggest factor
in deciding whether a change would be a blocker or not for 2.0.

Note that it is not the time to start micro management of every
single minor detail which needs to be in BuildStream 2.0, this
time will come when we are closer to nailing all of the major
topics.


Scope
=====
For each topic that we discussed, we were mostly able to agree
on whether a topic was going to block BuildStream 2.0 or not, in some
cases we've decided that we need to block on a clear proposal, so that
we can judge later on whether we need to block on it or not, and
in some cases we've agreed on a strategy for moving forward.

Here are the main talking points and results below.


Complimentary Services
----------------------

  * Remote Asset API migration[0][1]

    Retire the old artifact server for source and artifact caches,
    using remote asset API based cache instead.

    Blocker.

  * Replacement of artifact cache server.

    Blocker.

    Our preference is to not maintain an artifact server, and to recommend
    better, more scalable CAS services in the REAPI space for this purpose.

    In the event that there is not yet a good replacement for the artifact
    cache server provided by a third party project in the REAPI space
    by the time we are ready to release, then then we will keep bst-artifact-server
    as a thin layer around buildbox-casd.

  * Remote execution services

    Documentation blocker.

    We do not want to express any preference for a third party remote execution
    service, but we consider it a blocker that we provide links to documentation
    which makes such services easy to install and setup.


YAML Format changes
-------------------

  * Default dependencies per element kind[2][3]

    Not a blocker, very high on the wishlist.

  * Allow sources to access element variables[4]

    Blocker.

    For this item, we would like to remove Element.node_subst_vars() and
    related APIs, so that variables are unconditionally substituted in
    any values loaded from the configuration.

    This should of course work for Sources as well as Elements, using
    the parenting Element declaration for variable substitution context.

  * Resolve directives/conditionals in included files in the context of the
    project they were included from, rather than the including project context[5]

    There is already a working patch for this.

    Blocker: This must land with improved user facing documentation surrounding junctions.

  * Ensure errors/warnings instead of inadvertently overriding junction configurations[6]

    Blocker.

  * Support multiple elements in a single .bst file?

    This was discussed with regards to testing (test elements) a while ago[16]

    Blocker: A proposal in order to determine if this would be an API breaking change

  * Clear error reporting for BuildStream major version mismatches

    Plan:
    - The "format-version" will be changed to use "<major>.<minor>" annotations.
    - This will be renamed to "min-version", to express more clearly that this
      is a minimum version check, and not a choice of behavior or format to use.
    - The "min-version" will be a mandatory field in the BuildStream 2 project.conf
    - BuildStream 1 will be patched to recognize "min-version" and report a clear error
      about the format/tool mismatch
    - BuildStream 2 can likewise recognize the presence of "format-version" to report
      a similar error about the format/tool mismatch.

    Blocker.


REAPI Concerns
--------------
It was discussed here whether there are changes to the REAPI protocol
specification that we might be blocking on in order to release BuildStream 2.

There remains some API related to properties which is expected to land
very soon in the upstream API specification and we are not blocking on
anything else.

BuildStream 2 will use the v2 specification of the REAPI protocol, v2 of the
RE API is considered stable; it is not expected to break backward compatibility -
we have witnessed the remote-apis community to be mindful of this.


Sandboxing
----------

  * BuildBox only sandboxing solution[7][8]

    Blocker.

Asides from the above, sandbox capabilities and how they can
affect cache keys was discussed.

  - BuildBox provides 'capabilities' depending on host environment
  - A 'capability' can be considered a 'guarantee'
  - The YAML format can express the requirement of a 'capability',
    this requirement affects the cache keys.

An example of this is the `build-uid` and `build-gid` sandbox
configurations: if left unspecified, then the sandbox makes no
guarantee about which uid/gid is used for the process performing
a build.


Plugins
-------

  * Change the signatures of `Source.track` and `Source.fetch` to remove
    the need for `kwargs`

    Blocker.

  * Change Source API, to allow tracking, fetching and pushing using the
    Remote Asset API [19][20].

    Blocker.

  * Open up library API surface for using BuildStream from python[9].

    Currently this is not allowed, the python API is only plugin facing.

    This is something to consider separately from 2.0 concerns.

    Not a blocker.

  * How to handle "download only sources" to work with local files[10].

    There is a problem statement that we need a solution for obtaining sources
    which cannot be published and need to live either in the project as a local
    source, or in a project relative path/location.

    Blocker: We need a decision on how this is to be handled in order to
             decide if a solution would be a breaking change, this decision
             blocks 2.0.

  * Require virtual directory support for element plugins[11]

    Blocker.

  * Drop support for elements calling Sandbox.run() without batching.

    Currently the batching of commands in element plugins is explicit, however
    unbatched commands are suboptimal in a remote execution context.

    In some cases, the result of executed commands may inform the plugin
    as to what the following commands might be, so probably we need support
    for unbatched commands too.

    Related to this is the possibility of dropping the split between
    the prepare() method and the assemble() method.

    Blocker: As any of these will be breaking changes, an investigation
             into these changes blocks 2.0

  * Can we make a decision about which plugins remain in core, and which
    plugins go where otherwise[12][13] ?

    Consensus here is to defer to previous judgements at previous gatherings
    and not to rehash this all over again:
    - Some basic plugins, and plugins which require internal API, remain in core
    - Other plugins in core get migrated into domain specific repositories

    Further, it is agreed that core plugins for which we have API guarantees
    remain under the governance of BuildStream after being split into
    separate repositories (they can of course still be forked).

    Blocker: Plugins must be moved to their destination groupings/packages.

  * Make `GitSourceBase` and `DownloadableFileSource` public API from
    their respective plugin repositories

    Not a blocker.

  * Allow accessing plugins across junctions ?

    Wishlist, not a blocker.

    This will allow one to reuse plugins which are made available
    by a junctioned project, opening up the possibility of creating
    BuildStream projects which simply only provide plugins.

    This is intended to be an alternative to using git submodules
    as a means of using and sharing plugins safely, especially
    when their APIs are not stable, and also as a means of ensuring
    that projects do not require users to install packages separately
    on the host.

    Consensus: We can keep at least a simplified 'tar' plugin in core
               for the purpose of bootstrapping projects which need
               a way to get off the ground.

  * Clean error reporting for cases where people accidentally mix the wrong
    BuildStream core with the wrong plugins (bst 1 vs 2).

    Blocker.


CLI consistency related changes
-------------------------------
We have various changes already in the CLI and we have had many
patches stuck in review in the past[14][15].

We feel that if this continues, we will continue to have
fragmented opinions on what the CLI interfaces should be,
and as a result, we will not achieve the consistency we are
looking for.

Consensus: We need to appoint a small review committee for
           CLI changes in order to expedite the process.


Cache key stability
-------------------
It was clearly agreed that BuildStream 2 would feature stability
of cache keys in the long term.

This conversation resulted in identifying a couple of additional
blockers.

Blocker: A conversation took place while discussing `bst artifact` command
         subgroups where it was discovered that a project or element cannot be
         renamed without affecting some cache keys (see Sander's post here[17]
         along with the following conversation).

         This needs to be investigated and resolved in order to provide
         properly stable cache keys going forward.

Blocker: The usage of a source URL in an artifact cache key is questionable[18],
         originally we included this qualifier in the doubt that git commit
         shas (for example) are only a good enough qualifier to guarantee uniqueness
         of source code, but not good enough to guarantee uniqueness across
         every source code repository on planet earth.

         We are blocking on an investigation on how to improve this story
         such that we minimize what aspects of existing Source plugins
         get considered in Plugin.get_unique_key(), and whether we can
         use the digest for the Tree representing the source for this purpose.
         Qualifiers such as commit id's will then be source specific "qualifiers"
         much as described in the Remote Asset API.


Installation stories
--------------------

  * How do we propose to tackle installation of BuildStream 2 on
    systems where users also need to use BuildStream 1 ?

    We will not have a perfect parallel installation story for
    BuildStream 1 and BuildStream 2 installed on the host.

    Instead, we have agreed on the following plan:
    - Need good error reporting for version mismatches
    - Need easy to follow instructions to install bst1 & bst2 via python venv,
      installation instructions should be linked from version mismatch error reporting messages
    - In the distro installation story, we will recommend that BuildStream 2 replace BuildStream 1.

    Some concerns which were discussed are that, even if we tried to make these
    parallel installable, there are already conflicts in the dependencies required
    to run BuildStream 1 & 2 respectively within the same python environment.

    We expect however that the situation of using venv to use a version of
    BuildStream that is not supported on your distribution will be fairly
    temporary, and that projects will mostly migrate to BuildStream 2 in
    a sensible timeframe.

  * Easy bootstrapping/setup for external services (artifact cache, RE service)

    We do not want to express any preference for a third party CAS service,
    but we consider it a blocker that we provide links to documentation
    which makes such services easy to install and setup.

    Lets be wary of installation instructions in other projects in the REAPI
    space, and contribute to their installation and setup related documentation.

  * What stability guarantees / releases do we need for BuildBox components ?

    This was a tricky topic to tackle, essentially BuildBox is not ready to
    declare a stable API/ABI at this point, and this mostly revolves around
    internal/changing interfaces between buildbox-casd, buildbox-run and their
    common dependency buildbox-common.

    From a BuildStream perspective, we are relying on interfaces which will
    mostly be stable, those are the CLI for buildbox-run and buildbox-casd,
    and the GRPC interface for buildbox-casd.

    Consensus: We will not block on absolute stability of BuildBox, as we
               already work quite closely with BuildBox and do not expect
               there to be problems on this surface.


Does BuildStream need to have any special feature to support testing, e.g. `bst test` ?
---------------------------------------------------------------------
It was discussed that it would be undesirable for Elements to gain
unnecessary complexity, and it would be undesirable to have a situation
where an existing artifact can later be modified by virtue of it, and
it's dependencies having been "tested".

In the original discussions there were concerns also about scheduling,
and the availability of build trees at the time of test (testing an
element can often require the presence of the element's build tree).

While we prefer not to add any complexity in the core, we recognize that
testing is considered a first class operation by users.

Blocker: Revisit this discussion with a clear proposal, and evaluate
         later whether this would be a breaking change.


Future plans after 2.0
----------------------
We are committed to continue development on 2.x for the long term,
and do not have plans to start work on BuildStream 3 shortly after
releasing BuildStream 2.0.

This will signal a return to a stable development model where
future versions of BuildStream will remain backwards compatible
with earlier versions of BuildStream 2 with regards to all of
our stable API surfaces (CLI, Plugin API and YAML format), with
a new commitment to cache key stability.


Links
===============================
[0]: https://gitlab.com/BuildStream/buildstream/-/issues/1272
[1]: https://gitlab.com/BuildStream/buildstream/-/issues/1273
[2]: https://mail.gnome.org/archives/buildstream-list/2019-June/msg00015.html
[3]: https://gitlab.com/BuildStream/buildstream/-/issues/1061
[4]: https://gitlab.com/BuildStream/buildstream/-/issues/1127
[5]: https://gitlab.com/BuildStream/buildstream/-/merge_requests/1851
[6]: https://mail.gnome.org/archives/buildstream-list/2020-April/msg00005.html
[7]: https://mail.gnome.org/archives/buildstream-list/2019-December/msg00023.html
[8]: https://gitlab.com/BuildStream/buildstream/-/issues/719
[9]: https://mail.gnome.org/archives/buildstream-list/2020-January/msg00024.html
[10]: https://mail.gnome.org/archives/buildstream-list/2019-December/msg00000.html
[11]: https://gitlab.com/BuildStream/buildstream/-/issues/1262
[12]: https://mail.gnome.org/archives/buildstream-list/2019-February/msg00057.html
[13]: https://gitlab.com/BuildStream/buildstream/-/issues/1224
[14]: https://gitlab.com/BuildStream/buildstream/-/issues/1068
[15]: https://mail.gnome.org/archives/buildstream-list/2019-June/msg00004.html
[16]: https://mail.gnome.org/archives/buildstream-list/2018-October/msg00077.html
[17]: https://mail.gnome.org/archives/buildstream-list/2018-September/msg00028.html
[18]: https://gitlab.com/BuildStream/buildstream/issues/569
[19]: https://gitlab.com/BuildStream/buildstream/-/issues/1274
[20]: https://gitlab.com/BuildStream/buildstream/-/issues/1275
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]