Re: [BuildStream] BuildStream Gathering: Call for agenda

[Laurence Urhegyi <laurence urhegyi codethink co uk>]

Hi,

On 2019-01-11 16:48, Chandan Singh wrote:
> Call for topics
> ===============
>
> If you wish to propose a topic for the Gathering, please reply to this 
> thread.
> For each proposal, please also indicate which kind of slot you want -
> All hands (1 hr, everyone), Regular topic (45 mins) or Small topic (25 
> mins).

See below with suggested slot types:


How can we improve performance?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
All hands (1 hr).
Some key areas:
    * element_update_state
    * element graph annotation via (smarter) querying of servers
    * scheduler changes
    * workspace handling
    * YAML caching
In preparation, work will be done on profiling of large projects and 
collecting data in order to present the findings at the event. Expect a 
more detailed post to the list soon.


BuildStream 2.0
~~~~~~~~~~~~~~~
All hands (1 hr).
While it seems that people agree about the change in release plan on
the mailing list, there hasn't been an official confirmation or
announcement.

We may also want to (briefly) discuss potential breaking changes. I've
mentioned a few on the list [1]. It would be good to at least
conceptually discuss breaking changes and a suitable transition
strategy (e.g., does my suggested approach of adding compatibility code
to [external] plugins make sense?).

[1] 
https://mail.gnome.org/archives/buildstream-list/2019-January/msg00012.html


PartialCAS
~~~~~~~~~~
Regular topic (45 mins).
Stop resolving symlinks in _relative_symlink_target
     See: https://gitlab.com/BuildStream/buildstream/merge_requests/1023
Cache artifacts with virtual directories instead of filesystem
     See: https://gitlab.com/BuildStream/buildstream/merge_requests/991
Partial CAS - only download artifacts or source when needed
     See: 
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00096.html


buildbox-casd
~~~~~~~~~~~~~
Regular topic (45 mins).
Discuss replacing BuildStream's local CAS cache handling with buildbox-
casd. Write down casd requirements to allow full BuildStream
functionality.


Artifact server configuration in projects and junctions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Regular topic (45 mins).
The current artifact server configuration possibilities don't seem to
cover all needs. Especially with regards to servers configured in
subprojects. We should discuss improvements for this. In light of the
change in release plan, we likely do not need 100% backward
compatibility, but we should still make transition as convenient as
possible, of course.

Related issues:
https://gitlab.com/BuildStream/buildstream/issues/401
https://gitlab.com/BuildStream/buildstream/issues/709
https://gitlab.com/BuildStream/buildstream/issues/752
https://gitlab.com/BuildStream/buildstream/issues/782
https://gitlab.com/BuildStream/buildstream/issues/798


fork vs. multi-threading
~~~~~~~~~~~~~~~~~~~~~~~~
Regular topic (45 mins).
There are some issues with the current fork-based subprocess model. One
is that we can't share the HTTP/2 connection to the artifact server
across subprocesses[2]. Another is that we have to be very careful not
to use a multi-threaded library (such as gRPC) in the main process
because forking a multi-threaded process is highly problematic.

[2] https://gitlab.com/BuildStream/buildstream/issues/810

The fork model also adds complexity with regards to element state
handling as all state changes in a subprocess need to be propagated to
the main process somehow.

fork is also not natively available on Windows, which will likely be an
issue if/when we start a native Windows port.

Do we want to at least evaluate a change of the subprocess model for
BuildStream 2.0?


Multiple elements in bst shell
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Small topic (25 mins).
Specifically the 'build shell' element of this, rather than the 'runtime 
shell'.
    See: https://gitlab.com/BuildStream/buildstream/issues/422
    See: https://gitlab.com/BuildStream/buildstream/merge_requests/909