[BuildStream] Buildtrees in 1.4 release

[Laurence Urhegyi <laurence urhegyi codethink co uk>]

Hi,

I think it's time we discussed buildtrees and whether to include this 
functionality in the 1.4 release. I have had multiple conversations in 
private channels about this and think it's right to bring these onto the 
list.

In my simple mind, we can summarise the problem like this:

Buildtrees are generally always large and thus cause big problems when 
cached due to their size. This is annoying for users. There are three 
aspects of buildtrees to consider:
* creating
* pushing
* pulling

There have been discussions since the feature was originally conceived 
over whether the above should be optional or happen by default. Having 
downloads be optional has landed [0], but creation and pushing have not. 
There's a ticket for making the uploading of buildtrees configurable [1] 
(and some thought gone into it) but I don't think there is one for 
creation.

If we're being honest, it's looking unlikely that we'll be able to set 
configuration options for creation and pushing in time for 1.4.

Is this really acceptable to users, to not have these be optional?

I'd like others to weigh in here but I feel like hiding this feature 
with an experimental flag is the right thing to do for the  1.4 release.

Laurence

[0] https://gitlab.com/BuildStream/buildstream/merge_requests/786
[1] https://gitlab.com/BuildStream/buildstream/issues/566