Hi All,
This is a proposal to change the structure of the BuildStream project
format regarding Source references (normally git commit shas), such
that they can be stored separately. This would have to be opt-in by the
project.conf which would desire to use this new proposed approach.
I've added Michael on CC as he initially raised this issue on IRC.
Below is a more detailed proposal including a problem statement,
general solution and outline of implementation details.
Cheers,
-Tristan
Problem Statement
~~~~~~~~~~~~~~~~~
It appears that we have overlooked some issues in our endeavors to make
BuildStream convenient in cases where one desires to always build the
"latest of this branch" - a regular case for a group of developers
working on components of a common, integrated system.
With the knowledge that users would want to build the latest and
greatest, we added `bst build --track` and made `--track-save` an
explicit option in order to avoid modifying the elements with new
source references.
The intention here is to avoid having to doctor your ref-less project
whenever you want to pull new updates of the BuildStream project itself
with, e.g. `git pull --rebase` (if you are storing your project in
git).
So far so good - but we failed to look at the bigger picture here.
A.) If you want to open a workspace, BuildStream needs to know what
to use, we dont know what to stage into a workspace in the
absence of a reference.
B.) If you've run a build with `bst build --track` options and
without saving the refs, congratulations. You cannot test it.
o Since the refs were never stored, you cannot run a `bst shell`
on what you've built, let alone be confident that its the exact
binary output of what you just built (a later `bst track`
invocation can pull new references which occurred upstream
after your build completed).
o Neither can you run `bst checkout` to obtain the output of what
you just built, without the references of what it was.
This makes the case for not having the separation of --track and --track-save. Or rather reconsolidate?
Proposed Solution
~~~~~~~~~~~~~~~~~
What I'm proposing to fix this, is to store the source references in
a separate YAML dictionary stored beside `project.conf`. For the sake
of discussion, let's call this `project.refs`.
This is an interesting solution because BuildStream project maintainers
(i.e. those who maintain BuildStream projects in YAML) can decide
whether or not to revision this file; or only revision the file on
tagged release commits.
Thinking about this, is the idea to separate out essentially the bits that would be modified by bst?
If not, then is separating out the refs sufficient?
Will there be any other fields that a specialized source plugin could write? e.g. an etag for a http source?
Cheers,
Sander
