Re: Storing Source references (e.g. commit shas) separately

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Thu, 2018-01-18 at 11:17 +0100, Jürg Billeter wrote:
> Hi Tristan,
>
> The proposal looks sensible and useful to me.
>
> On Sun, 2018-01-14 at 21:39 +0900, Tristan Van Berkom wrote:
> >   Example:
> >   ~~~~~~~~
> >
> >   # The dictionary of source references
> >   references:
> >
> >     # foo.bst has one git source
> >     foo.bst:
> >     - ref: 02349cfbbf6c5c1242681aa50b828f841e0e3a42
> >
> >     # bar.bst has two tarball sources
> >     bar.bst:
> >     - ref: 0b78b483c179f6998a0df582aea3d77340bb1e9d887b52ed8fae677d535fd19d
> >     - ref: 185f0f175a90bcfc55cf3cf6ceff8d447a6269492c0ca1a1fc0748ea2c181363
>
> I'm not too happy about simply relying on the order of sources. The
> source list of an element can change and then BuildStream would
> (attempt to) use the wrong references, possibly resulting in confusing
> error messages.

Very good point I failed to consider.

Unfortunately there is nothing that identifies a source except for it's
position in the list, and it would be undesirable to add that I think.

> It's not an issue for the very common case of a single source, so maybe
> it doesn't matter in practice. We could write the whole source node to
> project.refs but that might make some aspects more complex.

I originally had that thought in mind in order to simplify how things
get saved, but it falls apart quickly.

Without clear separation of the data, it starts to be ambiguous where
to load the rest of the relevant information from, and moving the whole
source definition into this separate file defeats the purpose of the
separation.

So thinking of the side effects, it seems it would be both highly
unlikely to hit and the error messages could be improved to be more
helpful in the case that project.refs is in use.

 A.) Elements fairly rarely have more than one source

 B.) The order of sources in the Element is significant, so the
     chances that reordering them happens are fairly low as well

 C.) In the off chance that they do get reordered in the element.bst, 
     the error will occur at Source.fetch() time.

     If we can allow SourceError() exceptions to be more specific
     and indicate that it was a bad reference in a machine readable
     way, then we could get the core to raise a more informative
     error when the project.refs is enabled and the source has
     raised an error about a "bad ref".

All told, I think this is not such a bad thing.

Cheers,
    -Tristan