Re: Package manager integration with BuildStream

[Gökçen Nurlu <gokcennurlu gmail com>]

Hi Jonathan,

Many thanks for your time!

Jonathan Maw via Buildstream-list <buildstream-list gnome org>, 19 Haz
2018 Sal, 13:31 tarihinde şunu yazdı:
> I'm also going to avoid picking at specific details.
>
> My understanding of the purpose of SourceTransform plugins is that they
> need to be able to access the previous sources so they can (in Gopkg's
> case) find the configuration file that lists which packages to download.
>
> So, my questions are:
>
> 1. Is SourceTransform too specific? Could the ability to check previous
> sources be useful for other sources at some point? If so, we'd be able
> to avoid some complexity.

It'd surely give a lot of power to implementors but I honestly don't
know if that would be helpful or not.

In the current implementation, `SourceTransform._track()` and
`SourceTransform_fetch()` don't pollute previously staged files, it
just repeats staging for them in a temp folder. This looks a bit
inefficient, but those sources are already fetched, so.. (This was
taken from https://mail.gnome.org/archives/buildstream-list/2018-April/msg00036.html)
Technically, it is possible to make this new functionality live in
`source.py`. I imagine, probably with a flag like `is_transform` that
source implementors could set `True`. Then some private methods'
behaviours in `Element` and `Source` would change depending on that
value.

IMHO, SourceTransform's behaviour is significantly different than
Source and keeping them (i.e. what they provide) separate might be
cleaner from plugin developer's perspective.

> 2. In __ensure_previous_sources() we're fetching and tracking a bunch of
> sources. I'm concerned that ought to be done in the pipeline, instead.

SourceTransform's consistency & ref strictly depends on previous
source's files to work. This can be reading a pip.lock, Gopkg.lock,
requirements.txt with versions, yarn.lock(?) etc.. I don't think there
is a nice way to read them without actually retrieving them :( We
might want to keep looking for more light-weight ways, though.

About reusing pipeline for this, I don't know how to approach to that
and definitely open to suggestions. One idea in my mind is that: if
there is a SourceTransform-kind of source in sources list of an
element, maybe `Element` can behave differently then and do this
"ensuring" for us.

> 3. This has probably been discussed earlier, but would it be acceptable
> to embed the contents of the Gopkg.toml file into the source's config
> field, instead?
>     If the files are too big, I believe there was a proposal for
> buildstream to be able to include loose files into the yaml.
>     I suppose the main thing against sticking the package list in the
> project is that you wouldn't be able to track the package list and would
> have to include it manually :-/

We want to be able to pick any Source plugin (git,tar etc.) and
combine it with any SourceTransform plugin (dep, pip, cargo..) in an
element. SourceTransform doesn't have to know where files come from.
This way, we also use BuildStream's source caching AFAIK.

When populated after `track`, SourceTransform-kind plugins' REF field
will be probably big and ugly (literally dumping Gopkg.lock,
cargo.lock etc. there). I think plugin-implementors can workaround
that by enforcing a lock file of the corresponding package manager.
IMO, this is good because if we can provide a way to pass previous
sources' `refs` to SourceTransform-kind plugin, it can use them as its
own ref for tracking and caching and maybe we can even avoid
`__ensure_previous_sources` in `SourceTransform.track()`.


Best,
Gokcen