[BuildStream] Proposal: Allowing download-only sources to work on local files

[Tom Mewett <tom mewett codethink co uk>]

Hi list,

In August this year, I submitted a proposal for an implementation to
allow plugins which are currently remote-only to accept local files in
the project [1]. There was only a small amount of discussion, and also
I have reconsidered a few things, so I have decided to make another
post.

[1] https://mail.gnome.org/archives/buildstream-list/2019-August/msg00011.html


Background
----------

Currently, source plugins which operate on files are split into two
categories: those which only operate on files from the internet (and
derive from DownloadableFileSource; 'deb', 'tar', 'zip', etc.), and
those which only operate on files in the project ('local' and 'patch').

Sometimes, a BuildStream project may need to include proprietary
software. This software may come in an deb/zip/tar archive but may not
be available publicly for direct download. To work with this, one
method is to have elements source the archive from a location in the
project, but to not distribute the archive. Then, instructions are
provided for the user the obtain the software and put it in the right
place.

At present, this is not possible with the deb/zip/tar plugins. The
workarounds are:

- extract the archive and include it via an 'import' element
- use a 'manual' element which depends on ar/unzip/tar and extract and
  stage manually


Proposal
--------

Firstly, I'm going to use the term "URL" in quite a general way here.
That's just to keep with existing terminology; it should probably be
changed to avoid confusion.

In my view, neither of the above workarounds are favourable compared to
removing the asymmetry of local/remote files between plugins. I propose
that the act of "resolving a URL to a file" be generalised and have
plugins only operate on the result of that resolution, not knowing or
caring whether the file was downloaded or is local. In detail:

- All source plugins which operate on files are unified as subclasses of
  a single base class, say FileBasedSource
- This base class handles the 'url' and 'ref' keys of the source config
- First it checks whether 'url' is a fully-qualified URL or is just a
  relative path. If it's the former, it is fetched as necessary and
  stored in the source cache
- If the URL is a relative path, specifying a ref is optional. If it is
  given and is different from what is calculated, an error is thrown

The 'local' and 'remote' sources could then also be unified to some
kind of 'copy' source, as 'copy' would act as either one depending on
whether a local path or full URL was given.

If desirable, the 'url' key could be split into mutually-exclusive
'url' and 'path' keys which would decide the behaviour.

----

In my opinion, this could improve the usability of the plugins in
question. The user thinks of plugins as taking a "location" of either
local or remote files, only requiring refs/checksums for remote ones,
and caching them for convenience. It provides a sort of network
transparency.

Another advantage would be that local files imported into the project
can be given refs, meaning that they would not need to be present to
compute cache keys of depending elements. (This is not possible with
the current 'local' source.)

Please let me know what you think of this idea, or of any alternative
solutions to the problem.

Thanks,
Tom Mewett