Downloading from generated mirror

[Valentin David <valentin david codethink co uk>]

Hello all,

I have been looking at #330 (Support for mirroring of upstream sources)
and this will need to be working with #328 (Support for downloading
sources from mirrors).

For reference #330 will be able to generate a mirror by running command
"bst mirror", and this can be then served by a HTTP server. #328 adds
support for downloading files from generic mirrors (rather than
generated at the moment). What we want is the work for #328 to also be
able to download from mirrors generated by #330.

There will be some specific handling for generated mirrors. But
hopefully not too much.

I see two specific things required for this to work. And I would to
make sure I am correct. So please tell me if I am wrong.

 1. Path
 -------

Path from mirror root are generated based on original URL. But really
the plugins have some freedom about it. This path should be generated
by the plugin in order to generate the correct URL.

Since handling of URL for #328 is done as higher level, I suppose
plugins should provide a method returning the generated path.

 2. Multiple directories
 -----------------------

For some plugins, in #330, there is an extra component in the path. It
is a number incrementing from 0. There are some plugins that might get
new incompatible repositories or files for a same URL. And when we
detect this, we create a new directory.

 - ostree and git should not suffer of this problem, because we can
just compose repositories and collisions are unlikely.
 - zip and tar need it when the file is changed.
 - bzr needs it, when we detect that remote repository does not contain
at least all the commits we have locally.

The loop that tries mirrors should retry while counting on the last
path component for those plugins requiring it.

There must be now two exceptions raised from source plugins:
 - one that says the URL works but the reference is not available,
 - another one for any other error.
The loop should count until it find the second error.



#328 https://gitlab.com/BuildStream/buildstream/issues/328
MR for #328 https://gitlab.com/BuildStream/buildstream/merge_requests/404

#330 https://gitlab.com/BuildStream/buildstream/issues/330
MR for #330 https://gitlab.com/BuildStream/buildstream/merge_requests/440