Re: [BuildStream] Partial local CAS



Hi Jim,

On Mon, 2018-11-26 at 17:59 +0000, Jim MacArthur via BuildStream-list wrote:

In the simplest case, we could just check whether a build requirement 
exists on the remote execution storage service first, and if it does, 
and we are using remote execution, we don't need to download that 
artifact or source to the `bst` client machine. It's quite probable that 
someone else sharing the remote execution service will have uploaded 
that artifact already, or a previous run by ourselves will have done.

We generally still need to download the tree (Directory objects) even
for build-only dependencies as that's required for (virtual) staging.
We could potentially download Directory objects on demand instead of
downloading the whole tree, but I think the most important optimization
is to download blobs only if/when required. I consider always fetching
all Directory objects (via GetTree) to be acceptable, at least in a
first step.

The client needs to verify that the blobs are actually on the CAS
server. Either using FindMissingBlobs() or possibly have the
ReferenceStorage service's GetReference() method return success only if
all referenced objects are available. The latter would require a
ReferenceStorage service being available and might be problematic with
potentially optional subdirectories such as 'buildtree'. The former
requires a bit more work on the client side but should be less
problematic, so I would follow that path.

The second case is that the remote execution service could be made to 
find build-requirements itself; if the remote service has access to our 
artifact cache, or source cache when that appears, then it can query 
those for missing artifacts or blobs before it answers its own 
'findMissingBlobs' request.

I don't think we should go that far at this point. I'd stick to the
first optimization, handling this client side, and not requiring
protocol extensions.

Cheers,
Jürg



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]