[Notes] [Git][BuildStream/buildstream][tiagogomes/docs-improvements] 29

Tiago Gomes pushed to branch tiagogomes/docs-improvements at BuildStream / buildstream

Commits:

14e1a3b3

by Jürg Billeter at 2018-10-01T08:18:05Z

_platform: Fix get_cpu_count() with cap=None

Comparison between None and integer is not supported.

232662f1

by Jürg Billeter at 2018-10-01T08:53:06Z

Merge branch 'juerg/cpucount' into 'master'

_platform: Fix get_cpu_count() with cap=None

See merge request BuildStream/buildstream!840

f447aedd
by Tiago Gomes at 2018-10-01T10:33:11Z
```
ci: upper case variables

For consistency.
```
682dddce
by Tiago Gomes at 2018-10-01T10:35:12Z
```
ci: correct freedestop-sdk version comment
```

fafa8136

by Tiago Gomes at 2018-10-01T10:59:54Z

Merge branch 'tiagogomes/ci-minor-tweaks' into 'master'

Minor CI tweaks

See merge request BuildStream/buildstream!842

26e1a3c7

by Jürg Billeter at 2018-10-01T14:58:06Z

_artifactcache/casserver.py: Implement BatchUpdateBlobs

Fixes #676.

f47895c0

by Jürg Billeter at 2018-10-01T14:58:06Z

_artifactcache/cascache.py: Use BatchUpdateBlobs

This uses BatchUpdateBlobs instead of individual blob upload to speed up
artifact pushing, if the server supports it.

Fixes #677.

cf00c0a1

by Jürg Billeter at 2018-10-01T15:32:30Z

Merge branch 'juerg/cas-batch' into 'master'

CAS: Implement BatchUpdateBlobs support

Closes #677 and #676

See merge request BuildStream/buildstream!839

5f4ae90b

by Jürg Billeter at 2018-10-02T06:34:02Z

sandbox/_sandboxremote.py: Use helpers to set up cwd and env

This matches the other sandbox implementations.

0458bc4e

by Jürg Billeter at 2018-10-02T07:08:35Z

Merge branch 'juerg/sandbox' into 'master'

sandbox/_sandboxremote.py: Use helpers to set up cwd and env

See merge request BuildStream/buildstream!841

d5b396e0

by Phillip Smyth at 2018-10-02T16:40:18Z

Forcing CI to install Sphinx 1.7.9

.gitlab-ci.yml: forced pip3 install Sphinx 1.7.9
The theme we use in our docs "sphinx_rtd_theme"
Does not support the latest version of Sphinx (>1.8)
This results in search functionality failing

dae842fd

by Phillip Smyth at 2018-10-02T17:14:09Z

Merge branch 'docs_Search_not_working' into 'master'

Fixing: Search functionality in the Documentation has stopped working

Closes #681

See merge request BuildStream/buildstream!848

8dc16d3f

by Jürg Billeter at 2018-10-03T05:08:21Z

_platform/linux.py: Accept all configs for dummy sandbox

If Sandbox.run() is never called, sandbox config is irrelevant. If it is
called, the build will fail anyway with the dummy sandbox.

This matches the logic in the Darwin platform.

66446fc3

by Jürg Billeter at 2018-10-03T05:36:38Z

Merge branch 'juerg/dummy-sandbox' into 'master'

_platform/linux.py: Accept all configs for dummy sandbox

See merge request BuildStream/buildstream!843

29c19bea

by Tristan Van Berkom at 2018-10-03T07:33:48Z

git.py: Added timed activity around fetching, displaying the URL being fetched.

This will take care of silencing the status messages while checking submodules.

b645881c

by Tristan Van Berkom at 2018-10-03T07:33:48Z

source.py: Silence messages while consuming source fetchers

The source fetchers might be a list or a generator, when it is
a generator (like the git source does), then we want to ensure that
we silence the status messages which might occur as a result of
consuming a source fetcher from the generator.

This fixes the logs to be less verbose.

c9437616

by Tristan Van Berkom at 2018-10-03T08:07:15Z

Merge branch 'tristan/fix-status-messages' into 'master'

fix status messages

See merge request BuildStream/buildstream!845

11320fe2

by Tristan Van Berkom at 2018-10-03T09:33:39Z

_scheduler: Fix bookkeeping of terminated jobs

 * Enhanced the base Job class to bookkeep which jobs have
   been terminated, and to consider them as `skipped` when
   asked via the `skipped` property.

 * Enhanced the base Queue class to bookkeep the job statuses
   more carefully.

This fixes #479

91271964

by Tristan Van Berkom at 2018-10-03T09:59:40Z

Merge branch 'tristan/fix-terminated-jobs' into 'master'

_scheduler: Fix bookkeeping of terminated jobs

Closes #479

See merge request BuildStream/buildstream!850

3bf895d2

by Jonathan Maw at 2018-10-03T11:48:48Z

tests: Add regression test for mirroring with project.refs

e4969807

by Jonathan Maw at 2018-10-03T12:48:07Z

Merge branch 'jonathan/source-mirror-project-refs' into 'master'

tests: Add regression test for mirroring with project.refs

See merge request BuildStream/buildstream!823

a0814399

by Tristan Van Berkom at 2018-10-03T13:05:52Z

_scheduler/scheduler.py: Ignore interrupt events while terminating.

For some reason, we now receive a SIGINT from the main loop even
when the SIGINT occurred with the handler disconnected in an interactive
prompt. This patch simply ignores any received SIGINT events from the
main loop in the case that we are already in the process of terminating.

This fixes issue #693

0a1f8e3c

by Tristan Van Berkom at 2018-10-03T13:42:20Z

Merge branch 'tristan/fix-double-terminate-prompt' into 'master'

_scheduler/scheduler.py: Ignore interrupt events while terminating.

Closes #693

See merge request BuildStream/buildstream!852

11161f99

by Tristan Van Berkom at 2018-10-03T13:44:02Z

utils.py: Give save_file_atomic() a tempdir argument

Allow callers to decide where the temporary file will be created.

3e797bb9

by Tristan Van Berkom at 2018-10-03T13:44:02Z

_artifactcache/cascache.py: Don't create temporary files in the CAS storage

Use the designated tempdir when creating refs, we expect that temporary
files are not created in the storage directory ever, they should be
only ever created in the designated temporary directory.

This fixes race conditions where utils._get_dir_size() throws an
unhandled exception when attempting to stat the file which inadvertantly
disappears.

d9020e43

by Tristan Van Berkom at 2018-10-03T13:44:02Z

utils.py: Document _get_dir_size() expectations.

This function assumes that files do not disappear while
walking the given directory.

3e5ff5a9

by Tristan Van Berkom at 2018-10-03T14:09:51Z

Merge branch 'tristan/fix-cache-size-race' into 'master'

fix cache size race

See merge request BuildStream/buildstream!854

3247d22b
by Tiago Gomes at 2018-10-03T15:39:00Z
```
doc: show an additional level on ToC
```

875496d2

by Tiago Gomes at 2018-10-03T15:39:00Z

doc: updates considering website being live now

* Add a link to the website on the main page.
* Remove install instructions as they are now on the website.
* Remove Resources section as that information can be found at the
  website, and also looks bad.
* Move artifact server setup from the no longer existing Install section
  to the Using section.

24 changed files:

.gitlab-ci.yml
buildstream/_artifactcache/cascache.py
buildstream/_artifactcache/casserver.py
buildstream/_platform/darwin.py
buildstream/_platform/linux.py
buildstream/_platform/platform.py
buildstream/_scheduler/jobs/job.py
buildstream/_scheduler/queues/queue.py
buildstream/_scheduler/scheduler.py
buildstream/plugins/sources/git.py
buildstream/sandbox/_sandboxremote.py
buildstream/source.py
buildstream/utils.py
doc/source/index.rst
− doc/source/install_docker.rst
− doc/source/install_linux_distro.rst
− doc/source/install_source.rst
− doc/source/install_versions.rst
− doc/source/main_install.rst
doc/source/main_using.rst
− doc/source/release-badge.rst
− doc/source/snapshot-badge.rst
doc/source/install_artifacts.rst → doc/source/using_configuring_artifact_server.rst
tests/frontend/mirror.py

Changes:

.gitlab-ci.yml

@@ -145,7 +145,8 @@ docs:
    stage: test
    script:
    - export BST_SOURCE_CACHE="$(pwd)/cache/integration-cache/sources"
 -  - pip3 install sphinx
 +  # Currently sphinx_rtd_theme does not support Sphinx >1.8, this breaks search functionality
 +  - pip3 install sphinx==1.7.9
    - pip3 install sphinx-click
    - pip3 install sphinx_rtd_theme
    - cd dist && ./unpack.sh && cd buildstream
@@ -161,14 +162,14 @@ docs:
  .overnight-tests: &overnight-tests-template
    stage: test
    variables:
 -    bst_ext_url: git+https://gitlab.com/BuildStream/bst-external.git
 -    bst_ext_ref: 1d6ab71151b93c8cbc0a91a36ffe9270f3b835f1 # 0.5.1
 -    fd_sdk_ref: 88d7c22c2281b987faa02edd57df80d430eecf1f # 18.08.12
 +    BST_EXT_URL: git+https://gitlab.com/BuildStream/bst-external.git
 +    BST_EXT_REF: 1d6ab71151b93c8cbc0a91a36ffe9270f3b835f1 # 0.5.1
 +    FD_SDK_REF: 88d7c22c2281b987faa02edd57df80d430eecf1f # 18.08.11-35-g88d7c22c
    before_script:
    - (cd dist && ./unpack.sh && cd buildstream && pip3 install .)
 -  - pip3 install --user -e ${bst_ext_url}@${bst_ext_ref}#egg=bst_ext
 +  - pip3 install --user -e ${BST_EXT_URL}@${BST_EXT_REF}#egg=bst_ext
    - git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk.git
 -  - git -C freedesktop-sdk checkout ${fd_sdk_ref}
 +  - git -C freedesktop-sdk checkout ${FD_SDK_REF}
    only:
    - schedules

buildstream/_artifactcache/cascache.py

@@ -506,7 +506,7 @@ class CASCache(ArtifactCache):
      def set_ref(self, ref, tree):
          refpath = self._refpath(ref)
          os.makedirs(os.path.dirname(refpath), exist_ok=True)
 -        with utils.save_file_atomic(refpath, 'wb') as f:
 +        with utils.save_file_atomic(refpath, 'wb', tempdir=self.tmpdir) as f:
              f.write(tree.SerializeToString())
      # resolve_ref():
@@ -1048,10 +1048,29 @@ class CASCache(ArtifactCache):
                  missing_blobs[d.hash] = d
          # Upload any blobs missing on the server
 -        for blob_digest in missing_blobs.values():
 -            with open(self.objpath(blob_digest), 'rb') as f:
 -                assert os.fstat(f.fileno()).st_size == blob_digest.size_bytes
 -                self._send_blob(remote, blob_digest, f, u_uid=u_uid)
 +        self._send_blobs(remote, missing_blobs.values(), u_uid)
++
 +    def _send_blobs(self, remote, digests, u_uid=uuid.uuid4()):
 +        batch = _CASBatchUpdate(remote)
++
 +        for digest in digests:
 +            with open(self.objpath(digest), 'rb') as f:
 +                assert os.fstat(f.fileno()).st_size == digest.size_bytes
++
 +                if (digest.size_bytes >= remote.max_batch_total_size_bytes or
 +                        not remote.batch_update_supported):
 +                    # Too large for batch request, upload in independent request.
 +                    self._send_blob(remote, digest, f, u_uid=u_uid)
 +                else:
 +                    if not batch.add(digest, f):
 +                        # Not enough space left in batch request.
 +                        # Complete pending batch first.
 +                        batch.send()
 +                        batch = _CASBatchUpdate(remote)
 +                        batch.add(digest, f)
++
 +        # Send final batch
 +        batch.send()
  # Represents a single remote CAS cache.
@@ -1126,6 +1145,17 @@ class _CASRemote():
                  if e.code() != grpc.StatusCode.UNIMPLEMENTED:
                      raise
 +            # Check whether the server supports BatchUpdateBlobs()
 +            self.batch_update_supported = False
 +            try:
 +                request = remote_execution_pb2.BatchUpdateBlobsRequest()
 +                response = self.cas.BatchUpdateBlobs(request)
 +                self.batch_update_supported = True
 +            except grpc.RpcError as e:
 +                if (e.code() != grpc.StatusCode.UNIMPLEMENTED and
 +                        e.code() != grpc.StatusCode.PERMISSION_DENIED):
 +                    raise
++
              self._initialized = True
@@ -1173,6 +1203,46 @@ class _CASBatchRead():
              yield (response.digest, response.data)
 +# Represents a batch of blobs queued for upload.
 +#
 +class _CASBatchUpdate():
 +    def __init__(self, remote):
 +        self._remote = remote
 +        self._max_total_size_bytes = remote.max_batch_total_size_bytes
 +        self._request = remote_execution_pb2.BatchUpdateBlobsRequest()
 +        self._size = 0
 +        self._sent = False
++
 +    def add(self, digest, stream):
 +        assert not self._sent
++
 +        new_batch_size = self._size + digest.size_bytes
 +        if new_batch_size > self._max_total_size_bytes:
 +            # Not enough space left in current batch
 +            return False
++
 +        blob_request = self._request.requests.add()
 +        blob_request.digest.hash = digest.hash
 +        blob_request.digest.size_bytes = digest.size_bytes
 +        blob_request.data = stream.read(digest.size_bytes)
 +        self._size = new_batch_size
 +        return True
++
 +    def send(self):
 +        assert not self._sent
 +        self._sent = True
++
 +        if len(self._request.requests) == 0:
 +            return
++
 +        batch_response = self._remote.cas.BatchUpdateBlobs(self._request)
++
 +        for response in batch_response.responses:
 +            if response.status.code != grpc.StatusCode.OK.value[0]:
 +                raise ArtifactError("Failed to upload blob {}: {}".format(
 +                    response.digest.hash, response.status.code))
++
++
  def _grouper(iterable, n):
      while True:
          try:

buildstream/_artifactcache/casserver.py

@@ -68,7 +68,7 @@ def create_server(repo, *, enable_push):
          _ByteStreamServicer(artifactcache, enable_push=enable_push), server)
      remote_execution_pb2_grpc.add_ContentAddressableStorageServicer_to_server(
 -        _ContentAddressableStorageServicer(artifactcache), server)
 +        _ContentAddressableStorageServicer(artifactcache, enable_push=enable_push), server)
      remote_execution_pb2_grpc.add_CapabilitiesServicer_to_server(
          _CapabilitiesServicer(), server)
@@ -222,9 +222,10 @@ class _ByteStreamServicer(bytestream_pb2_grpc.ByteStreamServicer):
  class _ContentAddressableStorageServicer(remote_execution_pb2_grpc.ContentAddressableStorageServicer):
 -    def __init__(self, cas):
 +    def __init__(self, cas, *, enable_push):
          super().__init__()
          self.cas = cas
 +        self.enable_push = enable_push
      def FindMissingBlobs(self, request, context):
          response = remote_execution_pb2.FindMissingBlobsResponse()
@@ -260,6 +261,46 @@ class _ContentAddressableStorageServicer(remote_execution_pb2_grpc.ContentAddres
          return response
 +    def BatchUpdateBlobs(self, request, context):
 +        response = remote_execution_pb2.BatchUpdateBlobsResponse()
++
 +        if not self.enable_push:
 +            context.set_code(grpc.StatusCode.PERMISSION_DENIED)
 +            return response
++
 +        batch_size = 0
++
 +        for blob_request in request.requests:
 +            digest = blob_request.digest
++
 +            batch_size += digest.size_bytes
 +            if batch_size > _MAX_PAYLOAD_BYTES:
 +                context.set_code(grpc.StatusCode.INVALID_ARGUMENT)
 +                return response
++
 +            blob_response = response.responses.add()
 +            blob_response.digest.hash = digest.hash
 +            blob_response.digest.size_bytes = digest.size_bytes
++
 +            if len(blob_request.data) != digest.size_bytes:
 +                blob_response.status.code = grpc.StatusCode.FAILED_PRECONDITION
 +                continue
++
 +            try:
 +                _clean_up_cache(self.cas, digest.size_bytes)
++
 +                with tempfile.NamedTemporaryFile(dir=self.cas.tmpdir) as out:
 +                    out.write(blob_request.data)
 +                    out.flush()
 +                    server_digest = self.cas.add_object(path=out.name)
 +                    if server_digest.hash != digest.hash:
 +                        blob_response.status.code = grpc.StatusCode.FAILED_PRECONDITION
++
 +            except ArtifactTooLargeException:
 +                blob_response.status.code = grpc.StatusCode.RESOURCE_EXHAUSTED
++
 +        return response
++
  class _CapabilitiesServicer(remote_execution_pb2_grpc.CapabilitiesServicer):
      def GetCapabilities(self, request, context):

buildstream/_platform/darwin.py

@@ -41,10 +41,11 @@ class Darwin(Platform):
          return True
      def get_cpu_count(self, cap=None):
 -        if cap < os.cpu_count():
 -            return cap
 +        cpu_count = os.cpu_count()
 +        if cap is None:
 +            return cpu_count
          else:
 -            return os.cpu_count()
 +            return min(cpu_count, cap)
      def set_resource_limits(self, soft_limit=OPEN_MAX, hard_limit=None):
          super().set_resource_limits(soft_limit)

buildstream/_platform/linux.py

@@ -55,6 +55,10 @@ class Linux(Platform):
              return SandboxBwrap(*args, **kwargs)
      def check_sandbox_config(self, config):
 +        if not self._local_sandbox_available():
 +            # Accept all sandbox configs as it's irrelevant with the dummy sandbox (no Sandbox.run).
 +            return True
++
          if self._user_ns_available:
              # User namespace support allows arbitrary build UID/GID settings.
              return True

buildstream/_platform/platform.py

@@ -67,7 +67,11 @@ class Platform():
          return cls._instance
      def get_cpu_count(self, cap=None):
 -        return min(len(os.sched_getaffinity(0)), cap)
 +        cpu_count = len(os.sched_getaffinity(0))
 +        if cap is None:
 +            return cpu_count
 +        else:
 +            return min(cpu_count, cap)
      ##################################################################
      #                        Sandbox functions                       #

buildstream/_scheduler/jobs/job.py

@@ -119,6 +119,8 @@ class Job():
          self._result = None                    # Return value of child action in the parent
          self._tries = 0                        # Try count, for retryable jobs
          self._skipped_flag = False             # Indicate whether the job was skipped.
 +        self._terminated = False               # Whether this job has been explicitly terminated
++
          # If False, a retry will not be attempted regardless of whether _tries is less than _max_retries.
+         #
          self._retry_flag = True
@@ -190,6 +192,8 @@ class Job():
          # Terminate the process using multiprocessing API pathway
          self._process.terminate()
 +        self._terminated = True
++
      # terminate_wait()
+     #
      # Wait for terminated jobs to complete
@@ -273,18 +277,22 @@ class Job():
      # running the integration commands).
+     #
      # Args:
 -    #     (int): The plugin identifier for this task
 +    #     task_id (int): The plugin identifier for this task
+     #
      def set_task_id(self, task_id):
          self._task_id = task_id
      # skipped
+     #
 +    # This will evaluate to True if the job was skipped
 +    # during processing, or if it was forcefully terminated.
 +    #
      # Returns:
 -    #    bool: True if the job was skipped while processing.
 +    #    (bool): Whether the job should appear as skipped
 +    #
      @property
      def skipped(self):
 -        return self._skipped_flag
 +        return self._skipped_flag or self._terminated
      #######################################################
      #                  Abstract Methods                   #

buildstream/_scheduler/queues/queue.py

@@ -326,16 +326,20 @@ class Queue():
                            detail=traceback.format_exc())
              self.failed_elements.append(element)
          else:
+-
 -            # No exception occured, handle the success/failure state in the normal way
+             #
 +            # No exception occured in post processing
 +            #
++
 +            # All jobs get placed on the done queue for later processing.
              self._done_queue.append(job)
 -            if success:
 -                if not job.skipped:
 -                    self.processed_elements.append(element)
 -                else:
 -                    self.skipped_elements.append(element)
 +            # A Job can be skipped whether or not it has failed,
 +            # we want to only bookkeep them as processed or failed
 +            # if they are not skipped.
 +            if job.skipped:
 +                self.skipped_elements.append(element)
 +            elif success:
 +                self.processed_elements.append(element)
              else:
                  self.failed_elements.append(element)

buildstream/_scheduler/scheduler.py

@@ -387,6 +387,15 @@ class Scheduler():
      # A loop registered event callback for keyboard interrupts
+     #
      def _interrupt_event(self):
++
 +        # FIXME: This should not be needed, but for some reason we receive an
 +        #        additional SIGINT event when the user hits ^C a second time
 +        #        to inform us that they really intend to terminate; even though
 +        #        we have disconnected our handlers at this time.
 +        #
 +        if self.terminated:
 +            return
++
          # Leave this to the frontend to decide, if no
          # interrrupt callback was specified, then just terminate.
          if self._interrupt_callback:

buildstream/plugins/sources/git.py

@@ -184,10 +184,18 @@ class GitMirror(SourceFetcher):
                           cwd=self.mirror)
      def fetch(self, alias_override=None):
 -        self.ensure(alias_override)
 -        if not self.has_ref():
 -            self._fetch(alias_override)
 -        self.assert_ref()
 +        # Resolve the URL for the message
 +        resolved_url = self.source.translate_url(self.url,
 +                                                 alias_override=alias_override,
 +                                                 primary=self.primary)
++
 +        with self.source.timed_activity("Fetching from {}"
 +                                        .format(resolved_url),
 +                                        silent_nested=True):
 +            self.ensure(alias_override)
 +            if not self.has_ref():
 +                self._fetch(alias_override)
 +            self.assert_ref()
      def has_ref(self):
          if not self.ref:

buildstream/sandbox/_sandboxremote.py

@@ -177,15 +177,11 @@ class SandboxRemote(Sandbox):
          if not cascache.verify_digest_pushed(self._get_project(), upload_vdir.ref):
              raise SandboxError("Failed to verify that source has been pushed to the remote artifact cache.")
 -        # Set up environment and working directory
 -        if cwd is None:
 -            cwd = self._get_work_directory()
+-
 -        if cwd is None:
 -            cwd = '/'
+-
 -        if env is None:
 -            env = self._get_environment()
 +        # Fallback to the sandbox default settings for
 +        # the cwd and env.
 +        #
 +        cwd = self._get_work_directory(cwd=cwd)
 +        env = self._get_environment(cwd=cwd, env=env)
          # We want command args as a list of strings
          if isinstance(command, str):

buildstream/source.py

@@ -965,28 +965,48 @@ class Source(Plugin):
      # Tries to call fetch for every mirror, stopping once it succeeds
      def __do_fetch(self, **kwargs):
          project = self._get_project()
 -        source_fetchers = self.get_source_fetchers()
 +        context = self._get_context()
++
 +        # Silence the STATUS messages which might happen as a result
 +        # of checking the source fetchers.
 +        with context.silence():
 +            source_fetchers = self.get_source_fetchers()
          # Use the source fetchers if they are provided
+         #
          if source_fetchers:
 -            for fetcher in source_fetchers:
 -                alias = fetcher._get_alias()
 -                for uri in project.get_alias_uris(alias, first_pass=self.__first_pass):
 -                    try:
 -                        fetcher.fetch(uri)
 -                    # FIXME: Need to consider temporary vs. permanent failures,
 -                    #        and how this works with retries.
 -                    except BstError as e:
 -                        last_error = e
 -                        continue
+-
 -                    # No error, we're done with this fetcher
 -                    break
 -                else:
 -                    # No break occurred, raise the last detected error
 -                    raise last_error
 +            # Use a contorted loop here, this is to allow us to
 +            # silence the messages which can result from consuming
 +            # the items of source_fetchers, if it happens to be a generator.
 +            #
 +            source_fetchers = iter(source_fetchers)
 +            try:
++
 +                while True:
++
 +                    with context.silence():
 +                        fetcher = next(source_fetchers)
++
 +                    alias = fetcher._get_alias()
 +                    for uri in project.get_alias_uris(alias, first_pass=self.__first_pass):
 +                        try:
 +                            fetcher.fetch(uri)
 +                        # FIXME: Need to consider temporary vs. permanent failures,
 +                        #        and how this works with retries.
 +                        except BstError as e:
 +                            last_error = e
 +                            continue
++
 +                        # No error, we're done with this fetcher
 +                        break
++
 +                    else:
 +                        # No break occurred, raise the last detected error
 +                        raise last_error
++
 +            except StopIteration:
 +                pass
          # Default codepath is to reinstantiate the Source
+         #

buildstream/utils.py

@@ -502,7 +502,7 @@ def get_bst_version():
  @contextmanager
  def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
 -                     errors=None, newline=None, closefd=True, opener=None):
 +                     errors=None, newline=None, closefd=True, opener=None, tempdir=None):
      """Save a file with a temporary name and rename it into place when ready.
      This is a context manager which is meant for saving data to files.
@@ -529,8 +529,9 @@ def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
      # https://bugs.python.org/issue8604
      assert os.path.isabs(filename), "The utils.save_file_atomic() parameter ``filename`` must be an absolute path"
 -    dirname = os.path.dirname(filename)
 -    fd, tempname = tempfile.mkstemp(dir=dirname)
 +    if tempdir is None:
 +        tempdir = os.path.dirname(filename)
 +    fd, tempname = tempfile.mkstemp(dir=tempdir)
      os.close(fd)
      f = open(tempname, mode=mode, buffering=buffering, encoding=encoding,
@@ -562,6 +563,9 @@ def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
+ #
  # Get the disk usage of a given directory in bytes.
+ #
 +# This function assumes that files do not inadvertantly
 +# disappear while this function is running.
 +#
  # Arguments:
  #     (str) The path whose size to check.
+ #

doc/source/index.rst

@@ -13,20 +13,14 @@ They begin with a basic introduction to BuildStream, background
  information on basic concepts, and a guide to the BuildStream command line interface.
  Later sections provide detailed information on BuildStream internals.
 +If you are looking for information about how to install BuildStream,
 +please refer to the `BuildStream website <https://buildstream.build>`_.
  .. toctree::
 -   :maxdepth: 1
 +   :maxdepth: 2
     main_about
 -   main_install
     main_using
     main_core
     CONTRIBUTING
+-
 -Resources
 ----------
 -* GitLab repository: https://gitlab.com/BuildStream/buildstream
 -* Bug Tracking: https://gitlab.com/BuildStream/buildstream/issues
 -* Mailing list: https://mail.gnome.org/mailman/listinfo/buildstream-list
 -* IRC Channel: irc://irc.gnome.org/#buildstream

doc/source/install_docker.rst deleted

+-
+-
 -.. _docker:
+-
 -BuildStream inside Docker
 --------------------------
 -If your system cannot provide the base system requirements for BuildStream, then it is possible to run buildstream within a Docker image.
+-
 -The BuildStream project provides
 -`Docker images <https://hub.docker.com/r/buildstream/buildstream-fedora>`_
 -containing BuildStream and its dependencies.
 -This gives you an easy way to get started using BuildStream on any Unix-like
 -platform where Docker is available, including Mac OS X.
+-
 -We recommend using the
 -`bst-here wrapper script <https://gitlab.com/BuildStream/buildstream/blob/master/contrib/bst-here>`_
 -which automates the necessary container setup. You can download it and make
 -it executable like this:
+-
 -.. code:: bash
+-
 -  mkdir -p ~/.local/bin
 -  curl --get https://gitlab.com/BuildStream/buildstream/raw/master/contrib/bst-here > ~/.local/bin/bst-here
 -  chmod +x ~/.local/bin/bst-here
+-
 -Check if ``~/.local/bin`` appears in your PATH environment variable -- if it
 -doesn't, you should
 -`edit your ~/.profile so that it does <https://stackoverflow.com/questions/14637979/>`_.
+-
 -Once the script is available in your PATH, you can run ``bst-here`` to open a
 -shell session inside a new container based off the latest version of the
 -buildstream-fedora Docker image. The current working directory will be mounted
 -inside the container at ``/src``.
+-
 -You can also run individual BuildStream commands as ``bst-here COMMAND``. For
 -example: ``bst-here show systems/my-system.bst``. Note that BuildStream won't
 -be able to integrate with Bash tab-completion if you invoke it in this way.
+-
 -Two Docker volumes are set up by the ``bst-here`` script:
+-
 - * ``buildstream-cache --`` mounted at ``~/.cache/buildstream``
 - * ``buildstream-config --`` mounted at ``~/.config/``
+-
 -These are necessary so that your BuildStream cache and configuration files
 -persist between invocations of ``bst-here``.

doc/source/install_linux_distro.rst deleted

+-
+-
 -.. _install_linux_distro:
+-
 -Installing from distro packages
 -===============================
 -BuildStream is available on some linux distributions, here are
 -some install instructions for the linux distributions which
 -have packaged BuildStream.
+-
+-
 -Arch Linux
 -----------
 -Packages for Arch exist in `AUR <https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages>`_.
 -Two different package versions are available:
+-
 -* Latest release: `buildstream <https://aur.archlinux.org/packages/buildstream>`_
 -* Latest development snapshot: `buildstream-git <https://aur.archlinux.org/packages/buildstream-git>`_
+-
+-
 -Fedora
 -------
 -BuildStream is not yet in the official Fedora repositories, but you can
 -install it from a Copr::
+-
 -  sudo dnf copr enable bochecha/buildstream
 -  sudo dnf install buildstream
+-
 -Optionally, install the ``buildstream-docs`` package to have the BuildStream
 -documentation in Devhelp or GNOME Builder.

doc/source/install_source.rst deleted

+-
+-
 -Installing from source
 -======================
 -Until BuildStream is available in :ref:`your distro <install_linux_distro>`, you will
 -need to install it yourself from source.
+-
+-
 -Installing dependencies
 ------------------------
 -Before installing BuildStream from source, it is necessary to first install
 -the system dependencies. Below are some linux distribution specific instructions
 -for installing these dependencies.
+-
 -BuildStream requires the following base system requirements:
+-
 -* python3 >= 3.5
 -* bubblewrap >= 0.1.2
 -* fuse2
+-
 -BuildStream also depends on the host tools for the :mod:`Source <buildstream.source>` plugins.
 -Refer to the respective :ref:`source plugin <plugins_sources>` documentation for host tool
 -requirements of specific plugins.
+-
 -The default plugins with extra host dependencies are:
+-
 -* bzr
 -* deb
 -* git
 -* ostree
 -* patch
 -* pip
 -* tar
+-
 -If you intend to push built artifacts to a remote artifact server,
 -which requires special permissions, you will also need:
+-
 -* ssh
+-
+-
 -Arch Linux
 -~~~~~~~~~~
 -Install the dependencies with::
+-
 -  sudo pacman -S \
 -      python fuse2 bubblewrap \
 -      python-pip
+-
 -For the default plugins::
+-
 -  sudo pacman -S \
 -      bzr git lzip ostree patch python-gobject
+-
+-
 -The package *python-arpy* is required by the deb source plugin. This is not
 -obtainable via `pacman`, you must get *python-arpy* from AUR:
 -https://aur.archlinux.org/packages/python-arpy/
+-
 -To install::
+-
 -  wget https://aur.archlinux.org/cgit/aur.git/snapshot/python-arpy.tar.gz
 -  tar -xvf python-arpy.tar.gz
 -  cd python-arpy
 -  makepkg -si
+-
+-
 -Debian
 -~~~~~~
 -Install the dependencies with::
+-
 -  sudo apt-get install \
 -      python3 fuse bubblewrap \
 -      python3-pip python3-dev
+-
 -For the default plugins:
+-
+-
 -Stretch
 -+++++++
 -With stretch, you first need to ensure that you have the backports repository
 -setup as described `here <https://backports.debian.org/Instructions/>`_
+-
 -By adding the following line to your sources.list::
+-
 -  deb http://deb.debian.org/debian stretch-backports main
+-
 -And then running::
+-
 -  sudo apt update
+-
 -At this point you should be able to get the system requirements for the default plugins with::
+-
 -  sudo apt install \
 -      bzr git lzip patch python3-arpy python3-gi
 -  sudo apt install -t stretch-backports \
 -      gir1.2-ostree-1.0 ostree
+-
+-
 -Buster or Sid
 -+++++++++++++
 -For debian unstable or testing, only the following line should be enough
 -to get the system requirements for the default plugins installed::
+-
 -  sudo apt-get install \
 -      lzip gir1.2-ostree-1.0 git bzr ostree patch python3-arpy python3-gi
+-
+-
 -Fedora
 -~~~~~~
 -For recent fedora systems, the following line should get you the system
 -requirements you need::
+-
 -  dnf install -y \
 -      python3 fuse bubblewrap \
 -      python3-pip python3-devel
+-
 -For the default plugins::
+-
 -  dnf install -y \
 -      bzr git lzip patch ostree python3-gobject
 -  pip3 install --user arpy
+-
+-
 -Ubuntu
 -~~~~~~
+-
+-
 -Ubuntu 18.04 LTS or later
 -+++++++++++++++++++++++++
 -Install the dependencies with::
+-
 -  sudo apt install \
 -      python3 fuse bubblewrap \
 -      python3-pip python3-dev
+-
 -For the default plugins::
+-
 -  sudo apt install \
 -      bzr gir1.2-ostree-1.0 git lzip ostree patch python3-arpy python3-gi
+-
+-
 -Ubuntu 16.04 LTS
 -++++++++++++++++
 -On Ubuntu 16.04, neither `bubblewrap <https://github.com/projectatomic/bubblewrap/>`_
 -or `ostree <https://github.com/ostreedev/ostree>`_ are available in the official repositories.
 -You will need to install them in whichever way you see fit. Refer the the upstream documentation
 -for advice on this.
+-
+-
 -Installing
 -----------
 -Once you have the base system dependencies, you can install the BuildStream
 -python package as a regular user.
+-
+-
 -Installing from PyPI (recommended)
 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 -Since we only ever publish :ref:`release versions <install_semantic_versioning>` on
 -PyPI, it is currently recommended to use this installation path. This will
 -ensure that you always have the latest recommended version of BuildStream that
 -we recommend.
+-
 -To install from PyPI, you will additionally require:
+-
 -* pip for python3 (only required for setup)
 -* Python 3 development libraries and headers
+-
 -Simply run the following command::
+-
 -  pip3 install --user BuildStream
+-
 -This will install latest stable version of BuildStream and its pure python
 -dependencies into your user's homedir in ``~/.local``.
+-
 -Keep following the instructions below to ensure that the ``bst``
 -command is in your ``PATH`` and to enable bash completions for it.
+-
 -.. note::
+-
 -  If you want a specific version of BuildStream, you can install it using
 -  ``pip install --user BuildStream==<version-number>``
+-
+-
 -Upgrading from PyPI
 -+++++++++++++++++++
 -Once you have already installed BuildStream from PyPI, you can later update
 -to the latest recommended version like so::
+-
 -  pip install --user --upgrade BuildStream
+-
+-
 -.. _install_git_checkout:
+-
 -Installing from a git checkout
 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 -To install directly from the `git repository <https://gitlab.com/BuildStream/buildstream.git>`_
 -using python's ``pip`` package manager, you will additionally require:
+-
 -* pip for python3 (only required for setup)
 -* Python 3 development libraries and headers
 -* git (to checkout BuildStream)
+-
 -Before installing, please check the existing tags in the git repository
 -and determine which version you want to install, and whether you want
 -to install an official release version (recommended), or a development snapshot
 -to help us out testing the bleeding edge of development. Follow the
 -:ref:`semantic versioning guide <install_semantic_versioning>` to determine
 -which tag you intend to install.
+-
 -Run the following commands::
+-
 -  git clone https://gitlab.com/BuildStream/buildstream.git
 -  cd buildstream
 -  git checkout <desired release tag>
 -  pip3 install --user -e .
+-
 -This will install buildstream's pure python dependencies into
 -your user's homedir in ``~/.local`` and will run BuildStream directly
 -from the git checkout directory.
+-
 -Keep following the instructions below to ensure that the ``bst``
 -command is in your ``PATH`` and to enable bash completions for it.
+-
 -.. note::
+-
 -   We recommend the ``-e`` option because you can upgrade your
 -   installation by simply updating the checked out git repository.
+-
 -   If you want a full installation that is not linked to your
 -   git checkout, just omit the ``-e`` option from the above commands.
+-
+-
 -Upgrading from a git checkout
 -+++++++++++++++++++++++++++++
 -If you installed BuildStream from a local git checkout using ``-e`` option, all
 -you need to do to upgrade BuildStream is to update your local git checkout::
+-
 -  cd /path/to/buildstream
 -  git pull --rebase
+-
 -If you did not specify the ``-e`` option at install time or the dependancies
 -have changed, you will need to cleanly reinstall BuildStream::
+-
 -  pip3 uninstall buildstream
 -  cd /path/to/buildstream
 -  git pull --rebase
 -  pip3 install --user .
+-
 -.. note::
+-
 -   If BuildStream has added any dependencies since the last upgrade,
 -   you will need to uninstall and reinstall to ensure those dependencies
 -   are met, regardless of whether you have used the ``-e`` option at
 -   install time.
+-
+-
 -Post install setup
 -------------------
 -After having installed from source using any of the above methods, some
 -setup will be required to use BuildStream.
+-
+-
 -Adjust PATH
 -~~~~~~~~~~~
 -Since BuildStream is now installed under your local user's install directories,
 -you need to ensure that ``PATH`` is adjusted.
+-
 -A regular way to do this is to add the following line to the end of your ``~/.bashrc``::
+-
 -  export PATH="${PATH}:${HOME}/.local/bin"
+-
 -.. note::
+-
 -   You will have to restart your terminal in order for these changes to take effect.
+-
+-
 -Bash completions
 -~~~~~~~~~~~~~~~~
 -Bash completions are supported by sourcing the ``buildstream/data/bst``
 -script found in the BuildStream repository. On many systems this script
 -can be installed into a completions directory but when installing BuildStream
 -without a package manager this is not an option.
+-
 -To enable completions for an installation of BuildStream you
 -installed yourself from git, just append the script verbatim
 -to your ``~/.bash_completion``:
+-
 -.. literalinclude:: ../../buildstream/data/bst
 -   :language: yaml

doc/source/install_versions.rst deleted

+-
+-
 -.. _install_semantic_versioning:
+-
 -Semantic Versioning
 -===================
 -BuildStream follows the Semantic Versioning Convention `(SemVer) <https://semver.org/>`_,
 -and uses even minor point numbers to denote releases intended for users while
 -odd minor point numbers represent development snapshops.
+-
 -For example, for a given version number ``X.Y.Z``
 - * The ``X.<even number>.*`` versions are releases intended for users.
 - * The ``X.<odd number>.*`` versions are development spanshots intended for testing.
+-
 -If you are :ref:`installing from git <install_git_checkout>`, please look for the latest
 -tag to ensure you're getting the latest release.
+-
 -* Latest release:
+-
 -  .. include:: release-badge.rst
+-
 -* Latest development snapshot:
+-
 -  .. include:: snapshot-badge.rst

doc/source/main_install.rst deleted

+-
+-
 -.. _install:
+-
 -Install
 -=======
+-
 -.. include:: release-badge.rst
+-
 -.. include:: snapshot-badge.rst
+-
 -This section provides instructions for installing BuildStream and its
 -companion artifact server on various platforms, along with any installation
 -related materials.
+-
 -.. note::
+-
 -   BuildStream is currently only supported natively on Linux. Users of Unix-like
 -   systems where Docker is available can still use BuildStream by following the
 -   :ref:`Docker install guide <docker>`
+-
 -.. toctree::
 -   :maxdepth: 1
+-
 -   install_source
 -   install_linux_distro
 -   install_docker
 -   install_artifacts
 -   install_versions

doc/source/main_using.rst

@@ -15,3 +15,4 @@ guides and information on user preferences and configuration.
     using_examples
     using_config
     using_commands
 +   using_configuring_artifact_server

doc/source/release-badge.rst deleted

+-
 -.. Use this file to include the badge in the documentation, but not in
 -   the README.rst or gitlab rendered materials, that doesnt work.
+-
 -   This is partly a workaround for a sphinx issue, we will be able
 -   to avoid the raw html once this is implemented in sphinx:
+-
 -       https://github.com/sphinx-doc/sphinx/issues/2240
+-
 -   Using the <object> tag instead of the <img> tag which sphinx generates
 -   allows the svg to be "interactive", for us this basically means that
 -   the link we encode in the badge svg is used, rather than static urls
 -   which need to be used around the <img> tag.
+-
 -   WARNING: The custom CSS on the style tag will need to change if we
 -            change the theme, so that the <object> tag behaves similar
 -	    to how the <img> tag is themed by the style sheets.
+-
 -.. raw:: html
+-
 -   <a class="reference external image-reference">
 -     <object style="margin-bottom:24px;vertical-align:middle"
 -             data=""
 -	     type="image/svg+xml"/>
 -     </object>
 -   </a>

doc/source/snapshot-badge.rst deleted

+-
 -.. Use this file to include the badge in the documentation, but not in
 -   the README.rst or gitlab rendered materials, that doesnt work.
+-
 -   This is partly a workaround for a sphinx issue, we will be able
 -   to avoid the raw html once this is implemented in sphinx:
+-
 -       https://github.com/sphinx-doc/sphinx/issues/2240
+-
 -   Using the <object> tag instead of the <img> tag which sphinx generates
 -   allows the svg to be "interactive", for us this basically means that
 -   the link we encode in the badge svg is used, rather than static urls
 -   which need to be used around the <img> tag.
+-
 -   WARNING: The custom CSS on the style tag will need to change if we
 -            change the theme, so that the <object> tag behaves similar
 -	    to how the <img> tag is themed by the style sheets.
+-
 -.. raw:: html
+-
 -   <a class="reference external image-reference">
 -     <object style="margin-bottom:24px;vertical-align:middle"
 -             data=""
 -	     type="image/svg+xml"/>
 -     </object>
 -   </a>

doc/source/install_artifacts.rst → doc/source/using_configuring_artifact_server.rst

@@ -2,8 +2,8 @@
  .. _artifacts:
 -Installing an artifact server
 -=============================
 +Configuring Artifact Server
 +===========================
  BuildStream caches the results of builds in a local artifact cache, and will
  avoid building an element if there is a suitable build already present in the
  local artifact cache.
@@ -72,7 +72,7 @@ Installing the server
  ~~~~~~~~~~~~~~~~~~~~~
  You will also need to install BuildStream on the artifact server in order
  to receive uploaded artifacts over ssh. Follow the instructions for installing
 -BuildStream :ref:`here <install>`
 +BuildStream `here <https://buildstream.build/install.html>`_.
  When installing BuildStream on the artifact server, it must be installed
  in a system wide location, with ``pip3 install .`` in the BuildStream

tests/frontend/mirror.py

@@ -139,6 +139,82 @@ def test_mirror_fetch(cli, tmpdir, datafiles, kind):
      result.assert_success()
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("ref_storage", [("inline"), ("project.refs")])
 +@pytest.mark.parametrize("mirror", [("no-mirror"), ("mirror"), ("unrelated-mirror")])
 +def test_mirror_fetch_ref_storage(cli, tmpdir, datafiles, ref_storage, mirror):
 +    bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
 +    dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
 +    upstream_repodir = os.path.join(str(tmpdir), 'upstream')
 +    mirror_repodir = os.path.join(str(tmpdir), 'mirror')
 +    project_dir = os.path.join(str(tmpdir), 'project')
 +    os.makedirs(project_dir)
 +    element_dir = os.path.join(project_dir, 'elements')
++
 +    # Create repo objects of the upstream and mirror
 +    upstream_repo = create_repo('tar', upstream_repodir)
 +    upstream_ref = upstream_repo.create(bin_files_path)
 +    mirror_repo = upstream_repo.copy(mirror_repodir)
 +    mirror_ref = upstream_ref
 +    upstream_ref = upstream_repo.create(dev_files_path)
++
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            upstream_repo.source_config(ref=upstream_ref if ref_storage == 'inline' else None)
 +        ]
 +    }
 +    element_name = 'test.bst'
 +    element_path = os.path.join(element_dir, element_name)
 +    full_repo = element['sources'][0]['url']
 +    upstream_map, repo_name = os.path.split(full_repo)
 +    alias = 'foo'
 +    aliased_repo = alias + ':' + repo_name
 +    element['sources'][0]['url'] = aliased_repo
 +    full_mirror = mirror_repo.source_config()['url']
 +    mirror_map, _ = os.path.split(full_mirror)
 +    os.makedirs(element_dir)
 +    _yaml.dump(element, element_path)
++
 +    if ref_storage == 'project.refs':
 +        # Manually set project.refs to avoid caching the repo prematurely
 +        project_refs = {'projects': {
 +            'test': {
 +                element_name: [
 +                    {'ref': upstream_ref}
 +                ]
 +            }
 +        }}
 +        project_refs_path = os.path.join(project_dir, 'project.refs')
 +        _yaml.dump(project_refs, project_refs_path)
++
 +    project = {
 +        'name': 'test',
 +        'element-path': 'elements',
 +        'aliases': {
 +            alias: upstream_map + "/"
 +        },
 +        'ref-storage': ref_storage
 +    }
 +    if mirror != 'no-mirror':
 +        mirror_data = [{
 +            'name': 'middle-earth',
 +            'aliases': {alias: [mirror_map + '/']}
 +        }]
 +        if mirror == 'unrelated-mirror':
 +            mirror_data.insert(0, {
 +                'name': 'narnia',
 +                'aliases': {'frob': ['http://www.example.com/repo']}
 +            })
 +        project['mirrors'] = mirror_data
++
 +    project_file = os.path.join(project_dir, 'project.conf')
 +    _yaml.dump(project, project_file)
++
 +    result = cli.run(project=project_dir, args=['fetch', element_name])
 +    result.assert_success()
++
++
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
  def test_mirror_fetch_upstream_absent(cli, tmpdir, datafiles, kind):

[Notes] [Git][BuildStream/buildstream][tiagogomes/docs-improvements] 29 commits: _platform: Fix get_cpu_count() with cap=None

Tiago Gomes pushed to branch tiagogomes/docs-improvements at BuildStream / buildstream

Commits:

24 changed files:

Changes: