[Notes] [Git][BuildStream/buildstream][relative

Phillip Smyth pushed to branch relative_workspaces at BuildStream / buildstream

Commits:

4d6dbabe

by Tiago Gomes at 2018-08-06T15:03:25Z

PullQueue: fix resource used

Improve fetchers, builders and pushers documentation.

Closes #550.

a8073264

by Tiago Gomes at 2018-08-06T16:55:03Z

Merge branch 'tiagogomes/issue-550' into 'master'

PullQueue: fix resource used

Closes #550

See merge request BuildStream/buildstream!611

3d26ff6e

by Ed Baunton at 2018-08-07T05:04:50Z

remote.py: Add support for marking downloaded files executable

Add an optional flag to make files executable after having downloaded them.

Instead of leaving the permissioning of downloaded file in remote.py up
to the user's umask; expressly set permissions to 0644 or 0755 if
executable.

Bump format version to 13.

60a552a5

by Tristan Van Berkom at 2018-08-07T06:13:53Z

Merge branch 'edbaunton/executable-remote-source' into 'master'

remote.py: Add support for marking downloaded files executable

See merge request BuildStream/buildstream!581

01ff0177

by Javier Jardón at 2018-08-07T07:29:30Z

source/install_linux_distro.rst: Make clearer ArchLinux packages available

9b469fe6

by Tristan Van Berkom at 2018-08-07T09:12:46Z

Merge branch 'jjardon/doc_arch_releases' into 'master'

source/install_linux_distro.rst: Make clearer ArchLinux packages available

See merge request BuildStream/buildstream!613

289de2ef

by Adam Jones at 2018-08-07T09:24:58Z

Add systemd service file examples to artifacts documentation

change code style to ini

e42325b9

by Javier Jardón at 2018-08-07T10:29:46Z

Merge branch 'adamjones/systemd-cas' into 'master'

Include systemd file examples for google-cas cache

Closes #419

See merge request BuildStream/buildstream!524

6d349610

by Javier Jardón at 2018-08-07T10:53:23Z

source/install_linux_distro.rst: Fix formatting

493d19d2

by James Ennis at 2018-08-07T11:08:36Z

install_linux_distro.rst: Change install instructions for python-arpy package on Arch

  - python-arpy is unobtainable via pacman
  - Also a minor change to how we display install instructions
  from Fedora's dnf package manager

fc9869e2

by James Ennis at 2018-08-07T11:56:21Z

Merge branch 'jennis/change_arch_install_instructions' into 'master'

Change install instructions for python-arpy package on Arch

See merge request BuildStream/buildstream!614

42aa3999

by William Salmon at 2018-08-07T13:41:02Z

Add warning to git track if track and ref are not present

This is to address https://gitlab.com/BuildStream/buildstream/issues/471 that
documented unhelpfull behavour when tracking git sources.

2ceb5dec

by Will Salmon at 2018-08-07T14:46:36Z

Merge branch 'willsalmon/trackWarning' into 'master'

Add warning to git track if track and ref are not present

Closes #471

See merge request BuildStream/buildstream!580

eee4b674

by Jürg Billeter at 2018-08-07T15:36:35Z

Revert "Restrict version of pylint"

This reverts commit 4f168b9b6a02216e2fae24d758ae6b778e545869.

The latest version of pytest_pylint works fine with pylint 2, which
means there is no longer a reason to restrict the pylint version.

pylint 2 is required for Python 3.7.

ea27e389

by Jürg Billeter at 2018-08-07T15:36:35Z

_artifactcache/cascache.py: Fix for PEP 479 / Python 3.7

Do not rely on `StopIteration` bubbling up.

https://www.python.org/dev/peps/pep-0479/

fa5a59f0

by Jürg Billeter at 2018-08-07T16:48:21Z

Merge branch 'juerg/python3.7' into 'master'

Python 3.7 support

See merge request BuildStream/buildstream!615

ed653fbc

by Chandan Singh at 2018-08-08T00:06:28Z

.gitlab-ci.yml: Remove unused script "install.sh"

65f382f1

by Chandan Singh at 2018-08-08T10:52:32Z

Merge branch 'chandan/remove-dead-code' into 'master'

.gitlab-ci.yml: Remove unused script "install.sh"

See merge request BuildStream/buildstream!617

c68dcab8

by Tiago Gomes at 2018-08-09T09:18:43Z

cascache: move tmp directory one level up

The CAS uses a temp directory while manipulating the cache, temporary
files can be added and removed while adding artifacts to the cache here.

Since calculation of the cache size happens in parallel to artifact
cache additions, this causes race conditions in the size calculation
job, as we end up calling `stat` on temporary files which are being
removed in parallel.

Handle this by moving the temporary directory out of the way, and avoid
considering the tmp directory when calculating the cache size

35ab0335

by Tiago Gomes at 2018-08-09T10:16:24Z

Merge branch 'tiagogomes/issue-520' into 'master'

Fix race condition when calculating disk usage

See merge request BuildStream/buildstream!600

8aa33e23

by Valentin David at 2018-08-09T12:49:17Z

Keep original flags for create in SafeHardlinks.

When open(2) is used with flags O_CREAT|O_RDWR, the file descriptor
must be readable. Unfortunately O_RDWR was not passed which made
read fail with EBADF and mmap to signal SIGBUS.

This issue happened with man-db for example.

Fixes #143.

ef7810f3

by Valentin David at 2018-08-09T14:06:50Z

Merge branch 'valentindavid/fuse-create-flags' into 'master'

Keep original flags for create in SafeHardlinks.

Closes #143

See merge request BuildStream/buildstream!624

2d061173

by Javier Jardón at 2018-08-09T16:31:15Z

buildstream/_project.py:

Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for
systems where not all CPU can be used (in particular, Docker, CI etc).

As suggested in the multiprocessing docs
(http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count)
using len(os.sched_getaffinity(0)) might be better.

4e1488ee

by Javier Jardón at 2018-08-09T16:31:15Z

buildstream/_project.py: Restrict max-jobs

even if the machine have mores cores available

Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227

de955834

by Javier Jardón at 2018-08-09T17:22:24Z

Merge branch 'jjardon/max-jobs' into 'master'

buildstream/_project.py: Restrict max-jobs

See merge request BuildStream/buildstream!620

97595585
by Jim MacArthur at 2018-08-09T17:35:19Z
```
setup.py: Pin ruamel.yaml version to <= 0.15
```

a602365c

by Tristan Van Berkom at 2018-08-09T20:03:52Z

Merge branch 'jmac/pin-ruamel-version' into 'master'

Pin ruamel.yaml version to <= 0.15

See merge request BuildStream/buildstream!630

78bce629

by Phillip Smyth at 2018-08-10T09:33:29Z

Implementing relative workspaces

This fixes #191

A note has been added to NEWS explaining backwards
compatibility issues

25 changed files:

.gitlab-ci.yml
NEWS
buildstream/_artifactcache/artifactcache.py
buildstream/_artifactcache/cascache.py
buildstream/_artifactcache/casserver.py
buildstream/_frontend/widget.py
buildstream/_fuse/fuse.py
buildstream/_fuse/hardlinks.py
buildstream/_pipeline.py
buildstream/_project.py
buildstream/_scheduler/queues/pullqueue.py
buildstream/_stream.py
buildstream/_versions.py
buildstream/_workspaces.py
buildstream/data/userconfig.yaml
buildstream/element.py
buildstream/plugins/sources/git.py
buildstream/plugins/sources/remote.py
doc/source/install_artifacts.rst
doc/source/install_linux_distro.rst
setup.py
tests/frontend/workspace.py
tests/sources/git.py
tests/sources/remote.py
+ tests/sources/remote/unique-keys/target-custom-executable.bst

Changes:

.gitlab-ci.yml

@@ -26,15 +26,6 @@ source_dist:
    - tar -ztf dist/*
    - tarball=$(cd dist && echo $(ls *))
 -  # Create an installer script
 -  - |
 -    cat > dist/install.sh << EOF
 -    #!/bin/sh
 -    tar -zxf ${tarball}
 -    cd ${tarball%.tar.gz}
 -    pip3 install --no-index .
 -    EOF
+-
    # unpack tarball as `dist/buildstream` directory
    - |
      cat > dist/unpack.sh << EOF
@@ -44,7 +35,6 @@ source_dist:
      EOF
    # Make our helpers executable
 -  - chmod +x dist/install.sh
    - chmod +x dist/unpack.sh
    artifacts:
      paths:

NEWS

@@ -17,6 +17,10 @@ buildstream 1.1.5
      to not be scheduled and fail during artifact assembly,
      and display the retry prompt during an interactive session.
 +  o Due to enabling the use of relative workspaces, "Legacy" workspaces
 +    may need to be closed and remade before the changes will affect them.
 +    Downgrading after using this feature may result in workspaces
 +    not functioning correctly
  =================
  buildstream 1.1.4

buildstream/_artifactcache/artifactcache.py

@@ -80,6 +80,8 @@ class ArtifactCache():
          self.context = context
          self.required_artifacts = set()
          self.extractdir = os.path.join(context.artifactdir, 'extract')
 +        self.tmpdir = os.path.join(context.artifactdir, 'tmp')
++
          self.max_size = context.cache_quota
          self.estimated_size = None
@@ -89,7 +91,8 @@ class ArtifactCache():
          self._local = False
          self.cache_size = None
 -        os.makedirs(context.artifactdir, exist_ok=True)
 +        os.makedirs(self.extractdir, exist_ok=True)
 +        os.makedirs(self.tmpdir, exist_ok=True)
      ################################################
      #  Methods implemented on the abstract class   #

buildstream/_artifactcache/cascache.py

@@ -56,7 +56,8 @@ class CASCache(ArtifactCache):
          super().__init__(context)
          self.casdir = os.path.join(context.artifactdir, 'cas')
 -        os.makedirs(os.path.join(self.casdir, 'tmp'), exist_ok=True)
 +        os.makedirs(os.path.join(self.casdir, 'refs', 'heads'), exist_ok=True)
 +        os.makedirs(os.path.join(self.casdir, 'objects'), exist_ok=True)
          self._enable_push = enable_push
@@ -85,8 +86,6 @@ class CASCache(ArtifactCache):
              # artifact has already been extracted
              return dest
 -        os.makedirs(self.extractdir, exist_ok=True)
+-
          with tempfile.TemporaryDirectory(prefix='tmp', dir=self.extractdir) as tmpdir:
              checkoutdir = os.path.join(tmpdir, ref)
              self._checkout(checkoutdir, tree)
@@ -394,7 +393,7 @@ class CASCache(ArtifactCache):
          try:
              h = hashlib.sha256()
              # Always write out new file to avoid corruption if input file is modified
 -            with tempfile.NamedTemporaryFile(dir=os.path.join(self.casdir, 'tmp')) as out:
 +            with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:
                  # Set mode bits to 0644
                  os.chmod(out.name, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH)
@@ -764,7 +763,7 @@ class CASCache(ArtifactCache):
              # already in local cache
              return
 -        with tempfile.NamedTemporaryFile(dir=os.path.join(self.casdir, 'tmp')) as out:
 +        with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:
              self._fetch_blob(remote, tree, out)
              directory = remote_execution_pb2.Directory()
@@ -778,7 +777,7 @@ class CASCache(ArtifactCache):
                      # already in local cache
                      continue
 -                with tempfile.NamedTemporaryFile(dir=os.path.join(self.casdir, 'tmp')) as f:
 +                with tempfile.NamedTemporaryFile(dir=self.tmpdir) as f:
                      self._fetch_blob(remote, filenode.digest, f)
                      digest = self.add_object(path=f.name)
@@ -846,6 +845,9 @@ class _CASRemote():
  def _grouper(iterable, n):
 -    # pylint: disable=stop-iteration-return
      while True:
 -        yield itertools.chain([next(iterable)], itertools.islice(iterable, n - 1))
 +        try:
 +            current = next(iterable)
 +        except StopIteration:
 +            return
 +        yield itertools.chain([current], itertools.islice(iterable, n - 1))

buildstream/_artifactcache/casserver.py

@@ -161,7 +161,7 @@ class _ByteStreamServicer(bytestream_pb2_grpc.ByteStreamServicer):
          offset = 0
          finished = False
          resource_name = None
 -        with tempfile.NamedTemporaryFile(dir=os.path.join(self.cas.casdir, 'tmp')) as out:
 +        with tempfile.NamedTemporaryFile(dir=self.cas.tmpdir) as out:
              for request in request_iterator:
                  assert not finished
                  assert request.write_offset == offset

buildstream/_frontend/widget.py

@@ -418,7 +418,9 @@ class LogLine(Widget):
              if "%{workspace-dirs" in format_:
                  workspace = element._get_workspace()
                  if workspace is not None:
 -                    path = workspace.path.replace(os.getenv('HOME', '/root'), '~')
 +                    path = workspace.get_absolute_path()
 +                    if path.startswith("~/"):
 +                        path = os.path.join(os.getenv('HOME', '/root'), path[2:])
                      line = p.fmt_subst(line, 'workspace-dirs', "Workspace: {}".format(path))
                  else:
                      line = p.fmt_subst(

buildstream/_fuse/fuse.py

@@ -757,7 +757,11 @@ class FUSE(object):
          if self.raw_fi:
              return self.operations('create', path, mode, fi)
          else:
 -            fi.fh = self.operations('create', path, mode)
 +            # This line is different from upstream to fix issues
 +            # reading file opened with O_CREAT|O_RDWR.
 +            # See issue #143.
 +            fi.fh = self.operations('create', path, mode, fi.flags)
 +            # END OF MODIFICATION
              return 0
      def ftruncate(self, path, length, fip):

buildstream/_fuse/hardlinks.py

@@ -185,12 +185,12 @@ class SafeHardlinkOps(Operations):
          return os.open(full_path, flags)
 -    def create(self, path, mode, fi=None):
 +    def create(self, path, mode, flags):
          full_path = self._full_path(path)
          # If it already exists, ensure it's a copy first
          self._ensure_copy(full_path)
 -        return os.open(full_path, os.O_WRONLY | os.O_CREAT, mode)
 +        return os.open(full_path, flags, mode)
      def read(self, path, length, offset, fh):
          os.lseek(fh, offset, os.SEEK_SET)

buildstream/_pipeline.py

@@ -358,10 +358,24 @@ class Pipeline():
                      inconsistent.append(element)
          if inconsistent:
 -            detail = "Exact versions are missing for the following elements\n" + \
 -                     "Try tracking these elements first with `bst track`\n\n"
 +            detail = "Exact versions are missing for the following elements:\n\n"
++
 +            missingTrack = 0
              for element in inconsistent:
 -                detail += "  " + element._get_full_name() + "\n"
 +                detail += "  " + element._get_full_name()
 +                for source in element.sources():
 +                    if not source._get_consistency() and not source.get_ref():
 +                        if hasattr(source, 'tracking') and source.tracking is None:
 +                            detail += ": Source {} is missing ref and track. ".format(source._get_full_name()) + \
 +                                      "Please specify a ref or branch/tag to track."
 +                            missingTrack = 1
++
 +                detail += "\n"
++
 +            if missingTrack:
 +                detail += "\nThen track these elements with `bst track`\n"
 +            else:
 +                detail += "\nTry tracking these elements first with `bst track`\n"
              raise PipelineError("Inconsistent pipeline", detail=detail, reason="inconsistent-pipeline")
      #############################################################

buildstream/_project.py

@@ -19,7 +19,6 @@
  #        Tiago Gomes <tiago gomes codethink co uk>
  import os
 -import multiprocessing  # for cpu_count()
  from collections import Mapping, OrderedDict
  from pluginbase import PluginBase
  from . import utils
@@ -572,7 +571,10 @@ class Project():
          # Extend variables with automatic variables and option exports
          # Initialize it as a string as all variables are processed as strings.
 -        output.base_variables['max-jobs'] = str(multiprocessing.cpu_count())
 +        # Based on some testing (mainly on AWS), maximum effective
 +        # max-jobs value seems to be around 8-10 if we have enough cores
 +        # users should set values based on workload and build infrastructure
 +        output.base_variables['max-jobs'] = str(min(len(os.sched_getaffinity(0)), 8))
          # Export options into variables, if that was requested
          output.options.export_variables(output.base_variables)

buildstream/_scheduler/queues/pullqueue.py

@@ -29,7 +29,7 @@ class PullQueue(Queue):
      action_name = "Pull"
      complete_name = "Pulled"
 -    resources = [ResourceType.UPLOAD]
 +    resources = [ResourceType.DOWNLOAD]
      def process(self, element):
          # returns whether an artifact was downloaded or not

buildstream/_stream.py

@@ -460,7 +460,7 @@ class Stream():
                                                selection=PipelineSelection.REDIRECT,
                                                track_selection=PipelineSelection.REDIRECT)
          target = elements[0]
 -        workdir = os.path.abspath(directory)
 +        directory = os.path.abspath(directory)
          if not list(target.sources()):
              build_depends = [x.name for x in target.dependencies(Scope.BUILD, recurse=False)]
@@ -476,7 +476,7 @@ class Stream():
          workspace = workspaces.get_workspace(target._get_full_name())
          if workspace and not force:
              raise StreamError("Workspace '{}' is already defined at: {}"
 -                              .format(target.name, workspace.path))
 +                              .format(target.name, workspace.get_absolute_path()))
          # If we're going to checkout, we need at least a fetch,
          # if we were asked to track first, we're going to fetch anyway.
@@ -502,7 +502,7 @@ class Stream():
          except OSError as e:
              raise StreamError("Failed to create workspace directory: {}".format(e)) from e
 -        workspaces.create_workspace(target._get_full_name(), workdir)
 +        workspaces.create_workspace(target._get_full_name(), directory)
          if not no_checkout:
              with target.timed_activity("Staging sources to {}".format(directory)):
@@ -526,12 +526,12 @@ class Stream():
          # Remove workspace directory if prompted
          if remove_dir:
              with self._context.timed_activity("Removing workspace directory {}"
 -                                              .format(workspace.path)):
 +                                              .format(workspace.get_absolute_path())):
                  try:
 -                    shutil.rmtree(workspace.path)
 +                    shutil.rmtree(workspace.get_absolute_path())
                  except OSError as e:
                      raise StreamError("Could not remove  '{}': {}"
 -                                      .format(workspace.path, e)) from e
 +                                      .format(workspace.get_absolute_path(), e)) from e
          # Delete the workspace and save the configuration
          workspaces.delete_workspace(element_name)
@@ -574,28 +574,30 @@ class Stream():
          for element in elements:
              workspace = workspaces.get_workspace(element._get_full_name())
+-
 +            workspace_path = workspace.get_absolute_path()
              if soft:
                  workspace.prepared = False
                  self._message(MessageType.INFO, "Reset workspace state for {} at: {}"
 -                              .format(element.name, workspace.path))
 +                              .format(element.name, workspace_path))
                  continue
              with element.timed_activity("Removing workspace directory {}"
 -                                        .format(workspace.path)):
 +                                        .format(workspace_path)):
                  try:
 -                    shutil.rmtree(workspace.path)
 +                    shutil.rmtree(workspace_path)
                  except OSError as e:
                      raise StreamError("Could not remove  '{}': {}"
 -                                      .format(workspace.path, e)) from e
 +                                      .format(workspace_path, e)) from e
              workspaces.delete_workspace(element._get_full_name())
 -            workspaces.create_workspace(element._get_full_name(), workspace.path)
 +            workspaces.create_workspace(element._get_full_name(), workspace_path)
 -            with element.timed_activity("Staging sources to {}".format(workspace.path)):
 +            with element.timed_activity("Staging sources to {}".format(workspace_path)):
                  element._open_workspace()
 -            self._message(MessageType.INFO, "Reset workspace for {} at: {}".format(element.name, workspace.path))
 +            self._message(MessageType.INFO,
 +                          "Reset workspace for {} at: {}".format(element.name,
 +                                                                 workspace_path))
          workspaces.save_config()
@@ -632,7 +634,7 @@ class Stream():
          for element_name, workspace_ in self._context.get_workspaces().list():
              workspace_detail = {
                  'element': element_name,
 -                'directory': workspace_.path,
 +                'directory': workspace_.get_absolute_path(),
+             }
              workspaces.append(workspace_detail)

buildstream/_versions.py

@@ -23,7 +23,7 @@
  # This version is bumped whenever enhancements are made
  # to the `project.conf` format or the core element format.
+ #
 -BST_FORMAT_VERSION = 12
 +BST_FORMAT_VERSION = 13
  # The base BuildStream artifact version

buildstream/_workspaces.py

@@ -26,14 +26,6 @@ from ._exceptions import LoadError, LoadErrorReason
  BST_WORKSPACE_FORMAT_VERSION = 3
 -# Hold on to a list of members which get serialized
 -_WORKSPACE_MEMBERS = [
 -    'prepared',
 -    'path',
 -    'last_successful',
 -    'running_files'
 -]
+-
  # Workspace()
+ #
@@ -56,7 +48,7 @@ class Workspace():
      def __init__(self, toplevel_project, *, last_successful=None, path=None, prepared=False, running_files=None):
          self.prepared = prepared
          self.last_successful = last_successful
 -        self.path = path
 +        self._path = path
          self.running_files = running_files if running_files is not None else {}
          self._toplevel_project = toplevel_project
@@ -64,14 +56,20 @@ class Workspace():
      # to_dict()
+     #
 -    # Convert this object to a dict for serialization purposes
 +    # Convert a list of members which get serialized to a dict for serialization purposes
+     #
      # Returns:
      #     (dict) A dict representation of the workspace
+     #
      def to_dict(self):
 -        return {key: val for key, val in self.__dict__.items()
 -                if key in _WORKSPACE_MEMBERS and val is not None}
 +        ret = {
 +            'prepared': self.prepared,
 +            'path': self._path,
 +            'running_files': self.running_files
 +        }
 +        if self.last_successful is not None:
 +            ret["last_successful"] = self.last_successful
 +        return ret
      # from_dict():
+     #
@@ -103,15 +101,7 @@ class Workspace():
      #    True if the workspace differs from 'other', otherwise False
+     #
      def differs(self, other):
+-
 -        for member in _WORKSPACE_MEMBERS:
 -            member_a = getattr(self, member)
 -            member_b = getattr(other, member)
+-
 -            if member_a != member_b:
 -                return True
+-
 -        return False
 +        return self.to_dict() != other.to_dict()
      # invalidate_key()
+     #
@@ -133,7 +123,7 @@ class Workspace():
          if os.path.isdir(fullpath):
              utils.copy_files(fullpath, directory)
          else:
 -            destfile = os.path.join(directory, os.path.basename(self.path))
 +            destfile = os.path.join(directory, os.path.basename(self.get_absolute_path()))
              utils.safe_copy(fullpath, destfile)
      # add_running_files()
@@ -189,7 +179,7 @@ class Workspace():
                  filelist = utils.list_relative_paths(fullpath)
                  filelist = [(relpath, os.path.join(fullpath, relpath)) for relpath in filelist]
              else:
 -                filelist = [(self.path, fullpath)]
 +                filelist = [(self.get_absolute_path(), fullpath)]
              self._key = [(relpath, unique_key(fullpath)) for relpath, fullpath in filelist]
@@ -200,7 +190,7 @@ class Workspace():
      # Returns: The absolute path of the element's workspace.
+     #
      def get_absolute_path(self):
 -        return os.path.join(self._toplevel_project.directory, self.path)
 +        return os.path.join(self._toplevel_project.directory, self._path)
  # Workspaces()
@@ -236,6 +226,9 @@ class Workspaces():
      #    path (str) - The path in which the workspace should be kept
+     #
      def create_workspace(self, element_name, path):
 +        if path.startswith(self._toplevel_project.directory):
 +            path = os.path.relpath(path, self._toplevel_project.directory)
++
          self._workspaces[element_name] = Workspace(self._toplevel_project, path=path)
          return self._workspaces[element_name]

buildstream/data/userconfig.yaml

@@ -35,13 +35,13 @@ cache:
+ #
  scheduler:
 -  # Maximum number of simultaneous source downloading tasks.
 +  # Maximum number of simultaneous downloading tasks.
    fetchers: 10
    # Maximum number of simultaneous build tasks.
    builders: 4
 -  # Maximum number of simultaneous artifact uploading tasks.
 +  # Maximum number of simultaneous uploading tasks.
    pushers: 4
    # Maximum number of retries for network tasks.

buildstream/element.py

@@ -1403,7 +1403,8 @@ class Element(Plugin):
                      # If mount_workspaces is set and we're doing incremental builds,
                      # the workspace is already mounted into the sandbox.
                      if not (mount_workspaces and self.__can_build_incrementally()):
 -                        with self.timed_activity("Staging local files at {}".format(workspace.path)):
 +                        with self.timed_activity("Staging local files at {}"
 +                                                 .format(workspace.get_absolute_path())):
                              workspace.stage(temp_staging_directory)
                  else:
                      # No workspace, stage directly
@@ -1411,6 +1412,7 @@ class Element(Plugin):
                          source._stage(temp_staging_directory)
                  vdirectory.import_files(temp_staging_directory)
++
          # Ensure deterministic mtime of sources at build time
          vdirectory.set_deterministic_mtime()
          # Ensure deterministic owners of sources at build time
@@ -1566,7 +1568,7 @@ class Element(Plugin):
                          path_components = self.__staged_sources_directory.lstrip(os.sep).split(os.sep)
                          sandbox_vpath = sandbox_vroot.descend(path_components)
                          try:
 -                            sandbox_vpath.import_files(workspace.path)
 +                            sandbox_vpath.import_files(workspace.get_absolute_path())
                          except UtilError as e:
                              self.warn("Failed to preserve workspace state for failed build sysroot: {}"
                                        .format(e))
@@ -1893,7 +1895,7 @@ class Element(Plugin):
                  source._init_workspace(temp)
              # Now hardlink the files into the workspace target.
 -            utils.link_files(temp, workspace.path)
 +            utils.link_files(temp, workspace.get_absolute_path())
      # _get_workspace():
+     #

buildstream/plugins/sources/git.py

@@ -363,6 +363,12 @@ class GitSource(Source):
          # If self.tracking is not specified it's not an error, just silently return
          if not self.tracking:
 +            # Is there a better way to check if a ref is given.
 +            if self.mirror.ref is None:
 +                detail = 'Without a tracking branch ref can not be updated. Please ' + \
 +                         'provide a ref or a track.'
 +                raise SourceError("{}: No track or ref".format(self),
 +                                  detail=detail, reason="track-attempt-no-track")
              return None
          with self.timed_activity("Tracking {} from {}"

buildstream/plugins/sources/remote.py

@@ -35,6 +35,10 @@ remote - stage files from remote urls
     # If not specified, the basename of the url will be used.
     # filename: customfilename
 +   # Optionally specify whether the downloaded file should be
 +   # marked executable.
 +   # executable: true
++
     # Specify the url. Using an alias defined in your project
     # configuration is encouraged. 'bst track' will update the
     # sha256sum in 'ref' to the downloaded file's sha256sum.
@@ -43,6 +47,8 @@ remote - stage files from remote urls
     # Specify the ref. It's a sha256sum of the file you download.
     ref: 6c9f6f68a131ec6381da82f2bff978083ed7f4f7991d931bfa767b7965ebc94b
++
++
  .. note::
     The ``remote`` plugin is available since :ref:`format version 10 <project_format_version>`
@@ -60,22 +66,31 @@ class RemoteSource(DownloadableFileSource):
          super().configure(node)
          self.filename = self.node_get_member(node, str, 'filename', os.path.basename(self.url))
 +        self.executable = self.node_get_member(node, bool, 'executable', False)
          if os.sep in self.filename:
              raise SourceError('{}: filename parameter cannot contain directories'.format(self),
                                reason="filename-contains-directory")
 -        self.node_validate(node, DownloadableFileSource.COMMON_CONFIG_KEYS + ['filename'])
 +        self.node_validate(node, DownloadableFileSource.COMMON_CONFIG_KEYS + ['filename', 'executable'])
      def get_unique_key(self):
 -        return super().get_unique_key() + [self.filename]
 +        return super().get_unique_key() + [self.filename, self.executable]
      def stage(self, directory):
          # Same as in local plugin, don't use hardlinks to stage sources, they
          # are not write protected in the sandbox.
          dest = os.path.join(directory, self.filename)
          with self.timed_activity("Staging remote file to {}".format(dest)):
++
              utils.safe_copy(self._get_mirror_file(), dest)
 +            # To prevent user's umask introducing variability here, explicitly set
 +            # file modes.
 +            if self.executable:
 +                os.chmod(dest, 0o755)
 +            else:
 +                os.chmod(dest, 0o644)
++
  def setup():
      return RemoteSource

doc/source/install_artifacts.rst

@@ -143,6 +143,50 @@ Instance with push and requiring client authentication:
      bst-artifact-server --port 11002 --server-key server.key --server-cert server.crt --client-certs authorized.crt --enable-push /home/artifacts/artifacts
 +Managing the cache with systemd
 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
++
 +It is better to run the cache as a systemd service, especially if it is running on a dedicated server, as this will allow systemd to manage the cache, incase the server ever encounters any issues.
++
 +Below are two examples of how to run the cache server as a systemd service, one is for pull only and the other is configured for push & pull.
++
 +.. code:: ini
++
 +   #
 +   # Pull
 +   #
 +   [Unit]
 +   Description=Buildstream Artifact pull server
 +   After=remote-fs.target network-online.target
++
 +   [Service]
 +   Environment="LC_ALL=C.UTF-8"
 +   ExecStart=/usr/local/bin/bst-artifact-server --port 11001 --server-key {{certs_path}}/privkey.pem --
 +   server-cert {{certs_path}}/fullchain.pem {{artifacts_path}}
 +   User=artifacts
++
 +   [Install]
 +   WantedBy=multi-user.target
++
++
 +   #
 +   # Pull/Push
 +   #
 +   [Unit]
 +   Description=Buildstream Artifact pull/push server
 +   After=remote-fs.target network-online.target
++
 +   [Service]
 +   Environment="LC_ALL=C.UTF-8"
 +   ExecStart=/usr/local/bin/bst-artifact-server --port 11002 --server-key {{certs_path}}/privkey.pem --
 +   server-cert {{certs_path}}/fullchain.pem --client-certs /home/artifacts/authorized.crt --enable-push /
 +   {{artifacts_path}}
 +   User=artifacts
++
 +   [Install]
 +   WantedBy=multi-user.target
++
 +Here we define when systemd should start the service, which is after the networking stack has been started, we then define how to run the cache with the desired configuration, under the artifacts user. The {{ }} are there to denote where you should change these files to point to your desired locations.
  User configuration
  ~~~~~~~~~~~~~~~~~~

doc/source/install_linux_distro.rst

@@ -57,9 +57,20 @@ Install the dependencies with::
  For the default plugins::
    sudo pacman -S \
 -      bzr git lzip ostree patch python-arpy python-gobject
 +      bzr git lzip ostree patch python-gobject
 +The package *python-arpy* is required by the deb source plugin. This is not
 +obtainable via `pacman`, you must get *python-arpy* from AUR:
 +https://aur.archlinux.org/packages/python-arpy/
++
 +To install::
++
 +  wget https://aur.archlinux.org/cgit/aur.git/snapshot/python-arpy.tar.gz
 +  tar -xvf python-arpy.tar.gz
 +  cd python-arpy
 +  makepkg -si
++
  Debian
  ++++++
  Install the dependencies with::
@@ -216,17 +227,18 @@ Installing from distro packages
  Arch Linux
  ~~~~~~~~~~
 -Install `buildstream <https://aur.archlinux.org/packages/buildstream>`_
 -from `AUR <https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages>`_.
 -Alternatively, use
 -`buildstream-git <https://aur.archlinux.org/packages/buildstream-git>`_
 -for the lastest version of the development branch.
 +Packages for Arch exist in `AUR <https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages>`_.
 +Two different package versions are available:
++
 +* Latest release: `buildstream <https://aur.archlinux.org/packages/buildstream>`_
 +* Latest development snapshot: `buildstream-git <https://aur.archlinux.org/packages/buildstream-git>`_
++
  Fedora
  ~~~~~~
  BuildStream is not yet in the official Fedora repositories, but you can
 -install it from a Copr:
 +install it from a Copr::
    sudo dnf copr enable bochecha/buildstream
    sudo dnf install buildstream

setup.py

@@ -251,7 +251,7 @@ setup(name='BuildStream',
        install_requires=[
            'setuptools',
            'psutil',
 -          'ruamel.yaml',
 +          'ruamel.yaml <= 0.15',
            'pluginbase',
            'Click',
            'blessings',
@@ -272,6 +272,5 @@ setup(name='BuildStream',
                       'pytest-cov >= 2.5.0',
                       # Provide option to run tests in parallel, less reliable
                       'pytest-xdist',
 -                     'pytest >= 3.1.0',
 -                     'pylint >= 1.8 , < 2'],
 +                     'pytest >= 3.1.0'],
        zip_safe=False)

tests/frontend/workspace.py

@@ -18,12 +18,13 @@ DATA_DIR = os.path.join(
+ )
 -def open_workspace(cli, tmpdir, datafiles, kind, track, suffix=''):
 -    project = os.path.join(datafiles.dirname, datafiles.basename)
 -    bin_files_path = os.path.join(project, 'files', 'bin-files')
 -    element_path = os.path.join(project, 'elements')
 +def open_workspace(cli, tmpdir, datafiles, kind, track, suffix='', workspace_dir=None):
 +    if not workspace_dir:
 +        workspace_dir = os.path.join(str(tmpdir), 'workspace{}'.format(suffix))
 +    project_path = os.path.join(datafiles.dirname, datafiles.basename)
 +    bin_files_path = os.path.join(project_path, 'files', 'bin-files')
 +    element_path = os.path.join(project_path, 'elements')
      element_name = 'workspace-test-{}{}.bst'.format(kind, suffix)
 -    workspace = os.path.join(str(tmpdir), 'workspace{}'.format(suffix))
      # Create our repo object of the given source type with
      # the bin files, and then collect the initial ref.
@@ -45,7 +46,7 @@ def open_workspace(cli, tmpdir, datafiles, kind, track, suffix=''):
                              element_name))
      # Assert that there is no reference, a track & fetch is needed
 -    state = cli.get_element_state(project, element_name)
 +    state = cli.get_element_state(project_path, element_name)
      if track:
          assert state == 'no reference'
      else:
@@ -56,20 +57,20 @@ def open_workspace(cli, tmpdir, datafiles, kind, track, suffix=''):
      args = ['workspace', 'open']
      if track:
          args.append('--track')
 -    args.extend([element_name, workspace])
 +    args.extend([element_name, workspace_dir])
 +    result = cli.run(project=project_path, args=args)
 -    result = cli.run(project=project, args=args)
      result.assert_success()
      # Assert that we are now buildable because the source is
      # now cached.
 -    assert cli.get_element_state(project, element_name) == 'buildable'
 +    assert cli.get_element_state(project_path, element_name) == 'buildable'
      # Check that the executable hello file is found in the workspace
 -    filename = os.path.join(workspace, 'usr', 'bin', 'hello')
 +    filename = os.path.join(workspace_dir, 'usr', 'bin', 'hello')
      assert os.path.exists(filename)
 -    return (element_name, project, workspace)
 +    return (element_name, project_path, workspace_dir)
  @pytest.mark.datafiles(DATA_DIR)
@@ -190,6 +191,45 @@ def test_close(cli, tmpdir, datafiles, kind):
      assert not os.path.exists(workspace)
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_close_external_after_move_project(cli, tmpdir, datafiles):
 +    tmp_parent = os.path.dirname(str(tmpdir))
 +    workspace_dir = os.path.join(tmp_parent, "workspace")
 +    element_name, project_path, _ = open_workspace(cli, tmpdir, datafiles, 'git', False, "", workspace_dir)
 +    assert os.path.exists(workspace_dir)
 +    tmp_dir = os.path.join(tmp_parent, 'external_project')
 +    shutil.move(project_path, tmp_dir)
 +    assert os.path.exists(tmp_dir)
++
 +    # Close the workspace
 +    result = cli.run(configure=False, project=tmp_dir, args=[
 +        'workspace', 'close', '--remove-dir', element_name
 +    ])
 +    result.assert_success()
++
 +    # Assert the workspace dir has been deleted
 +    assert not os.path.exists(workspace_dir)
 +    shutil.move(tmp_dir, project_path)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_close_internal_after_move_project(cli, tmpdir, datafiles):
 +    element_name, project, _ = open_workspace(cli, tmpdir, datafiles, 'git', False)
 +    tmp_dir = os.path.join(os.path.dirname(str(tmpdir)), 'external_project')
 +    shutil.move(str(tmpdir), tmp_dir)
 +    assert os.path.exists(tmp_dir)
++
 +    # Close the workspace
 +    result = cli.run(configure=False, project=tmp_dir, args=[
 +        'workspace', 'close', '--remove-dir', element_name
 +    ])
 +    result.assert_success()
++
 +    # Assert the workspace dir has been deleted
 +    workspace = os.path.join(tmp_dir, 'workspace')
 +    assert not os.path.exists(workspace)
++
++
  @pytest.mark.datafiles(DATA_DIR)
  def test_close_removed(cli, tmpdir, datafiles):
      element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, 'git', False)

tests/sources/git.py

@@ -359,3 +359,45 @@ def test_submodule_track_ignore_inconsistent(cli, tmpdir, datafiles):
      # Assert that we are just fine without it, and emit a warning to the user.
      assert "Ignoring inconsistent submodule" in result.stderr
++
++
 +@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
 +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
 +def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
++
 +    # Create the repo from 'repofiles' subdir
 +    repo = create_repo('git', str(tmpdir))
 +    ref = repo.create(os.path.join(project, 'repofiles'))
++
 +    # Write out our test target
 +    gitsource = repo.source_config(ref=None)
 +    gitsource.pop('track')
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            gitsource
 +        ]
 +    }
++
 +    _yaml.dump(element, os.path.join(project, 'target.bst'))
++
 +    # Track will encounter an inconsistent submodule without any ref
 +    result = cli.run(project=project, args=['track', 'target.bst'])
 +    result.assert_main_error(ErrorDomain.STREAM, None)
 +    result.assert_task_error(ErrorDomain.SOURCE, 'track-attempt-no-track')
++
 +    # Assert that we are just fine without it, and emit a warning to the user.
 +    assert "FAILURE git source at" in result.stderr
 +    assert "Without a tracking branch ref can not be updated. Please " + \
 +        "provide a ref or a track." in result.stderr
++
 +    # Track will encounter an inconsistent submodule without any ref
 +    result = cli.run(project=project, args=['build', 'target.bst'])
 +    result.assert_main_error(ErrorDomain.PIPELINE, 'inconsistent-pipeline')
 +    result.assert_task_error(None, None)
++
 +    # Assert that we are just fine without it, and emit a warning to the user.
 +    assert "Exact versions are missing for the following elements" in result.stderr
 +    assert "is missing ref and track." in result.stderr
 +    assert "Then track these elements with `bst track`" in result.stderr

tests/sources/remote.py

  import os
 +import stat
  import pytest
  from buildstream._exceptions import ErrorDomain
@@ -82,7 +83,14 @@ def test_simple_file_build(cli, tmpdir, datafiles):
      result.assert_success()
      # Note that the url of the file in target.bst is actually /dir/file
      # but this tests confirms we take the basename
 -    assert(os.path.exists(os.path.join(checkoutdir, 'file')))
 +    checkout_file = os.path.join(checkoutdir, 'file')
 +    assert(os.path.exists(checkout_file))
++
 +    mode = os.stat(checkout_file).st_mode
 +    # Assert not executable by anyone
 +    assert(not (mode & (stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)))
 +    # Assert not writeable by anyone other than me
 +    assert(not (mode & (stat.S_IWGRP | stat.S_IWOTH)))
  @pytest.mark.datafiles(os.path.join(DATA_DIR, 'single-file-custom-name'))
@@ -119,6 +127,7 @@ def test_unique_key(cli, tmpdir, datafiles):
      generate_project(project, tmpdir)
      assert cli.get_element_state(project, 'target.bst') == "fetch needed"
      assert cli.get_element_state(project, 'target-custom.bst') == "fetch needed"
 +    assert cli.get_element_state(project, 'target-custom-executable.bst') == "fetch needed"
      # Try to fetch it
      result = cli.run(project=project, args=[
          'fetch', 'target.bst'
@@ -127,7 +136,31 @@ def test_unique_key(cli, tmpdir, datafiles):
      # We should download the file only once
      assert cli.get_element_state(project, 'target.bst') == 'buildable'
      assert cli.get_element_state(project, 'target-custom.bst') == 'buildable'
 +    assert cli.get_element_state(project, 'target-custom-executable.bst') == 'buildable'
      # But the cache key is different because the 'filename' is different.
      assert cli.get_element_key(project, 'target.bst') != \
 -        cli.get_element_key(project, 'target-custom.bst')
 +        cli.get_element_key(project, 'target-custom.bst') != \
 +        cli.get_element_key(project, 'target-custom-executable.bst')
++
++
 +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'unique-keys'))
 +def test_executable(cli, tmpdir, datafiles):
 +    '''This test confirms that the 'ecxecutable' parameter is honoured.
 +    '''
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    generate_project(project, tmpdir)
 +    checkoutdir = os.path.join(str(tmpdir), "checkout")
 +    assert cli.get_element_state(project, 'target-custom-executable.bst') == "fetch needed"
 +    # Try to fetch it
 +    result = cli.run(project=project, args=[
 +        'build', 'target-custom-executable.bst'
 +    ])
++
 +    result = cli.run(project=project, args=[
 +        'checkout', 'target-custom-executable.bst', checkoutdir
 +    ])
 +    mode = os.stat(os.path.join(checkoutdir, 'some-custom-file')).st_mode
 +    assert (mode & stat.S_IEXEC)
 +    # Assert executable by anyone
 +    assert(mode & (stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH))

tests/sources/remote/unique-keys/target-custom-executable.bst

 +kind: import
 +description: test
 +sources:
 +- kind: remote
 +  url: tmpdir:/dir/file
 +  ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
 +  filename: some-custom-file
 +  executable: true

[Notes] [Git][BuildStream/buildstream][relative_workspaces] 28 commits: PullQueue: fix resource used

Phillip Smyth pushed to branch relative_workspaces at BuildStream / buildstream

Commits:

25 changed files:

Changes: