[Notes] [Git][BuildStream/buildstream][463-make-dependency-type-default-

Jonathan Maw pushed to branch 463-make-dependency-type-default-to-build at BuildStream / buildstream

Commits:

8aa33e23

by Valentin David at 2018-08-09T12:49:17Z

Keep original flags for create in SafeHardlinks.

When open(2) is used with flags O_CREAT|O_RDWR, the file descriptor
must be readable. Unfortunately O_RDWR was not passed which made
read fail with EBADF and mmap to signal SIGBUS.

This issue happened with man-db for example.

Fixes #143.

ef7810f3

by Valentin David at 2018-08-09T14:06:50Z

Merge branch 'valentindavid/fuse-create-flags' into 'master'

Keep original flags for create in SafeHardlinks.

Closes #143

See merge request BuildStream/buildstream!624

2d061173

by Javier Jardón at 2018-08-09T16:31:15Z

buildstream/_project.py:

Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for
systems where not all CPU can be used (in particular, Docker, CI etc).

As suggested in the multiprocessing docs
(http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count)
using len(os.sched_getaffinity(0)) might be better.

4e1488ee

by Javier Jardón at 2018-08-09T16:31:15Z

buildstream/_project.py: Restrict max-jobs

even if the machine have mores cores available

Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227

de955834

by Javier Jardón at 2018-08-09T17:22:24Z

Merge branch 'jjardon/max-jobs' into 'master'

buildstream/_project.py: Restrict max-jobs

See merge request BuildStream/buildstream!620

97595585
by Jim MacArthur at 2018-08-09T17:35:19Z
```
setup.py: Pin ruamel.yaml version to <= 0.15
```

a602365c

by Tristan Van Berkom at 2018-08-09T20:03:52Z

Merge branch 'jmac/pin-ruamel-version' into 'master'

Pin ruamel.yaml version to <= 0.15

See merge request BuildStream/buildstream!630

db0478ab

by Phillip Smyth at 2018-08-10T10:17:47Z

Implementing relative workspaces

This fixes #191

A note has been added to NEWS explaining backwards
compatibility issues

04cee9a9

by Phillip Smyth at 2018-08-10T11:12:31Z

Merge branch 'relative_workspaces' into 'master'

Patch for issue #191 support relative workspaces

Closes #191

See merge request BuildStream/buildstream!504

5dcecbad

by Valentin David at 2018-08-10T12:18:55Z

Set environment in bwrap command line instead of its environment

Fixes #498

2e8db54e

by Valentin David at 2018-08-10T12:53:19Z

Merge branch 'valentindavid/498_bwrap_environment' into 'master'

Set environment in bwrap command line instead of its environment

Closes #498

See merge request BuildStream/buildstream!565

be7b57ea

by Jonathan Maw at 2018-08-10T14:20:59Z

Add 'build-depends' and 'runtime-depends' fields to elements

80b174fe

by Jonathan Maw at 2018-08-10T14:20:59Z

tests: Add tests for loading builddeps and runtime deps

b733e10a

by Jonathan Maw at 2018-08-10T14:20:59Z

versions: Bump format version to 14

Format version raised because of a change to the core element format.

670fb008

by Jonathan Maw at 2018-08-10T14:20:59Z

docs: Add documentation of build-depends and runtime-depends fields

23 changed files:

NEWS
buildstream/_frontend/widget.py
buildstream/_fuse/fuse.py
buildstream/_fuse/hardlinks.py
buildstream/_loader/loadelement.py
buildstream/_loader/types.py
buildstream/_project.py
buildstream/_stream.py
buildstream/_versions.py
buildstream/_workspaces.py
buildstream/element.py
buildstream/sandbox/_sandboxbwrap.py
doc/source/format_declaring.rst
setup.py
tests/frontend/workspace.py
tests/loader/dependencies.py
+ tests/loader/dependencies/elements/builddep-list.bst
tests/loader/dependencies/elements/firstdep.bst
+ tests/loader/dependencies/elements/list-combine.bst
+ tests/loader/dependencies/elements/list-overlap.bst
+ tests/loader/dependencies/elements/runtimedep-list.bst
+ tests/loader/dependencies/elements/seconddep.bst
+ tests/loader/dependencies/elements/thirddep.bst

Changes:

NEWS

@@ -17,6 +17,10 @@ buildstream 1.1.5
      to not be scheduled and fail during artifact assembly,
      and display the retry prompt during an interactive session.
 +  o Due to enabling the use of relative workspaces, "Legacy" workspaces
 +    may need to be closed and remade before the changes will affect them.
 +    Downgrading after using this feature may result in workspaces
 +    not functioning correctly
  =================
  buildstream 1.1.4

buildstream/_frontend/widget.py

@@ -418,7 +418,9 @@ class LogLine(Widget):
              if "%{workspace-dirs" in format_:
                  workspace = element._get_workspace()
                  if workspace is not None:
 -                    path = workspace.path.replace(os.getenv('HOME', '/root'), '~')
 +                    path = workspace.get_absolute_path()
 +                    if path.startswith("~/"):
 +                        path = os.path.join(os.getenv('HOME', '/root'), path[2:])
                      line = p.fmt_subst(line, 'workspace-dirs', "Workspace: {}".format(path))
                  else:
                      line = p.fmt_subst(

buildstream/_fuse/fuse.py

@@ -757,7 +757,11 @@ class FUSE(object):
          if self.raw_fi:
              return self.operations('create', path, mode, fi)
          else:
 -            fi.fh = self.operations('create', path, mode)
 +            # This line is different from upstream to fix issues
 +            # reading file opened with O_CREAT|O_RDWR.
 +            # See issue #143.
 +            fi.fh = self.operations('create', path, mode, fi.flags)
 +            # END OF MODIFICATION
              return 0
      def ftruncate(self, path, length, fip):

buildstream/_fuse/hardlinks.py

@@ -185,12 +185,12 @@ class SafeHardlinkOps(Operations):
          return os.open(full_path, flags)
 -    def create(self, path, mode, fi=None):
 +    def create(self, path, mode, flags):
          full_path = self._full_path(path)
          # If it already exists, ensure it's a copy first
          self._ensure_copy(full_path)
 -        return os.open(full_path, os.O_WRONLY | os.O_CREAT, mode)
 +        return os.open(full_path, flags, mode)
      def read(self, path, length, offset, fh):
          os.lseek(fh, offset, os.SEEK_SET)

buildstream/_loader/loadelement.py

@@ -71,6 +71,7 @@ class LoadElement():
              'kind', 'depends', 'sources', 'sandbox',
              'variables', 'environment', 'environment-nocache',
              'config', 'public', 'description',
 +            'build-depends', 'runtime-depends',
          ])
          # Extract the Dependencies
@@ -127,28 +128,44 @@ class LoadElement():
  # Returns:
  #    (list): a list of Dependency objects
+ #
 -def _extract_depends_from_node(node):
 -    depends = _yaml.node_get(node, list, Symbol.DEPENDS, default_value=[])
 +def _extract_depends_from_node(node, *, key=None):
 +    if key is None:
 +        build_depends = _extract_depends_from_node(node, key=Symbol.BUILD_DEPENDS)
 +        runtime_depends = _extract_depends_from_node(node, key=Symbol.RUNTIME_DEPENDS)
 +        depends = _extract_depends_from_node(node, key=Symbol.DEPENDS)
 +        return build_depends + runtime_depends + depends
 +    elif key == Symbol.BUILD_DEPENDS:
 +        default_dep_type = Symbol.BUILD
 +    elif key == Symbol.RUNTIME_DEPENDS:
 +        default_dep_type = Symbol.RUNTIME
 +    elif key == Symbol.DEPENDS:
 +        default_dep_type = None
++
 +    depends = _yaml.node_get(node, list, key, default_value=[])
      output_deps = []
      for dep in depends:
 -        dep_provenance = _yaml.node_get_provenance(node, key=Symbol.DEPENDS, indices=[depends.index(dep)])
 +        dep_provenance = _yaml.node_get_provenance(node, key=key, indices=[depends.index(dep)])
          if isinstance(dep, str):
 -            dependency = Dependency(dep, provenance=dep_provenance)
 +            dependency = Dependency(dep, provenance=dep_provenance, dep_type=default_dep_type)
          elif isinstance(dep, Mapping):
 -            _yaml.node_validate(dep, ['filename', 'type', 'junction'])
+-
 -            # Make type optional, for this we set it to None
 -            dep_type = _yaml.node_get(dep, str, Symbol.TYPE, default_value=None)
 -            if dep_type is None or dep_type == Symbol.ALL:
 -                dep_type = None
 -            elif dep_type not in [Symbol.BUILD, Symbol.RUNTIME]:
 -                provenance = _yaml.node_get_provenance(dep, key=Symbol.TYPE)
 -                raise LoadError(LoadErrorReason.INVALID_DATA,
 -                                "{}: Dependency type '{}' is not 'build', 'runtime' or 'all'"
 -                                .format(provenance, dep_type))
 +            if default_dep_type:
 +                _yaml.node_validate(dep, ['filename', 'junction'])
 +                dep_type = default_dep_type
 +            else:
 +                _yaml.node_validate(dep, ['filename', 'type', 'junction'])
++
 +                # Make type optional, for this we set it to None
 +                dep_type = _yaml.node_get(dep, str, Symbol.TYPE, default_value=None)
 +                if dep_type is None or dep_type == Symbol.ALL:
 +                    dep_type = None
 +                elif dep_type not in [Symbol.BUILD, Symbol.RUNTIME]:
 +                    provenance = _yaml.node_get_provenance(dep, key=Symbol.TYPE)
 +                    raise LoadError(LoadErrorReason.INVALID_DATA,
 +                                    "{}: Dependency type '{}' is not 'build', 'runtime' or 'all'"
 +                                    .format(provenance, dep_type))
              filename = _yaml.node_get(dep, str, Symbol.FILENAME)
              junction = _yaml.node_get(dep, str, Symbol.JUNCTION, default_value=None)
@@ -159,13 +176,13 @@ def _extract_depends_from_node(node):
          else:
              index = depends.index(dep)
 -            p = _yaml.node_get_provenance(node, key=Symbol.DEPENDS, indices=[index])
 +            p = _yaml.node_get_provenance(node, key=key, indices=[index])
              raise LoadError(LoadErrorReason.INVALID_DATA,
                              "{}: Dependency is not specified as a string or a dictionary".format(p))
          output_deps.append(dependency)
 -    # Now delete "depends", we dont want it anymore
 -    del node[Symbol.DEPENDS]
 +    # Now delete the field, we dont want it anymore
 +    del node[key]
      return output_deps

buildstream/_loader/types.py

@@ -26,6 +26,8 @@ class Symbol():
      FILENAME = "filename"
      KIND = "kind"
      DEPENDS = "depends"
 +    BUILD_DEPENDS = "build-depends"
 +    RUNTIME_DEPENDS = "runtime-depends"
      SOURCES = "sources"
      CONFIG = "config"
      VARIABLES = "variables"

buildstream/_project.py

@@ -19,7 +19,6 @@
  #        Tiago Gomes <tiago gomes codethink co uk>
  import os
 -import multiprocessing  # for cpu_count()
  from collections import Mapping, OrderedDict
  from pluginbase import PluginBase
  from . import utils
@@ -572,7 +571,10 @@ class Project():
          # Extend variables with automatic variables and option exports
          # Initialize it as a string as all variables are processed as strings.
 -        output.base_variables['max-jobs'] = str(multiprocessing.cpu_count())
 +        # Based on some testing (mainly on AWS), maximum effective
 +        # max-jobs value seems to be around 8-10 if we have enough cores
 +        # users should set values based on workload and build infrastructure
 +        output.base_variables['max-jobs'] = str(min(len(os.sched_getaffinity(0)), 8))
          # Export options into variables, if that was requested
          output.options.export_variables(output.base_variables)

buildstream/_stream.py

@@ -460,7 +460,7 @@ class Stream():
                                                selection=PipelineSelection.REDIRECT,
                                                track_selection=PipelineSelection.REDIRECT)
          target = elements[0]
 -        workdir = os.path.abspath(directory)
 +        directory = os.path.abspath(directory)
          if not list(target.sources()):
              build_depends = [x.name for x in target.dependencies(Scope.BUILD, recurse=False)]
@@ -476,7 +476,7 @@ class Stream():
          workspace = workspaces.get_workspace(target._get_full_name())
          if workspace and not force:
              raise StreamError("Workspace '{}' is already defined at: {}"
 -                              .format(target.name, workspace.path))
 +                              .format(target.name, workspace.get_absolute_path()))
          # If we're going to checkout, we need at least a fetch,
          # if we were asked to track first, we're going to fetch anyway.
@@ -502,7 +502,7 @@ class Stream():
          except OSError as e:
              raise StreamError("Failed to create workspace directory: {}".format(e)) from e
 -        workspaces.create_workspace(target._get_full_name(), workdir)
 +        workspaces.create_workspace(target._get_full_name(), directory)
          if not no_checkout:
              with target.timed_activity("Staging sources to {}".format(directory)):
@@ -526,12 +526,12 @@ class Stream():
          # Remove workspace directory if prompted
          if remove_dir:
              with self._context.timed_activity("Removing workspace directory {}"
 -                                              .format(workspace.path)):
 +                                              .format(workspace.get_absolute_path())):
                  try:
 -                    shutil.rmtree(workspace.path)
 +                    shutil.rmtree(workspace.get_absolute_path())
                  except OSError as e:
                      raise StreamError("Could not remove  '{}': {}"
 -                                      .format(workspace.path, e)) from e
 +                                      .format(workspace.get_absolute_path(), e)) from e
          # Delete the workspace and save the configuration
          workspaces.delete_workspace(element_name)
@@ -574,28 +574,30 @@ class Stream():
          for element in elements:
              workspace = workspaces.get_workspace(element._get_full_name())
+-
 +            workspace_path = workspace.get_absolute_path()
              if soft:
                  workspace.prepared = False
                  self._message(MessageType.INFO, "Reset workspace state for {} at: {}"
 -                              .format(element.name, workspace.path))
 +                              .format(element.name, workspace_path))
                  continue
              with element.timed_activity("Removing workspace directory {}"
 -                                        .format(workspace.path)):
 +                                        .format(workspace_path)):
                  try:
 -                    shutil.rmtree(workspace.path)
 +                    shutil.rmtree(workspace_path)
                  except OSError as e:
                      raise StreamError("Could not remove  '{}': {}"
 -                                      .format(workspace.path, e)) from e
 +                                      .format(workspace_path, e)) from e
              workspaces.delete_workspace(element._get_full_name())
 -            workspaces.create_workspace(element._get_full_name(), workspace.path)
 +            workspaces.create_workspace(element._get_full_name(), workspace_path)
 -            with element.timed_activity("Staging sources to {}".format(workspace.path)):
 +            with element.timed_activity("Staging sources to {}".format(workspace_path)):
                  element._open_workspace()
 -            self._message(MessageType.INFO, "Reset workspace for {} at: {}".format(element.name, workspace.path))
 +            self._message(MessageType.INFO,
 +                          "Reset workspace for {} at: {}".format(element.name,
 +                                                                 workspace_path))
          workspaces.save_config()
@@ -632,7 +634,7 @@ class Stream():
          for element_name, workspace_ in self._context.get_workspaces().list():
              workspace_detail = {
                  'element': element_name,
 -                'directory': workspace_.path,
 +                'directory': workspace_.get_absolute_path(),
+             }
              workspaces.append(workspace_detail)

buildstream/_versions.py

@@ -23,7 +23,7 @@
  # This version is bumped whenever enhancements are made
  # to the `project.conf` format or the core element format.
+ #
 -BST_FORMAT_VERSION = 13
 +BST_FORMAT_VERSION = 14
  # The base BuildStream artifact version

buildstream/_workspaces.py

@@ -26,14 +26,6 @@ from ._exceptions import LoadError, LoadErrorReason
  BST_WORKSPACE_FORMAT_VERSION = 3
 -# Hold on to a list of members which get serialized
 -_WORKSPACE_MEMBERS = [
 -    'prepared',
 -    'path',
 -    'last_successful',
 -    'running_files'
 -]
+-
  # Workspace()
+ #
@@ -56,7 +48,7 @@ class Workspace():
      def __init__(self, toplevel_project, *, last_successful=None, path=None, prepared=False, running_files=None):
          self.prepared = prepared
          self.last_successful = last_successful
 -        self.path = path
 +        self._path = path
          self.running_files = running_files if running_files is not None else {}
          self._toplevel_project = toplevel_project
@@ -64,14 +56,20 @@ class Workspace():
      # to_dict()
+     #
 -    # Convert this object to a dict for serialization purposes
 +    # Convert a list of members which get serialized to a dict for serialization purposes
+     #
      # Returns:
      #     (dict) A dict representation of the workspace
+     #
      def to_dict(self):
 -        return {key: val for key, val in self.__dict__.items()
 -                if key in _WORKSPACE_MEMBERS and val is not None}
 +        ret = {
 +            'prepared': self.prepared,
 +            'path': self._path,
 +            'running_files': self.running_files
 +        }
 +        if self.last_successful is not None:
 +            ret["last_successful"] = self.last_successful
 +        return ret
      # from_dict():
+     #
@@ -103,15 +101,7 @@ class Workspace():
      #    True if the workspace differs from 'other', otherwise False
+     #
      def differs(self, other):
+-
 -        for member in _WORKSPACE_MEMBERS:
 -            member_a = getattr(self, member)
 -            member_b = getattr(other, member)
+-
 -            if member_a != member_b:
 -                return True
+-
 -        return False
 +        return self.to_dict() != other.to_dict()
      # invalidate_key()
+     #
@@ -133,7 +123,7 @@ class Workspace():
          if os.path.isdir(fullpath):
              utils.copy_files(fullpath, directory)
          else:
 -            destfile = os.path.join(directory, os.path.basename(self.path))
 +            destfile = os.path.join(directory, os.path.basename(self.get_absolute_path()))
              utils.safe_copy(fullpath, destfile)
      # add_running_files()
@@ -189,7 +179,7 @@ class Workspace():
                  filelist = utils.list_relative_paths(fullpath)
                  filelist = [(relpath, os.path.join(fullpath, relpath)) for relpath in filelist]
              else:
 -                filelist = [(self.path, fullpath)]
 +                filelist = [(self.get_absolute_path(), fullpath)]
              self._key = [(relpath, unique_key(fullpath)) for relpath, fullpath in filelist]
@@ -200,7 +190,7 @@ class Workspace():
      # Returns: The absolute path of the element's workspace.
+     #
      def get_absolute_path(self):
 -        return os.path.join(self._toplevel_project.directory, self.path)
 +        return os.path.join(self._toplevel_project.directory, self._path)
  # Workspaces()
@@ -236,6 +226,9 @@ class Workspaces():
      #    path (str) - The path in which the workspace should be kept
+     #
      def create_workspace(self, element_name, path):
 +        if path.startswith(self._toplevel_project.directory):
 +            path = os.path.relpath(path, self._toplevel_project.directory)
++
          self._workspaces[element_name] = Workspace(self._toplevel_project, path=path)
          return self._workspaces[element_name]

buildstream/element.py

@@ -1403,7 +1403,8 @@ class Element(Plugin):
                      # If mount_workspaces is set and we're doing incremental builds,
                      # the workspace is already mounted into the sandbox.
                      if not (mount_workspaces and self.__can_build_incrementally()):
 -                        with self.timed_activity("Staging local files at {}".format(workspace.path)):
 +                        with self.timed_activity("Staging local files at {}"
 +                                                 .format(workspace.get_absolute_path())):
                              workspace.stage(temp_staging_directory)
                  else:
                      # No workspace, stage directly
@@ -1566,7 +1567,7 @@ class Element(Plugin):
                          path_components = self.__staged_sources_directory.lstrip(os.sep).split(os.sep)
                          sandbox_vpath = sandbox_vroot.descend(path_components)
                          try:
 -                            sandbox_vpath.import_files(workspace.path)
 +                            sandbox_vpath.import_files(workspace.get_absolute_path())
                          except UtilError as e:
                              self.warn("Failed to preserve workspace state for failed build sysroot: {}"
                                        .format(e))
@@ -1893,7 +1894,7 @@ class Element(Plugin):
                  source._init_workspace(temp)
              # Now hardlink the files into the workspace target.
 -            utils.link_files(temp, workspace.path)
 +            utils.link_files(temp, workspace.get_absolute_path())
      # _get_workspace():
+     #

buildstream/sandbox/_sandboxbwrap.py

@@ -89,6 +89,11 @@ class SandboxBwrap(Sandbox):
          # Grab the full path of the bwrap binary
          bwrap_command = [utils.get_host_tool('bwrap')]
 +        for k, v in env.items():
 +            bwrap_command += ['--setenv', k, v]
 +        for k in os.environ.keys() - env.keys():
 +            bwrap_command += ['--unsetenv', k]
++
          # Create a new pid namespace, this also ensures that any subprocesses
          # are cleaned up when the bwrap process exits.
          bwrap_command += ['--unshare-pid']
@@ -194,7 +199,7 @@ class SandboxBwrap(Sandbox):
                  stdin = stack.enter_context(open(os.devnull, "r"))
              # Run bubblewrap !
 -            exit_code = self.run_bwrap(bwrap_command, stdin, stdout, stderr, env,
 +            exit_code = self.run_bwrap(bwrap_command, stdin, stdout, stderr,
                                         (flags & SandboxFlags.INTERACTIVE))
              # Cleanup things which bwrap might have left behind, while
@@ -245,7 +250,7 @@ class SandboxBwrap(Sandbox):
          return exit_code
 -    def run_bwrap(self, argv, stdin, stdout, stderr, env, interactive):
 +    def run_bwrap(self, argv, stdin, stdout, stderr, interactive):
          # Wrapper around subprocess.Popen() with common settings.
+         #
          # This function blocks until the subprocess has terminated.
@@ -321,7 +326,6 @@ class SandboxBwrap(Sandbox):
                  # The default is to share file descriptors from the parent process
                  # to the subprocess, which is rarely good for sandboxing.
                  close_fds=True,
 -                env=env,
                  stdin=stdin,
                  stdout=stdout,
                  stderr=stderr,

doc/source/format_declaring.rst

@@ -98,6 +98,68 @@ relative filename to the elements they depend on here.
  See :ref:`format_dependencies` for more information on the dependency model.
 +.. _format_build_depends:
++
 +Build-Depends
 +~~~~~~~~~~~~~
++
 +.. code:: yaml
++
 +   # Specify some build-dependencies
 +   build-depends:
 +   - element1.bst
 +   - element2.bst
++
 +Build dependencies between elements can be specified with the ``build-depends`` attribute.
 +The above code snippet is equivalent to:
++
 +.. code:: yaml
++
 +   # Specify some build-dependencies
 +   depends:
 +   - filename: element1.bst
 +     type: build
 +   - filename: element2.bst
 +     type: build
++
 +See :ref:`format_dependencies` for more information on the dependency model.
++
 +.. note::
++
 +   The ``build-depends`` configuration is available since :ref:`format version 14 <project_format_version>`
++
++
 +.. _format_runtime_depends:
++
 +Runtime-Depends
 +~~~~~~~~~~~~~~~
++
 +.. code:: yaml
++
 +   # Specify some runtime-dependencies
 +   runtime-depends:
 +   - element1.bst
 +   - element2.bst
++
 +Runtime dependencies between elements can be specified with the ``runtime-depends`` attribute.
 +The above code snippet is equivalent to:
++
 +.. code:: yaml
++
 +   # Specify some runtime-dependencies
 +   depends:
 +   - filename: element1.bst
 +     type: runtime
 +   - filename: element2.bst
 +     type: runtime
++
 +See :ref:`format_dependencies` for more information on the dependency model.
++
 +.. note::
++
 +   The ``runtime-depends`` configuration is available since :ref:`format version 14 <project_format_version>`
++
++
  .. _format_sources:
  Sources
@@ -276,8 +338,8 @@ attributes are suitable.
  .. note::
 -   Note the order in which element dependencies are declared in the ``depends``
 -   list is not meaningful.
 +   Note the order in which element dependencies are declared in the ``depends``,
 +   ``build-depends`` and ``runtime-depends`` lists are not meaningful.
  Dependency dictionary:
@@ -299,6 +361,8 @@ Attributes:
  * ``type``
    This attribute is used to express the :ref:`dependency type <format_dependencies_types>`.
 +  This field is not permitted in :ref:`Build-Depends <format_build_depends>` or
 +  :ref:`Runtime-Depends <format_runtime_depends>`.
  * ``junction``

setup.py

@@ -251,7 +251,7 @@ setup(name='BuildStream',
        install_requires=[
            'setuptools',
            'psutil',
 -          'ruamel.yaml',
 +          'ruamel.yaml <= 0.15',
            'pluginbase',
            'Click',
            'blessings',

tests/frontend/workspace.py

@@ -18,12 +18,13 @@ DATA_DIR = os.path.join(
+ )
 -def open_workspace(cli, tmpdir, datafiles, kind, track, suffix=''):
 -    project = os.path.join(datafiles.dirname, datafiles.basename)
 -    bin_files_path = os.path.join(project, 'files', 'bin-files')
 -    element_path = os.path.join(project, 'elements')
 +def open_workspace(cli, tmpdir, datafiles, kind, track, suffix='', workspace_dir=None):
 +    if not workspace_dir:
 +        workspace_dir = os.path.join(str(tmpdir), 'workspace{}'.format(suffix))
 +    project_path = os.path.join(datafiles.dirname, datafiles.basename)
 +    bin_files_path = os.path.join(project_path, 'files', 'bin-files')
 +    element_path = os.path.join(project_path, 'elements')
      element_name = 'workspace-test-{}{}.bst'.format(kind, suffix)
 -    workspace = os.path.join(str(tmpdir), 'workspace{}'.format(suffix))
      # Create our repo object of the given source type with
      # the bin files, and then collect the initial ref.
@@ -45,7 +46,7 @@ def open_workspace(cli, tmpdir, datafiles, kind, track, suffix=''):
                              element_name))
      # Assert that there is no reference, a track & fetch is needed
 -    state = cli.get_element_state(project, element_name)
 +    state = cli.get_element_state(project_path, element_name)
      if track:
          assert state == 'no reference'
      else:
@@ -56,20 +57,20 @@ def open_workspace(cli, tmpdir, datafiles, kind, track, suffix=''):
      args = ['workspace', 'open']
      if track:
          args.append('--track')
 -    args.extend([element_name, workspace])
 +    args.extend([element_name, workspace_dir])
 +    result = cli.run(project=project_path, args=args)
 -    result = cli.run(project=project, args=args)
      result.assert_success()
      # Assert that we are now buildable because the source is
      # now cached.
 -    assert cli.get_element_state(project, element_name) == 'buildable'
 +    assert cli.get_element_state(project_path, element_name) == 'buildable'
      # Check that the executable hello file is found in the workspace
 -    filename = os.path.join(workspace, 'usr', 'bin', 'hello')
 +    filename = os.path.join(workspace_dir, 'usr', 'bin', 'hello')
      assert os.path.exists(filename)
 -    return (element_name, project, workspace)
 +    return (element_name, project_path, workspace_dir)
  @pytest.mark.datafiles(DATA_DIR)
@@ -190,6 +191,46 @@ def test_close(cli, tmpdir, datafiles, kind):
      assert not os.path.exists(workspace)
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_close_external_after_move_project(cli, tmpdir, datafiles):
 +    tmp_parent = os.path.dirname(str(tmpdir))
 +    workspace_dir = os.path.join(tmp_parent, "workspace")
 +    element_name, project_path, _ = open_workspace(cli, tmpdir, datafiles, 'git', False, "", workspace_dir)
 +    assert os.path.exists(workspace_dir)
 +    tmp_dir = os.path.join(tmp_parent, 'external_project')
 +    shutil.move(project_path, tmp_dir)
 +    assert os.path.exists(tmp_dir)
++
 +    # Close the workspace
 +    result = cli.run(configure=False, project=tmp_dir, args=[
 +        'workspace', 'close', '--remove-dir', element_name
 +    ])
 +    result.assert_success()
++
 +    # Assert the workspace dir has been deleted
 +    assert not os.path.exists(workspace_dir)
 +    # Move directory back inside tmp directory so it can be recognised
 +    shutil.move(tmp_dir, project_path)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_close_internal_after_move_project(cli, tmpdir, datafiles):
 +    element_name, project, _ = open_workspace(cli, tmpdir, datafiles, 'git', False)
 +    tmp_dir = os.path.join(os.path.dirname(str(tmpdir)), 'external_project')
 +    shutil.move(str(tmpdir), tmp_dir)
 +    assert os.path.exists(tmp_dir)
++
 +    # Close the workspace
 +    result = cli.run(configure=False, project=tmp_dir, args=[
 +        'workspace', 'close', '--remove-dir', element_name
 +    ])
 +    result.assert_success()
++
 +    # Assert the workspace dir has been deleted
 +    workspace = os.path.join(tmp_dir, 'workspace')
 +    assert not os.path.exists(workspace)
++
++
  @pytest.mark.datafiles(DATA_DIR)
  def test_close_removed(cli, tmpdir, datafiles):
      element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, 'git', False)

tests/loader/dependencies.py

@@ -3,6 +3,7 @@ import pytest
  from buildstream._exceptions import LoadError, LoadErrorReason
  from buildstream._loader import Loader, MetaElement
 +from tests.testutils import cli
  from . import make_loader
  DATA_DIR = os.path.join(
@@ -27,7 +28,7 @@ def test_two_files(datafiles):
      assert(len(element.dependencies) == 1)
      firstdep = element.dependencies[0]
      assert(isinstance(firstdep, MetaElement))
 -    assert(firstdep.kind == 'thefirstdep')
 +    assert(firstdep.kind == 'manual')
  @pytest.mark.datafiles(DATA_DIR)
@@ -47,7 +48,7 @@ def test_shared_dependency(datafiles):
+     #
      firstdep = element.dependencies[0]
      assert(isinstance(firstdep, MetaElement))
 -    assert(firstdep.kind == 'thefirstdep')
 +    assert(firstdep.kind == 'manual')
      assert(len(firstdep.dependencies) == 0)
      # The second specified dependency is 'shareddep'
@@ -86,7 +87,7 @@ def test_dependency_dict(datafiles):
      assert(len(element.dependencies) == 1)
      firstdep = element.dependencies[0]
      assert(isinstance(firstdep, MetaElement))
 -    assert(firstdep.kind == 'thefirstdep')
 +    assert(firstdep.kind == 'manual')
  @pytest.mark.datafiles(DATA_DIR)
@@ -186,3 +187,49 @@ def test_all_dependency(datafiles):
      assert(isinstance(firstdep, MetaElement))
      firstbuilddep = element.build_dependencies[0]
      assert(firstdep == firstbuilddep)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_list_build_dependency(cli, datafiles):
 +    project = str(datafiles)
++
 +    # Check that the pipeline includes the build dependency
 +    deps = cli.get_pipeline(project, ['elements/builddep-list.bst'], scope="build")
 +    assert "elements/firstdep.bst" in deps
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_list_runtime_dependency(cli, datafiles):
 +    project = str(datafiles)
++
 +    # Check that the pipeline includes the runtime dependency
 +    deps = cli.get_pipeline(project, ['elements/runtimedep-list.bst'], scope="run")
 +    assert "elements/firstdep.bst" in deps
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_list_dependencies_combined(cli, datafiles):
 +    project = str(datafiles)
++
 +    # Check that runtime deps get combined
 +    rundeps = cli.get_pipeline(project, ['elements/list-combine.bst'], scope="run")
 +    assert "elements/firstdep.bst" not in rundeps
 +    assert "elements/seconddep.bst" in rundeps
 +    assert "elements/thirddep.bst" in rundeps
++
 +    # Check that build deps get combined
 +    builddeps = cli.get_pipeline(project, ['elements/list-combine.bst'], scope="build")
 +    assert "elements/firstdep.bst" in builddeps
 +    assert "elements/seconddep.bst" not in builddeps
 +    assert "elements/thirddep.bst" in builddeps
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_list_overlap(cli, datafiles):
 +    project = str(datafiles)
++
 +    # Check that dependencies get merged
 +    rundeps = cli.get_pipeline(project, ['elements/list-overlap.bst'], scope="run")
 +    assert "elements/firstdep.bst" in rundeps
 +    builddeps = cli.get_pipeline(project, ['elements/list-overlap.bst'], scope="build")
 +    assert "elements/firstdep.bst" in builddeps

tests/loader/dependencies/elements/builddep-list.bst

 +kind: stack
 +description: This element has a build-only dependency specified via build-depends
 +build-depends:
 +  - elements/firstdep.bst

tests/loader/dependencies/elements/firstdep.bst

 -kind: thefirstdep
 +kind: manual
  description: This is the first dependency

tests/loader/dependencies/elements/list-combine.bst

 +kind: stack
 +description: This element depends on three elements in different ways
 +build-depends:
 +- elements/firstdep.bst
 +runtime-depends:
 +- elements/seconddep.bst
 +depends:
 +- elements/thirddep.bst

tests/loader/dependencies/elements/list-overlap.bst

 +kind: stack
 +description: This element depends on two elements in different ways
 +build-depends:
 +- elements/firstdep.bst
 +depends:
 +- filename: elements/firstdep.bst
 +  type: runtime

tests/loader/dependencies/elements/runtimedep-list.bst

 +kind: stack
 +description: This element has a runtime-only dependency
 +runtime-depends:
 +  - elements/firstdep.bst

tests/loader/dependencies/elements/seconddep.bst

	1	+kind: manual
	2	+description: This is the second dependency

tests/loader/dependencies/elements/thirddep.bst

	1	+kind: manual
	2	+description: This is the third dependency

[Notes] [Git][BuildStream/buildstream][463-make-dependency-type-default-to-build] 15 commits: Keep original flags for create in SafeHardlinks.

Jonathan Maw pushed to branch 463-make-dependency-type-default-to-build at BuildStream / buildstream

Commits:

23 changed files:

Changes: