[Notes] [Git][BuildStream/buildstream][valentindavid/fix_included

Valentin David pushed to branch valentindavid/fix_included_junction_track at BuildStream / buildstream

Commits:

ed653fbc

by Chandan Singh at 2018-08-08T00:06:28Z

.gitlab-ci.yml: Remove unused script "install.sh"

65f382f1

by Chandan Singh at 2018-08-08T10:52:32Z

Merge branch 'chandan/remove-dead-code' into 'master'

.gitlab-ci.yml: Remove unused script "install.sh"

See merge request BuildStream/buildstream!617

c68dcab8

by Tiago Gomes at 2018-08-09T09:18:43Z

cascache: move tmp directory one level up

The CAS uses a temp directory while manipulating the cache, temporary
files can be added and removed while adding artifacts to the cache here.

Since calculation of the cache size happens in parallel to artifact
cache additions, this causes race conditions in the size calculation
job, as we end up calling `stat` on temporary files which are being
removed in parallel.

Handle this by moving the temporary directory out of the way, and avoid
considering the tmp directory when calculating the cache size

35ab0335

by Tiago Gomes at 2018-08-09T10:16:24Z

Merge branch 'tiagogomes/issue-520' into 'master'

Fix race condition when calculating disk usage

See merge request BuildStream/buildstream!600

8aa33e23

by Valentin David at 2018-08-09T12:49:17Z

Keep original flags for create in SafeHardlinks.

When open(2) is used with flags O_CREAT|O_RDWR, the file descriptor
must be readable. Unfortunately O_RDWR was not passed which made
read fail with EBADF and mmap to signal SIGBUS.

This issue happened with man-db for example.

Fixes #143.

ef7810f3

by Valentin David at 2018-08-09T14:06:50Z

Merge branch 'valentindavid/fuse-create-flags' into 'master'

Keep original flags for create in SafeHardlinks.

Closes #143

See merge request BuildStream/buildstream!624

2d061173

by Javier Jardón at 2018-08-09T16:31:15Z

buildstream/_project.py:

Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for
systems where not all CPU can be used (in particular, Docker, CI etc).

As suggested in the multiprocessing docs
(http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count)
using len(os.sched_getaffinity(0)) might be better.

4e1488ee

by Javier Jardón at 2018-08-09T16:31:15Z

buildstream/_project.py: Restrict max-jobs

even if the machine have mores cores available

Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227

de955834

by Javier Jardón at 2018-08-09T17:22:24Z

Merge branch 'jjardon/max-jobs' into 'master'

buildstream/_project.py: Restrict max-jobs

See merge request BuildStream/buildstream!620

97595585
by Jim MacArthur at 2018-08-09T17:35:19Z
```
setup.py: Pin ruamel.yaml version to <= 0.15
```

a602365c

by Tristan Van Berkom at 2018-08-09T20:03:52Z

Merge branch 'jmac/pin-ruamel-version' into 'master'

Pin ruamel.yaml version to <= 0.15

See merge request BuildStream/buildstream!630

db0478ab

by Phillip Smyth at 2018-08-10T10:17:47Z

Implementing relative workspaces

This fixes #191

A note has been added to NEWS explaining backwards
compatibility issues

04cee9a9

by Phillip Smyth at 2018-08-10T11:12:31Z

Merge branch 'relative_workspaces' into 'master'

Patch for issue #191 support relative workspaces

Closes #191

See merge request BuildStream/buildstream!504

5dcecbad

by Valentin David at 2018-08-10T12:18:55Z

Set environment in bwrap command line instead of its environment

Fixes #498

2e8db54e

by Valentin David at 2018-08-10T12:53:19Z

Merge branch 'valentindavid/498_bwrap_environment' into 'master'

Set environment in bwrap command line instead of its environment

Closes #498

See merge request BuildStream/buildstream!565

7faef88e

by Valentin David at 2018-08-10T14:44:49Z

Fix tracking of junctions used in project.conf.

Stream._load() now returns early without resolving build pipeline when
only tracking. Resolving track pipelines does not require to fully
load project configurations when when elements to track are only
junctions.

However build pipelines require to fully load project
configurations. This might not be possible in the case a project
configuration includes a file from a junction that yet needs to be
tracked.

Fixes #565.

16 changed files:

.gitlab-ci.yml
NEWS
buildstream/_artifactcache/artifactcache.py
buildstream/_artifactcache/cascache.py
buildstream/_artifactcache/casserver.py
buildstream/_frontend/widget.py
buildstream/_fuse/fuse.py
buildstream/_fuse/hardlinks.py
buildstream/_project.py
buildstream/_stream.py
buildstream/_workspaces.py
buildstream/element.py
buildstream/sandbox/_sandboxbwrap.py
setup.py
tests/frontend/track.py
tests/frontend/workspace.py

Changes:

.gitlab-ci.yml

@@ -26,15 +26,6 @@ source_dist:
    - tar -ztf dist/*
    - tarball=$(cd dist && echo $(ls *))
 -  # Create an installer script
 -  - |
 -    cat > dist/install.sh << EOF
 -    #!/bin/sh
 -    tar -zxf ${tarball}
 -    cd ${tarball%.tar.gz}
 -    pip3 install --no-index .
 -    EOF
+-
    # unpack tarball as `dist/buildstream` directory
    - |
      cat > dist/unpack.sh << EOF
@@ -44,7 +35,6 @@ source_dist:
      EOF
    # Make our helpers executable
 -  - chmod +x dist/install.sh
    - chmod +x dist/unpack.sh
    artifacts:
      paths:

NEWS

@@ -17,6 +17,10 @@ buildstream 1.1.5
      to not be scheduled and fail during artifact assembly,
      and display the retry prompt during an interactive session.
 +  o Due to enabling the use of relative workspaces, "Legacy" workspaces
 +    may need to be closed and remade before the changes will affect them.
 +    Downgrading after using this feature may result in workspaces
 +    not functioning correctly
  =================
  buildstream 1.1.4

buildstream/_artifactcache/artifactcache.py

@@ -80,6 +80,8 @@ class ArtifactCache():
          self.context = context
          self.required_artifacts = set()
          self.extractdir = os.path.join(context.artifactdir, 'extract')
 +        self.tmpdir = os.path.join(context.artifactdir, 'tmp')
++
          self.max_size = context.cache_quota
          self.estimated_size = None
@@ -89,7 +91,8 @@ class ArtifactCache():
          self._local = False
          self.cache_size = None
 -        os.makedirs(context.artifactdir, exist_ok=True)
 +        os.makedirs(self.extractdir, exist_ok=True)
 +        os.makedirs(self.tmpdir, exist_ok=True)
      ################################################
      #  Methods implemented on the abstract class   #

buildstream/_artifactcache/cascache.py

@@ -56,7 +56,8 @@ class CASCache(ArtifactCache):
          super().__init__(context)
          self.casdir = os.path.join(context.artifactdir, 'cas')
 -        os.makedirs(os.path.join(self.casdir, 'tmp'), exist_ok=True)
 +        os.makedirs(os.path.join(self.casdir, 'refs', 'heads'), exist_ok=True)
 +        os.makedirs(os.path.join(self.casdir, 'objects'), exist_ok=True)
          self._enable_push = enable_push
@@ -85,8 +86,6 @@ class CASCache(ArtifactCache):
              # artifact has already been extracted
              return dest
 -        os.makedirs(self.extractdir, exist_ok=True)
+-
          with tempfile.TemporaryDirectory(prefix='tmp', dir=self.extractdir) as tmpdir:
              checkoutdir = os.path.join(tmpdir, ref)
              self._checkout(checkoutdir, tree)
@@ -394,7 +393,7 @@ class CASCache(ArtifactCache):
          try:
              h = hashlib.sha256()
              # Always write out new file to avoid corruption if input file is modified
 -            with tempfile.NamedTemporaryFile(dir=os.path.join(self.casdir, 'tmp')) as out:
 +            with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:
                  # Set mode bits to 0644
                  os.chmod(out.name, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH)
@@ -764,7 +763,7 @@ class CASCache(ArtifactCache):
              # already in local cache
              return
 -        with tempfile.NamedTemporaryFile(dir=os.path.join(self.casdir, 'tmp')) as out:
 +        with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:
              self._fetch_blob(remote, tree, out)
              directory = remote_execution_pb2.Directory()
@@ -778,7 +777,7 @@ class CASCache(ArtifactCache):
                      # already in local cache
                      continue
 -                with tempfile.NamedTemporaryFile(dir=os.path.join(self.casdir, 'tmp')) as f:
 +                with tempfile.NamedTemporaryFile(dir=self.tmpdir) as f:
                      self._fetch_blob(remote, filenode.digest, f)
                      digest = self.add_object(path=f.name)

buildstream/_artifactcache/casserver.py

@@ -161,7 +161,7 @@ class _ByteStreamServicer(bytestream_pb2_grpc.ByteStreamServicer):
          offset = 0
          finished = False
          resource_name = None
 -        with tempfile.NamedTemporaryFile(dir=os.path.join(self.cas.casdir, 'tmp')) as out:
 +        with tempfile.NamedTemporaryFile(dir=self.cas.tmpdir) as out:
              for request in request_iterator:
                  assert not finished
                  assert request.write_offset == offset

buildstream/_frontend/widget.py

@@ -418,7 +418,9 @@ class LogLine(Widget):
              if "%{workspace-dirs" in format_:
                  workspace = element._get_workspace()
                  if workspace is not None:
 -                    path = workspace.path.replace(os.getenv('HOME', '/root'), '~')
 +                    path = workspace.get_absolute_path()
 +                    if path.startswith("~/"):
 +                        path = os.path.join(os.getenv('HOME', '/root'), path[2:])
                      line = p.fmt_subst(line, 'workspace-dirs', "Workspace: {}".format(path))
                  else:
                      line = p.fmt_subst(

buildstream/_fuse/fuse.py

@@ -757,7 +757,11 @@ class FUSE(object):
          if self.raw_fi:
              return self.operations('create', path, mode, fi)
          else:
 -            fi.fh = self.operations('create', path, mode)
 +            # This line is different from upstream to fix issues
 +            # reading file opened with O_CREAT|O_RDWR.
 +            # See issue #143.
 +            fi.fh = self.operations('create', path, mode, fi.flags)
 +            # END OF MODIFICATION
              return 0
      def ftruncate(self, path, length, fip):

buildstream/_fuse/hardlinks.py

@@ -185,12 +185,12 @@ class SafeHardlinkOps(Operations):
          return os.open(full_path, flags)
 -    def create(self, path, mode, fi=None):
 +    def create(self, path, mode, flags):
          full_path = self._full_path(path)
          # If it already exists, ensure it's a copy first
          self._ensure_copy(full_path)
 -        return os.open(full_path, os.O_WRONLY | os.O_CREAT, mode)
 +        return os.open(full_path, flags, mode)
      def read(self, path, length, offset, fh):
          os.lseek(fh, offset, os.SEEK_SET)

buildstream/_project.py

@@ -19,7 +19,6 @@
  #        Tiago Gomes <tiago gomes codethink co uk>
  import os
 -import multiprocessing  # for cpu_count()
  from collections import Mapping, OrderedDict
  from pluginbase import PluginBase
  from . import utils
@@ -572,7 +571,10 @@ class Project():
          # Extend variables with automatic variables and option exports
          # Initialize it as a string as all variables are processed as strings.
 -        output.base_variables['max-jobs'] = str(multiprocessing.cpu_count())
 +        # Based on some testing (mainly on AWS), maximum effective
 +        # max-jobs value seems to be around 8-10 if we have enough cores
 +        # users should set values based on workload and build infrastructure
 +        output.base_variables['max-jobs'] = str(min(len(os.sched_getaffinity(0)), 8))
          # Export options into variables, if that was requested
          output.options.export_variables(output.base_variables)

buildstream/_stream.py

@@ -267,8 +267,11 @@ class Stream():
                except_targets=None,
                cross_junctions=False):
 +        # We pass no target to build. Only to track. Passing build targets
 +        # would fully load project configuration which might not be
 +        # possible before tracking is done.
          _, elements = \
 -            self._load(targets, targets,
 +            self._load([], targets,
                         selection=selection, track_selection=selection,
                         except_targets=except_targets,
                         track_except_targets=except_targets,
@@ -460,7 +463,7 @@ class Stream():
                                                selection=PipelineSelection.REDIRECT,
                                                track_selection=PipelineSelection.REDIRECT)
          target = elements[0]
 -        workdir = os.path.abspath(directory)
 +        directory = os.path.abspath(directory)
          if not list(target.sources()):
              build_depends = [x.name for x in target.dependencies(Scope.BUILD, recurse=False)]
@@ -476,7 +479,7 @@ class Stream():
          workspace = workspaces.get_workspace(target._get_full_name())
          if workspace and not force:
              raise StreamError("Workspace '{}' is already defined at: {}"
 -                              .format(target.name, workspace.path))
 +                              .format(target.name, workspace.get_absolute_path()))
          # If we're going to checkout, we need at least a fetch,
          # if we were asked to track first, we're going to fetch anyway.
@@ -502,7 +505,7 @@ class Stream():
          except OSError as e:
              raise StreamError("Failed to create workspace directory: {}".format(e)) from e
 -        workspaces.create_workspace(target._get_full_name(), workdir)
 +        workspaces.create_workspace(target._get_full_name(), directory)
          if not no_checkout:
              with target.timed_activity("Staging sources to {}".format(directory)):
@@ -526,12 +529,12 @@ class Stream():
          # Remove workspace directory if prompted
          if remove_dir:
              with self._context.timed_activity("Removing workspace directory {}"
 -                                              .format(workspace.path)):
 +                                              .format(workspace.get_absolute_path())):
                  try:
 -                    shutil.rmtree(workspace.path)
 +                    shutil.rmtree(workspace.get_absolute_path())
                  except OSError as e:
                      raise StreamError("Could not remove  '{}': {}"
 -                                      .format(workspace.path, e)) from e
 +                                      .format(workspace.get_absolute_path(), e)) from e
          # Delete the workspace and save the configuration
          workspaces.delete_workspace(element_name)
@@ -574,28 +577,30 @@ class Stream():
          for element in elements:
              workspace = workspaces.get_workspace(element._get_full_name())
+-
 +            workspace_path = workspace.get_absolute_path()
              if soft:
                  workspace.prepared = False
                  self._message(MessageType.INFO, "Reset workspace state for {} at: {}"
 -                              .format(element.name, workspace.path))
 +                              .format(element.name, workspace_path))
                  continue
              with element.timed_activity("Removing workspace directory {}"
 -                                        .format(workspace.path)):
 +                                        .format(workspace_path)):
                  try:
 -                    shutil.rmtree(workspace.path)
 +                    shutil.rmtree(workspace_path)
                  except OSError as e:
                      raise StreamError("Could not remove  '{}': {}"
 -                                      .format(workspace.path, e)) from e
 +                                      .format(workspace_path, e)) from e
              workspaces.delete_workspace(element._get_full_name())
 -            workspaces.create_workspace(element._get_full_name(), workspace.path)
 +            workspaces.create_workspace(element._get_full_name(), workspace_path)
 -            with element.timed_activity("Staging sources to {}".format(workspace.path)):
 +            with element.timed_activity("Staging sources to {}".format(workspace_path)):
                  element._open_workspace()
 -            self._message(MessageType.INFO, "Reset workspace for {} at: {}".format(element.name, workspace.path))
 +            self._message(MessageType.INFO,
 +                          "Reset workspace for {} at: {}".format(element.name,
 +                                                                 workspace_path))
          workspaces.save_config()
@@ -632,7 +637,7 @@ class Stream():
          for element_name, workspace_ in self._context.get_workspaces().list():
              workspace_detail = {
                  'element': element_name,
 -                'directory': workspace_.path,
 +                'directory': workspace_.get_absolute_path(),
+             }
              workspaces.append(workspace_detail)
@@ -822,6 +827,12 @@ class Stream():
+     #
      # A convenience method for loading element lists
+     #
 +    # If `targets` is not empty used project configuration will be
 +    # fully loaded. If `targets` is empty, tracking will still be
 +    # resolved for elements in `track_targets`, but no build pipeline
 +    # will be resolved. This is behavior is import for track() to
 +    # not trigger full loading of project configuration.
 +    #
      # Args:
      #    targets (list of str): Main targets to load
      #    track_targets (list of str): Tracking targets
@@ -869,7 +880,7 @@ class Stream():
+         #
          # This can happen with `bst build --track`
+         #
 -        if not self._pipeline.targets_include(elements, track_elements):
 +        if targets and not self._pipeline.targets_include(elements, track_elements):
              raise StreamError("Specified tracking targets that are not "
                                "within the scope of primary targets")
@@ -905,6 +916,10 @@ class Stream():
          for element in track_selected:
              element._schedule_tracking()
 +        if not targets:
 +            self._pipeline.resolve_elements(track_selected)
 +            return [], track_selected
++
          # ArtifactCache.setup_remotes expects all projects to be fully loaded
          for project in self._context.get_projects():
              project.ensure_fully_loaded()

buildstream/_workspaces.py

@@ -26,14 +26,6 @@ from ._exceptions import LoadError, LoadErrorReason
  BST_WORKSPACE_FORMAT_VERSION = 3
 -# Hold on to a list of members which get serialized
 -_WORKSPACE_MEMBERS = [
 -    'prepared',
 -    'path',
 -    'last_successful',
 -    'running_files'
 -]
+-
  # Workspace()
+ #
@@ -56,7 +48,7 @@ class Workspace():
      def __init__(self, toplevel_project, *, last_successful=None, path=None, prepared=False, running_files=None):
          self.prepared = prepared
          self.last_successful = last_successful
 -        self.path = path
 +        self._path = path
          self.running_files = running_files if running_files is not None else {}
          self._toplevel_project = toplevel_project
@@ -64,14 +56,20 @@ class Workspace():
      # to_dict()
+     #
 -    # Convert this object to a dict for serialization purposes
 +    # Convert a list of members which get serialized to a dict for serialization purposes
+     #
      # Returns:
      #     (dict) A dict representation of the workspace
+     #
      def to_dict(self):
 -        return {key: val for key, val in self.__dict__.items()
 -                if key in _WORKSPACE_MEMBERS and val is not None}
 +        ret = {
 +            'prepared': self.prepared,
 +            'path': self._path,
 +            'running_files': self.running_files
 +        }
 +        if self.last_successful is not None:
 +            ret["last_successful"] = self.last_successful
 +        return ret
      # from_dict():
+     #
@@ -103,15 +101,7 @@ class Workspace():
      #    True if the workspace differs from 'other', otherwise False
+     #
      def differs(self, other):
+-
 -        for member in _WORKSPACE_MEMBERS:
 -            member_a = getattr(self, member)
 -            member_b = getattr(other, member)
+-
 -            if member_a != member_b:
 -                return True
+-
 -        return False
 +        return self.to_dict() != other.to_dict()
      # invalidate_key()
+     #
@@ -133,7 +123,7 @@ class Workspace():
          if os.path.isdir(fullpath):
              utils.copy_files(fullpath, directory)
          else:
 -            destfile = os.path.join(directory, os.path.basename(self.path))
 +            destfile = os.path.join(directory, os.path.basename(self.get_absolute_path()))
              utils.safe_copy(fullpath, destfile)
      # add_running_files()
@@ -189,7 +179,7 @@ class Workspace():
                  filelist = utils.list_relative_paths(fullpath)
                  filelist = [(relpath, os.path.join(fullpath, relpath)) for relpath in filelist]
              else:
 -                filelist = [(self.path, fullpath)]
 +                filelist = [(self.get_absolute_path(), fullpath)]
              self._key = [(relpath, unique_key(fullpath)) for relpath, fullpath in filelist]
@@ -200,7 +190,7 @@ class Workspace():
      # Returns: The absolute path of the element's workspace.
+     #
      def get_absolute_path(self):
 -        return os.path.join(self._toplevel_project.directory, self.path)
 +        return os.path.join(self._toplevel_project.directory, self._path)
  # Workspaces()
@@ -236,6 +226,9 @@ class Workspaces():
      #    path (str) - The path in which the workspace should be kept
+     #
      def create_workspace(self, element_name, path):
 +        if path.startswith(self._toplevel_project.directory):
 +            path = os.path.relpath(path, self._toplevel_project.directory)
++
          self._workspaces[element_name] = Workspace(self._toplevel_project, path=path)
          return self._workspaces[element_name]

buildstream/element.py

@@ -1403,7 +1403,8 @@ class Element(Plugin):
                      # If mount_workspaces is set and we're doing incremental builds,
                      # the workspace is already mounted into the sandbox.
                      if not (mount_workspaces and self.__can_build_incrementally()):
 -                        with self.timed_activity("Staging local files at {}".format(workspace.path)):
 +                        with self.timed_activity("Staging local files at {}"
 +                                                 .format(workspace.get_absolute_path())):
                              workspace.stage(temp_staging_directory)
                  else:
                      # No workspace, stage directly
@@ -1566,7 +1567,7 @@ class Element(Plugin):
                          path_components = self.__staged_sources_directory.lstrip(os.sep).split(os.sep)
                          sandbox_vpath = sandbox_vroot.descend(path_components)
                          try:
 -                            sandbox_vpath.import_files(workspace.path)
 +                            sandbox_vpath.import_files(workspace.get_absolute_path())
                          except UtilError as e:
                              self.warn("Failed to preserve workspace state for failed build sysroot: {}"
                                        .format(e))
@@ -1893,7 +1894,7 @@ class Element(Plugin):
                  source._init_workspace(temp)
              # Now hardlink the files into the workspace target.
 -            utils.link_files(temp, workspace.path)
 +            utils.link_files(temp, workspace.get_absolute_path())
      # _get_workspace():
+     #

buildstream/sandbox/_sandboxbwrap.py

@@ -89,6 +89,11 @@ class SandboxBwrap(Sandbox):
          # Grab the full path of the bwrap binary
          bwrap_command = [utils.get_host_tool('bwrap')]
 +        for k, v in env.items():
 +            bwrap_command += ['--setenv', k, v]
 +        for k in os.environ.keys() - env.keys():
 +            bwrap_command += ['--unsetenv', k]
++
          # Create a new pid namespace, this also ensures that any subprocesses
          # are cleaned up when the bwrap process exits.
          bwrap_command += ['--unshare-pid']
@@ -194,7 +199,7 @@ class SandboxBwrap(Sandbox):
                  stdin = stack.enter_context(open(os.devnull, "r"))
              # Run bubblewrap !
 -            exit_code = self.run_bwrap(bwrap_command, stdin, stdout, stderr, env,
 +            exit_code = self.run_bwrap(bwrap_command, stdin, stdout, stderr,
                                         (flags & SandboxFlags.INTERACTIVE))
              # Cleanup things which bwrap might have left behind, while
@@ -245,7 +250,7 @@ class SandboxBwrap(Sandbox):
          return exit_code
 -    def run_bwrap(self, argv, stdin, stdout, stderr, env, interactive):
 +    def run_bwrap(self, argv, stdin, stdout, stderr, interactive):
          # Wrapper around subprocess.Popen() with common settings.
+         #
          # This function blocks until the subprocess has terminated.
@@ -321,7 +326,6 @@ class SandboxBwrap(Sandbox):
                  # The default is to share file descriptors from the parent process
                  # to the subprocess, which is rarely good for sandboxing.
                  close_fds=True,
 -                env=env,
                  stdin=stdin,
                  stdout=stdout,
                  stderr=stderr,

setup.py

@@ -251,7 +251,7 @@ setup(name='BuildStream',
        install_requires=[
            'setuptools',
            'psutil',
 -          'ruamel.yaml',
 +          'ruamel.yaml <= 0.15',
            'pluginbase',
            'Click',
            'blessings',

tests/frontend/track.py

@@ -612,3 +612,25 @@ def test_track_include_junction(cli, tmpdir, datafiles, ref_storage, kind):
          # Assert that we are now buildable because the source is
          # now cached.
          assert cli.get_element_state(project, element_name) == 'buildable'
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("ref_storage", [('inline'), ('project.refs')])
 +@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
 +def test_track_junction_included(cli, tmpdir, datafiles, ref_storage, kind):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    element_path = os.path.join(project, 'elements')
 +    subproject_path = os.path.join(project, 'files', 'sub-project')
 +    sub_element_path = os.path.join(subproject_path, 'elements')
 +    junction_path = os.path.join(element_path, 'junction.bst')
++
 +    configure_project(project, {
 +        'ref-storage': ref_storage,
 +        '(@)': ['junction.bst:test.yml']
 +    })
++
 +    generate_junction(str(tmpdir.join('junction_repo')),
 +                      subproject_path, junction_path, store_ref=False)
++
 +    result = cli.run(project=project, args=['track', 'junction.bst'])
 +    result.assert_success()

tests/frontend/workspace.py

@@ -18,12 +18,13 @@ DATA_DIR = os.path.join(
+ )
 -def open_workspace(cli, tmpdir, datafiles, kind, track, suffix=''):
 -    project = os.path.join(datafiles.dirname, datafiles.basename)
 -    bin_files_path = os.path.join(project, 'files', 'bin-files')
 -    element_path = os.path.join(project, 'elements')
 +def open_workspace(cli, tmpdir, datafiles, kind, track, suffix='', workspace_dir=None):
 +    if not workspace_dir:
 +        workspace_dir = os.path.join(str(tmpdir), 'workspace{}'.format(suffix))
 +    project_path = os.path.join(datafiles.dirname, datafiles.basename)
 +    bin_files_path = os.path.join(project_path, 'files', 'bin-files')
 +    element_path = os.path.join(project_path, 'elements')
      element_name = 'workspace-test-{}{}.bst'.format(kind, suffix)
 -    workspace = os.path.join(str(tmpdir), 'workspace{}'.format(suffix))
      # Create our repo object of the given source type with
      # the bin files, and then collect the initial ref.
@@ -45,7 +46,7 @@ def open_workspace(cli, tmpdir, datafiles, kind, track, suffix=''):
                              element_name))
      # Assert that there is no reference, a track & fetch is needed
 -    state = cli.get_element_state(project, element_name)
 +    state = cli.get_element_state(project_path, element_name)
      if track:
          assert state == 'no reference'
      else:
@@ -56,20 +57,20 @@ def open_workspace(cli, tmpdir, datafiles, kind, track, suffix=''):
      args = ['workspace', 'open']
      if track:
          args.append('--track')
 -    args.extend([element_name, workspace])
 +    args.extend([element_name, workspace_dir])
 +    result = cli.run(project=project_path, args=args)
 -    result = cli.run(project=project, args=args)
      result.assert_success()
      # Assert that we are now buildable because the source is
      # now cached.
 -    assert cli.get_element_state(project, element_name) == 'buildable'
 +    assert cli.get_element_state(project_path, element_name) == 'buildable'
      # Check that the executable hello file is found in the workspace
 -    filename = os.path.join(workspace, 'usr', 'bin', 'hello')
 +    filename = os.path.join(workspace_dir, 'usr', 'bin', 'hello')
      assert os.path.exists(filename)
 -    return (element_name, project, workspace)
 +    return (element_name, project_path, workspace_dir)
  @pytest.mark.datafiles(DATA_DIR)
@@ -190,6 +191,46 @@ def test_close(cli, tmpdir, datafiles, kind):
      assert not os.path.exists(workspace)
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_close_external_after_move_project(cli, tmpdir, datafiles):
 +    tmp_parent = os.path.dirname(str(tmpdir))
 +    workspace_dir = os.path.join(tmp_parent, "workspace")
 +    element_name, project_path, _ = open_workspace(cli, tmpdir, datafiles, 'git', False, "", workspace_dir)
 +    assert os.path.exists(workspace_dir)
 +    tmp_dir = os.path.join(tmp_parent, 'external_project')
 +    shutil.move(project_path, tmp_dir)
 +    assert os.path.exists(tmp_dir)
++
 +    # Close the workspace
 +    result = cli.run(configure=False, project=tmp_dir, args=[
 +        'workspace', 'close', '--remove-dir', element_name
 +    ])
 +    result.assert_success()
++
 +    # Assert the workspace dir has been deleted
 +    assert not os.path.exists(workspace_dir)
 +    # Move directory back inside tmp directory so it can be recognised
 +    shutil.move(tmp_dir, project_path)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_close_internal_after_move_project(cli, tmpdir, datafiles):
 +    element_name, project, _ = open_workspace(cli, tmpdir, datafiles, 'git', False)
 +    tmp_dir = os.path.join(os.path.dirname(str(tmpdir)), 'external_project')
 +    shutil.move(str(tmpdir), tmp_dir)
 +    assert os.path.exists(tmp_dir)
++
 +    # Close the workspace
 +    result = cli.run(configure=False, project=tmp_dir, args=[
 +        'workspace', 'close', '--remove-dir', element_name
 +    ])
 +    result.assert_success()
++
 +    # Assert the workspace dir has been deleted
 +    workspace = os.path.join(tmp_dir, 'workspace')
 +    assert not os.path.exists(workspace)
++
++
  @pytest.mark.datafiles(DATA_DIR)
  def test_close_removed(cli, tmpdir, datafiles):
      element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, 'git', False)

[Notes] [Git][BuildStream/buildstream][valentindavid/fix_included_junction_track] 16 commits: .gitlab-ci.yml: Remove unused script "install.sh"

Valentin David pushed to branch valentindavid/fix_included_junction_track at BuildStream / buildstream

Commits:

16 changed files:

Changes: