[Notes] [Git][BuildStream/buildstream][willsalmon/defaultWorkspaces] 9 c

Will Salmon pushed to branch willsalmon/defaultWorkspaces at BuildStream / buildstream

Commits:

de59ebdb

by ctolentino8 at 2018-11-02T16:41:54Z

plugins/sources/pip.py: Accomodate characters '-','.','_' for packages

8d7cf806

by ctolentino8 at 2018-11-02T16:41:54Z

tests/integration/project/files: Remove pypi-repo

9c2f9bf7

by Chandan Singh at 2018-11-02T17:09:46Z

Merge branch 'chiaratolentino/fix-pip-source-regex' into 'master'

plugins/sources/pip.py: Accomodate characters '-','.','_' for packages

See merge request BuildStream/buildstream!914

3788e701

by Jürg Billeter at 2018-11-03T11:52:00Z

element.py: Pass bare_directory also to SandboxRemote()

Fixes: 825963b5 ("element: Make "--sysroot" take a bare directory")

82e971ef

by Jürg Billeter at 2018-11-05T11:33:20Z

Merge branch 'juerg/remote-execution' into 'master'

element.py: Pass bare_directory also to SandboxRemote()

See merge request BuildStream/buildstream!928

62942bfd

by Valentin David at 2018-11-05T12:14:20Z

Use less fetchers in overnight build to work-around limitations from servers.

Issue happens with sourceware.org server.

Fixes #742

442da2f9

by Javier Jardón at 2018-11-05T12:41:54Z

Merge branch 'valentindavid/limit_fetchers_in_overnight_build' into 'master'

Use less fetchers in overnight build to work-around limitations from servers.

Closes #742

See merge request BuildStream/buildstream!929

6b6d735c

by William Salmon at 2018-11-05T15:49:23Z

tests/frontend/workspace.py: Refactor support function

The Refactor now makes it possible to test for the forth coming multi
element workspace open commands and to test for failure.

a48f0bad

by William Salmon at 2018-11-05T17:12:55Z

Workspace CLI update

This is to update the workspace CLI to as agreed on the mailing list
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00046.html

This patch also introduces the default workspace directory.

24 changed files:

.gitlab-ci.yml
buildstream/_context.py
buildstream/_frontend/cli.py
buildstream/_stream.py
buildstream/data/userconfig.yaml
buildstream/element.py
buildstream/plugins/sources/pip.py
buildstream/utils.py
tests/examples/developing.py
tests/examples/junctions.py
tests/frontend/buildcheckout.py
tests/frontend/cross_junction_workspace.py
tests/frontend/workspace.py
tests/integration/pip_source.py
− tests/integration/project/files/pypi-repo/app2/App2-0.1.tar.gz
− tests/integration/project/files/pypi-repo/app2/index.html
− tests/integration/project/files/pypi-repo/hellolib/HelloLib-0.1.tar.gz
− tests/integration/project/files/pypi-repo/hellolib/index.html
tests/integration/shell.py
tests/integration/workspace.py
tests/plugins/filter.py
tests/sources/pip.py
tests/testutils/__init__.py
+ tests/testutils/python_repo.py

Changes:

.gitlab-ci.yml

@@ -166,6 +166,12 @@ docs:
      BST_EXT_REF: 1d6ab71151b93c8cbc0a91a36ffe9270f3b835f1 # 0.5.1
      FD_SDK_REF: 88d7c22c2281b987faa02edd57df80d430eecf1f # 18.08.11-35-g88d7c22c
    before_script:
 +  - |
 +    mkdir -p "${HOME}/.config"
 +    cat <<EOF >"${HOME}/.config/buildstream.conf"
 +    scheduler:
 +      fetchers: 2
 +    EOF
    - (cd dist && ./unpack.sh && cd buildstream && pip3 install .)
    - pip3 install --user -e ${BST_EXT_URL}@${BST_EXT_REF}#egg=bst_ext
    - git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk.git

buildstream/_context.py

@@ -60,6 +60,9 @@ class Context():
          # The directory where build sandboxes will be created
          self.builddir = None
 +        # Default root location for workspaces
 +        self.workspacedir = None
++
          # The local binary artifact cache directory
          self.artifactdir = None
@@ -161,10 +164,10 @@ class Context():
          _yaml.node_validate(defaults, [
              'sourcedir', 'builddir', 'artifactdir', 'logdir',
              'scheduler', 'artifacts', 'logging', 'projects',
 -            'cache'
 +            'cache', 'workspacedir',
          ])
 -        for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir']:
 +        for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir', 'workspacedir']:
              # Allow the ~ tilde expansion and any environment variables in
              # path specification in the config files.
+             #

buildstream/_frontend/cli.py

@@ -6,6 +6,7 @@ from .. import _yaml
  from .._exceptions import BstError, LoadError, AppError
  from .._versions import BST_FORMAT_VERSION
  from .complete import main_bashcomplete, complete_path, CompleteUnhandled
 +from ..utils import DirectoryDescription
  ##################################################################
@@ -678,28 +679,36 @@ def workspace():
  @click.option('--no-checkout', default=False, is_flag=True,
                help="Do not checkout the source, only link to the given directory")
  @click.option('--force', '-f', default=False, is_flag=True,
 -              help="Overwrite files existing in checkout directory")
 +              help="The workspace will be created even if the directory in which it will be created is not empty " +
 +              "or if a workspace for that element already exists")
  @click.option('--track', 'track_', default=False, is_flag=True,
                help="Track and fetch new source references before checking out the workspace")
 -@click.argument('element',
 -                type=click.Path(readable=False))
 -@click.argument('directory', type=click.Path(file_okay=False))
 +@click.option('--directory', type=click.Path(file_okay=False), default=None,
 +              help="Only for use when a single Element is given: Set the directory to use to create the workspace")
 +@click.argument('elements', nargs=-1, type=click.Path(readable=False))
  @click.pass_obj
 -def workspace_open(app, no_checkout, force, track_, element, directory):
 +def workspace_open(app, no_checkout, force, track_, directory, elements):
      """Open a workspace for manual source modification"""
+-
 -    if os.path.exists(directory):
+-
 -        if not os.path.isdir(directory):
 -            click.echo("Checkout directory is not a directory: {}".format(directory), err=True)
 +    directories = []
 +    if directory is not None:
 +        if len(elements) > 1:
 +            click.echo("Directory option can only be used if a single element is given", err=True)
              sys.exit(-1)
 +        if os.path.exists(directory):
 +            if not os.path.isdir(directory):
 +                click.echo("Directory path is not a directory: {}".format(directory), err=True)
 +                sys.exit(-1)
 -        if not (no_checkout or force) and os.listdir(directory):
 -            click.echo("Checkout directory is not empty: {}".format(directory), err=True)
 -            sys.exit(-1)
 +            if not (no_checkout or force) and os.listdir(directory):
 +                click.echo("Directory path is not empty: {}".format(directory), err=True)
 +                sys.exit(-1)
 +        directories.append(DirectoryDescription(directory, use_default=False))
 +    else:
 +        for element in elements:
 +            directories.append(DirectoryDescription(element.rstrip('.bst')))
      with app.initialized():
 -        app.stream.workspace_open(element, directory,
 +        app.stream.workspace_open(elements, directories,
                                    no_checkout=no_checkout,
                                    track_first=track_,
                                    force=force)

buildstream/_stream.py

@@ -36,6 +36,7 @@ from . import utils, _yaml, _site
  from . import Scope, Consistency
++
  # Stream()
+ #
  # This is the main, toplevel calling interface in BuildStream core.
@@ -448,44 +449,29 @@ class Stream():
      # Open a project workspace
+     #
      # Args:
 -    #    target (str): The target element to open the workspace for
 -    #    directory (str): The directory to stage the source in
 +    #    target (list): List of target elements to open workspaces for
 +    #    directory (list): List of DirectoryDescription objects to stage the source in
      #    no_checkout (bool): Whether to skip checking out the source
      #    track_first (bool): Whether to track and fetch first
      #    force (bool): Whether to ignore contents in an existing directory
+     #
 -    def workspace_open(self, target, directory, *,
 +    def workspace_open(self, targets, directories, *,
                         no_checkout,
                         track_first,
                         force):
 +        # This function is a little funny but it is trying to be as atomic as possible.
          if track_first:
 -            track_targets = (target,)
 +            track_targets = targets
          else:
              track_targets = ()
 -        elements, track_elements = self._load((target,), track_targets,
 +        elements, track_elements = self._load(targets, track_targets,
                                                selection=PipelineSelection.REDIRECT,
                                                track_selection=PipelineSelection.REDIRECT)
 -        target = elements[0]
 -        directory = os.path.abspath(directory)
+-
 -        if not list(target.sources()):
 -            build_depends = [x.name for x in target.dependencies(Scope.BUILD, recurse=False)]
 -            if not build_depends:
 -                raise StreamError("The given element has no sources")
 -            detail = "Try opening a workspace on one of its dependencies instead:\n"
 -            detail += "  \n".join(build_depends)
 -            raise StreamError("The given element has no sources", detail=detail)
          workspaces = self._context.get_workspaces()
 -        # Check for workspace config
 -        workspace = workspaces.get_workspace(target._get_full_name())
 -        if workspace and not force:
 -            raise StreamError("Workspace '{}' is already defined at: {}"
 -                              .format(target.name, workspace.get_absolute_path()))
+-
          # If we're going to checkout, we need at least a fetch,
          # if we were asked to track first, we're going to fetch anyway.
+         #
@@ -495,29 +481,69 @@ class Stream():
                  track_elements = elements
              self._fetch(elements, track_elements=track_elements)
 -        if not no_checkout and target._get_consistency() != Consistency.CACHED:
 -            raise StreamError("Could not stage uncached source. " +
 -                              "Use `--track` to track and " +
 -                              "fetch the latest version of the " +
 -                              "source.")
 +        expanded_directories = []
 +        #  To try to be more atomic, loop through the elements and raise any errors we can early
 +        for target, directory_obj in zip(elements, directories):
++
 +            if not list(target.sources()):
 +                build_depends = [x.name for x in target.dependencies(Scope.BUILD, recurse=False)]
 +                if not build_depends:
 +                    raise StreamError("The element {}  has no sources".format(target.name))
 +                detail = "Try opening a workspace on one of its dependencies instead:\n"
 +                detail += "  \n".join(build_depends)
 +                raise StreamError("The element {} has no sources".format(target.name), detail=detail)
++
 +            # Check for workspace config
 +            workspace = workspaces.get_workspace(target._get_full_name())
 +            if workspace and not force:
 +                raise StreamError("Workspace '{}' is already defined at: {}"
 +                                  .format(target.name, workspace.get_absolute_path()))
++
 +            if not no_checkout and target._get_consistency() != Consistency.CACHED:
 +                raise StreamError("Could not stage uncached source. For {} ".format(target.name) +
 +                                  "Use `--track` to track and " +
 +                                  "fetch the latest version of the " +
 +                                  "source.")
++
 +            if directory_obj.use_default:
 +                directory = os.path.abspath(os.path.join(self._context.workspacedir, directory_obj.directory))
 +            else:
 +                directory = directory_obj.directory
 -        if workspace:
 -            workspaces.delete_workspace(target._get_full_name())
 -            workspaces.save_config()
 -            shutil.rmtree(directory)
 -        try:
 -            os.makedirs(directory, exist_ok=True)
 -        except OSError as e:
 -            raise StreamError("Failed to create workspace directory: {}".format(e)) from e
 +            expanded_directories.append(directory)
 -        workspaces.create_workspace(target._get_full_name(), directory)
 +        # So far this function has tried to catch as many issues as possible with out making any changes
 +        # Now it dose the bits that can not be made atomic.
 +        targetGenerator = zip(elements, expanded_directories)
 +        for target, directory in targetGenerator:
 +            self._message(MessageType.INFO, "Creating workspace for element {}"
 +                          .format(target.name))
 -        if not no_checkout:
 -            with target.timed_activity("Staging sources to {}".format(directory)):
 -                target._open_workspace()
 +            workspace = workspaces.get_workspace(target._get_full_name())
 +            if workspace:
 +                workspaces.delete_workspace(target._get_full_name())
 +                workspaces.save_config()
 +                shutil.rmtree(directory)
 +            try:
 +                os.makedirs(directory, exist_ok=True)
 +            except OSError as e:
 +                todo_elements = " ".join([str(target.name) for target, directory_dict in targetGenerator])
 +                if todo_elements:
 +                    # This output should make creating the remaining workspaces as easy as possible.
 +                    todo_elements = "\nDid not try to create workspaces for " + todo_elements
 +                raise StreamError("Failed to create workspace directory: {}".format(e) + todo_elements) from e
 -        workspaces.save_config()
 -        self._message(MessageType.INFO, "Saved workspace configuration")
 +            workspaces.create_workspace(target._get_full_name(), directory)
++
 +            if not no_checkout:
 +                with target.timed_activity("Staging sources to {}".format(directory)):
 +                    target._open_workspace()
++
 +            # Saving the workspace once it is set up means that if the next workspace fails to be created before
 +            # the configuration gets saved. The successfully created workspace still gets saved.
 +            workspaces.save_config()
 +            self._message(MessageType.INFO, "Added element {} to the workspace configuration"
 +                          .format(target._get_full_name()))
      # workspace_close
+     #

buildstream/data/userconfig.yaml


 # Location to store build logs
 logdir: ${XDG_CACHE_HOME}/buildstream/logs
 
+# Default root location for workspaces, blank for no default set.
+workspacedir: .
+
 #
 #    Cache
 #

buildstream/element.py

@@ -2180,6 +2180,7 @@ class Element(Plugin):
                                      stderr=stderr,
                                      config=config,
                                      server_url=self.__remote_execution_url,
 +                                    bare_directory=bare_directory,
                                      allow_real_directory=False)
              yield sandbox

buildstream/plugins/sources/pip.py

@@ -96,7 +96,7 @@ _PYTHON_VERSIONS = [
  # Names of source distribution archives must be of the form
  # '%{package-name}-%{version}.%{extension}'.
  _SDIST_RE = re.compile(
 -    r'^([a-zA-Z0-9]+?)-(.+).(?:tar|tar.bz2|tar.gz|tar.xz|tar.Z|zip)$',
 +    r'^([\w.-]+?)-((?:[\d.]+){2,})\.(?:tar|tar.bz2|tar.gz|tar.xz|tar.Z|zip)$',
      re.IGNORECASE)
@@ -225,12 +225,27 @@ class PipSource(Source):
      def _parse_sdist_names(self, basedir):
          reqs = []
          for f in os.listdir(basedir):
 -            pkg_match = _SDIST_RE.match(f)
 -            if pkg_match:
 -                reqs.append(pkg_match.groups())
 +            pkg = _match_package_name(f)
 +            if pkg is not None:
 +                reqs.append(pkg)
          return sorted(reqs)
 +# Extract the package name and version of a source distribution
 +#
 +# Args:
 +#    filename (str): Filename of the source distribution
 +#
 +# Returns:
 +#    (tuple): A tuple of (package_name, version)
 +#
 +def _match_package_name(filename):
 +    pkg_match = _SDIST_RE.match(filename)
 +    if pkg_match is None:
 +        return None
 +    return pkg_match.groups()
++
++
  def setup():
      return PipSource

buildstream/utils.py

@@ -105,6 +105,20 @@ class FileListResult():
          return ret
 +class DirectoryDescription():
 +    """
 +    This object is used to keep information about directories in a nice tidy object.
 +    """
 +    def __init__(self, directory, *, use_default=True):
 +        """
 +        Args:
 +            directory (str): The path to the directory this object describes
 +            use_default (bool): Weather to process the directory so it is in the default folder.
 +        """
 +        self.directory = directory
 +        self.use_default = use_default
++
++
  def list_relative_paths(directory, *, list_dirs=True):
      """A generator for walking directory relative paths
@@ -559,6 +573,19 @@ def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
          raise
 +# DirectoryDescription()
 +#
 +# This object is used to keep information about directories in a nice tidy object.
 +#
 +# Args:
 +#    directory (str): The path to the directory this object describes
 +#    use_default (bool): Weather to process the directory so it is in the default folder.
 +#
 +class DirectoryDescription():
 +    def __init__(self, directory, *, use_default=True):
 +        self.directory = directory
 +        self.use_default = use_default
++
  # _get_dir_size():
+ #
  # Get the disk usage of a given directory in bytes.

tests/examples/developing.py

@@ -55,7 +55,7 @@ def test_open_workspace(cli, tmpdir, datafiles):
      project = os.path.join(datafiles.dirname, datafiles.basename)
      workspace_dir = os.path.join(str(tmpdir), "workspace_hello")
 -    result = cli.run(project=project, args=['workspace', 'open', '-f', 'hello.bst', workspace_dir])
 +    result = cli.run(project=project, args=['workspace', 'open', '-f', '--directory', workspace_dir, 'hello.bst', ])
      result.assert_success()
      result = cli.run(project=project, args=['workspace', 'list'])
@@ -72,7 +72,7 @@ def test_make_change_in_workspace(cli, tmpdir, datafiles):
      project = os.path.join(datafiles.dirname, datafiles.basename)
      workspace_dir = os.path.join(str(tmpdir), "workspace_hello")
 -    result = cli.run(project=project, args=['workspace', 'open', '-f', 'hello.bst', workspace_dir])
 +    result = cli.run(project=project, args=['workspace', 'open', '-f', '--directory', workspace_dir, 'hello.bst'])
      result.assert_success()
      result = cli.run(project=project, args=['workspace', 'list'])

tests/examples/junctions.py

@@ -44,7 +44,7 @@ def test_open_cross_junction_workspace(cli, tmpdir, datafiles):
      workspace_dir = os.path.join(str(tmpdir), "workspace_hello_junction")
      result = cli.run(project=project,
 -                     args=['workspace', 'open', 'hello-junction.bst:hello.bst', workspace_dir])
 +                     args=['workspace', 'open', '--directory', workspace_dir, 'hello-junction.bst:hello.bst'])
      result.assert_success()
      result = cli.run(project=project,

tests/frontend/buildcheckout.py

@@ -509,7 +509,7 @@ def test_build_checkout_workspaced_junction(cli, tmpdir, datafiles):
      # Now open a workspace on the junction
+     #
 -    result = cli.run(project=project, args=['workspace', 'open', 'junction.bst', workspace])
 +    result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, 'junction.bst'])
      result.assert_success()
      filename = os.path.join(workspace, 'files', 'etc-files', 'etc', 'animal.conf')

tests/frontend/cross_junction_workspace.py

@@ -47,7 +47,7 @@ def open_cross_junction(cli, tmpdir):
      workspace = tmpdir.join("workspace")
      element = 'sub.bst:data.bst'
 -    args = ['workspace', 'open', element, str(workspace)]
 +    args = ['workspace', 'open', '--directory', str(workspace), element]
      result = cli.run(project=project, args=args)
      result.assert_success()

tests/frontend/workspace.py

@@ -21,9 +21,11 @@
  #           Phillip Smyth <phillip smyth codethink co uk>
  #           Jonathan Maw <jonathan maw codethink co uk>
  #           Richard Maw <richard maw codethink co uk>
 +#           William Salmon <will salmon codethink co uk>
+ #
  import os
 +import stat
  import pytest
  import shutil
  import subprocess
@@ -43,65 +45,118 @@ DATA_DIR = os.path.join(
+ )
 -def open_workspace(cli, tmpdir, datafiles, kind, track, suffix='', workspace_dir=None,
 -                   project_path=None, element_attrs=None):
 -    if not workspace_dir:
 -        workspace_dir = os.path.join(str(tmpdir), 'workspace{}'.format(suffix))
 -    if not project_path:
 -        project_path = os.path.join(datafiles.dirname, datafiles.basename)
 -    else:
 -        shutil.copytree(os.path.join(datafiles.dirname, datafiles.basename), project_path)
 -    bin_files_path = os.path.join(project_path, 'files', 'bin-files')
 -    element_path = os.path.join(project_path, 'elements')
 -    element_name = 'workspace-test-{}{}.bst'.format(kind, suffix)
 +class WorkspaceCreater():
 +    def __init__(self, cli, tmpdir, datafiles, project_path=None):
 +        self.cli = cli
 +        self.tmpdir = tmpdir
 +        self.datafiles = datafiles
++
 +        if not project_path:
 +            project_path = os.path.join(datafiles.dirname, datafiles.basename)
 +        else:
 +            shutil.copytree(os.path.join(datafiles.dirname, datafiles.basename), project_path)
++
 +        self.project_path = project_path
 +        self.bin_files_path = os.path.join(project_path, 'files', 'bin-files')
++
 +        self.workspace_cmd = os.path.join(self.project_path, 'workspace_cmd')
++
 +    def create_workspace_element(self, kind, track, suffix='', workspace_dir=None,
 +                                 element_attrs=None):
 +        element_name = 'workspace-test-{}{}.bst'.format(kind, suffix)
 +        element_path = os.path.join(self.project_path, 'elements')
 +        if not workspace_dir:
 +            workspace_dir = os.path.join(self.workspace_cmd, element_name.rstrip('.bst'))
++
 +        # Create our repo object of the given source type with
 +        # the bin files, and then collect the initial ref.
 +        repo = create_repo(kind, str(self.tmpdir))
 +        ref = repo.create(self.bin_files_path)
 +        if track:
 +            ref = None
++
 +        # Write out our test target
 +        element = {
 +            'kind': 'import',
 +            'sources': [
 +                repo.source_config(ref=ref)
 +            ]
 +        }
 +        if element_attrs:
 +            element = {**element, **element_attrs}
 +        _yaml.dump(element,
 +                   os.path.join(element_path,
 +                                element_name))
 +        return element_name, element_path, workspace_dir
 -    # Create our repo object of the given source type with
 -    # the bin files, and then collect the initial ref.
 -    #
 -    repo = create_repo(kind, str(tmpdir))
 -    ref = repo.create(bin_files_path)
 -    if track:
 -        ref = None
 +    def create_workspace_elements(self, kinds, track, suffixs=None, workspace_dir_usr=None,
 +                                  element_attrs=None):
 -    # Write out our test target
 -    element = {
 -        'kind': 'import',
 -        'sources': [
 -            repo.source_config(ref=ref)
 -        ]
 -    }
 -    if element_attrs:
 -        element = {**element, **element_attrs}
 -    _yaml.dump(element,
 -               os.path.join(element_path,
 -                            element_name))
 +        element_tuples = []
 -    # Assert that there is no reference, a track & fetch is needed
 -    state = cli.get_element_state(project_path, element_name)
 -    if track:
 -        assert state == 'no reference'
 -    else:
 -        assert state == 'fetch needed'
 +        if suffixs is None:
 +            suffixs = ['', ] * len(kinds)
 +        else:
 +            if len(suffixs) != len(kinds):
 +                raise "terable error"
 -    # Now open the workspace, this should have the effect of automatically
 -    # tracking & fetching the source from the repo.
 -    args = ['workspace', 'open']
 -    if track:
 -        args.append('--track')
 -    args.extend([element_name, workspace_dir])
 -    result = cli.run(project=project_path, args=args)
 +        for suffix, kind in zip(suffixs, kinds):
 +            element_name, element_path, workspace_dir = \
 +                self.create_workspace_element(kind, track, suffix, workspace_dir_usr,
 +                                              element_attrs)
 -    result.assert_success()
 +            # Assert that there is no reference, a track & fetch is needed
 +            state = self.cli.get_element_state(self.project_path, element_name)
 +            if track:
 +                assert state == 'no reference'
 +            else:
 +                assert state == 'fetch needed'
 +            element_tuples.append((element_name, workspace_dir))
 -    # Assert that we are now buildable because the source is
 -    # now cached.
 -    assert cli.get_element_state(project_path, element_name) == 'buildable'
 +        return element_tuples
 -    # Check that the executable hello file is found in the workspace
 -    filename = os.path.join(workspace_dir, 'usr', 'bin', 'hello')
 -    assert os.path.exists(filename)
 +    def open_workspaces(self, kinds, track, suffixs=None, workspace_dir=None,
 +                        element_attrs=None):
++
 +        element_tuples = self.create_workspace_elements(kinds, track, suffixs, workspace_dir,
 +                                                        element_attrs)
 +        os.makedirs(self.workspace_cmd, exist_ok=True)
++
 +        # Now open the workspace, this should have the effect of automatically
 +        # tracking & fetching the source from the repo.
 +        args = ['workspace', 'open']
 +        if track:
 +            args.append('--track')
 +        if workspace_dir is not None:
 +            assert len(element_tuples) == 1, "test logic error"
 +            _, workspace_dir = element_tuples[0]
 +            args.extend(['--directory', workspace_dir])
++
 +        args.extend([element_name for element_name, workspace_dir_suffix in element_tuples])
 +        result = self.cli.run(cwd=self.workspace_cmd, project=self.project_path, args=args)
 -    return (element_name, project_path, workspace_dir)
 +        result.assert_success()
++
 +        for element_name, workspace_dir in element_tuples:
 +            # Assert that we are now buildable because the source is
 +            # now cached.
 +            assert self.cli.get_element_state(self.project_path, element_name) == 'buildable'
++
 +            # Check that the executable hello file is found in the workspace
 +            filename = os.path.join(workspace_dir, 'usr', 'bin', 'hello')
 +            assert os.path.exists(filename)
++
 +        return element_tuples
++
++
 +def open_workspace(cli, tmpdir, datafiles, kind, track, suffix='', workspace_dir=None,
 +                   project_path=None, element_attrs=None):
 +    workspace_object = WorkspaceCreater(cli, tmpdir, datafiles, project_path)
 +    workspaces = workspace_object.open_workspaces((kind, ), track, (suffix, ), workspace_dir,
 +                                                  element_attrs)
 +    assert len(workspaces) == 1
 +    element_name, workspace = workspaces[0]
 +    return element_name, workspace_object.project_path, workspace
  @pytest.mark.datafiles(DATA_DIR)
@@ -128,6 +183,103 @@ def test_open_bzr_customize(cli, tmpdir, datafiles):
      assert(expected_output_str in str(output))
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_open_multi(cli, tmpdir, datafiles):
++
 +    workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
 +    workspaces = workspace_object.open_workspaces(repo_kinds, False)
++
 +    for (elname, workspace), kind in zip(workspaces, repo_kinds):
 +        assert kind in elname
 +        workspace_lsdir = os.listdir(workspace)
 +        if kind == 'git':
 +            assert('.git' in workspace_lsdir)
 +        elif kind == 'bzr':
 +            assert('.bzr' in workspace_lsdir)
 +        else:
 +            assert not ('.git' in workspace_lsdir)
 +            assert not ('.bzr' in workspace_lsdir)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_open_multi_unwritable(cli, tmpdir, datafiles):
 +    workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
++
 +    element_tuples = workspace_object.create_workspace_elements(repo_kinds, False, repo_kinds)
 +    os.makedirs(workspace_object.workspace_cmd, exist_ok=True)
++
 +    # Now open the workspace, this should have the effect of automatically
 +    # tracking & fetching the source from the repo.
 +    args = ['workspace', 'open']
 +    args.extend([element_name for element_name, workspace_dir_suffix in element_tuples])
 +    cli.configure({'workspacedir': workspace_object.workspace_cmd})
++
 +    cwdstat = os.stat(workspace_object.workspace_cmd)
 +    try:
 +        os.chmod(workspace_object.workspace_cmd, cwdstat.st_mode - stat.S_IWRITE)
 +        result = workspace_object.cli.run(project=workspace_object.project_path, args=args)
 +    finally:
 +        # Using this finally to make sure we always put thing back how they should be.
 +        os.chmod(workspace_object.workspace_cmd, cwdstat.st_mode)
++
 +    result.assert_main_error(ErrorDomain.STREAM, None)
 +    # Normally we avoid checking stderr in favour of using the mechine readable result.assert_main_error
 +    # But Tristan was very keen that the names of the elements left needing workspaces were present in the out put
 +    assert (" ".join([element_name for element_name, workspace_dir_suffix in element_tuples[1:]]) in result.stderr)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_open_multi_with_directory(cli, tmpdir, datafiles):
 +    workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
++
 +    element_tuples = workspace_object.create_workspace_elements(repo_kinds, False, repo_kinds)
 +    os.makedirs(workspace_object.workspace_cmd, exist_ok=True)
++
 +    # Now open the workspace, this should have the effect of automatically
 +    # tracking & fetching the source from the repo.
 +    args = ['workspace', 'open']
 +    args.extend(['--directory', 'any/dir/should/fail'])
++
 +    args.extend([element_name for element_name, workspace_dir_suffix in element_tuples])
 +    result = workspace_object.cli.run(cwd=workspace_object.workspace_cmd, project=workspace_object.project_path,
 +                                      args=args)
++
 +    result.assert_main_error(ErrorDomain.ARTIFACT, None)
 +    assert ("Directory option can only be used if a single element is given" in result.stderr)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_open_defaultlocation(cli, tmpdir, datafiles):
 +    workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
++
 +    ((element_name, workspace_dir), ) = workspace_object.create_workspace_elements(['git'], False, ['git'])
 +    os.makedirs(workspace_object.workspace_cmd, exist_ok=True)
++
 +    # Now open the workspace, this should have the effect of automatically
 +    # tracking & fetching the source from the repo.
 +    args = ['workspace', 'open']
 +    args.append(element_name)
++
 +    # In the other tests we set the cmd to workspace_object.workspace_cmd with the optional
 +    # argument, cwd for the workspace_object.cli.run function. But hear we set the default
 +    # workspace location to workspace_object.workspace_cmd and run the cli.run function with
 +    # no cwd option so that it runs in the project directory.
 +    cli.configure({'workspacedir': workspace_object.workspace_cmd})
 +    result = workspace_object.cli.run(project=workspace_object.project_path,
 +                                      args=args)
++
 +    result.assert_success()
++
 +    assert cli.get_element_state(workspace_object.project_path, element_name) == 'buildable'
++
 +    # Check that the executable hello file is found in the workspace
 +    # even though the cli.run function was not run with cwd = workspace_object.workspace_cmd
 +    # the workspace should be created in there as we used the 'workspacedir' configuration
 +    # option.
 +    filename = os.path.join(workspace_dir, 'usr', 'bin', 'hello')
 +    assert os.path.exists(filename)
++
++
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.parametrize("kind", repo_kinds)
  def test_open_track(cli, tmpdir, datafiles, kind):
@@ -150,7 +302,7 @@ def test_open_force(cli, tmpdir, datafiles, kind):
      # Now open the workspace again with --force, this should happily succeed
      result = cli.run(project=project, args=[
 -        'workspace', 'open', '--force', element_name, workspace
 +        'workspace', 'open', '--force', '--directory', workspace, element_name
      ])
      result.assert_success()
@@ -165,7 +317,7 @@ def test_open_force_open(cli, tmpdir, datafiles, kind):
      # Now open the workspace again with --force, this should happily succeed
      result = cli.run(project=project, args=[
 -        'workspace', 'open', '--force', element_name, workspace
 +        'workspace', 'open', '--force', '--directory', workspace, element_name
      ])
      result.assert_success()
@@ -196,7 +348,7 @@ def test_open_force_different_workspace(cli, tmpdir, datafiles, kind):
      # Now open the workspace again with --force, this should happily succeed
      result = cli.run(project=project, args=[
 -        'workspace', 'open', '--force', element_name2, workspace
 +        'workspace', 'open', '--force', '--directory', workspace, element_name2
      ])
      # Assert that the file in workspace 1 has been replaced
@@ -504,7 +656,7 @@ def test_buildable_no_ref(cli, tmpdir, datafiles):
      # Now open the workspace. We don't need to checkout the source though.
      workspace = os.path.join(str(tmpdir), 'workspace-no-ref')
      os.makedirs(workspace)
 -    args = ['workspace', 'open', '--no-checkout', element_name, workspace]
 +    args = ['workspace', 'open', '--no-checkout', '--directory', workspace, element_name]
      result = cli.run(project=project, args=args)
      result.assert_success()
@@ -766,7 +918,7 @@ def test_list_supported_workspace(cli, tmpdir, datafiles, workspace_cfg, expecte
                              element_name))
      # Make a change to the workspaces file
 -    result = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
 +    result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
      result.assert_success()
      result = cli.run(project=project, args=['workspace', 'close', '--remove-dir', element_name])
      result.assert_success()

tests/integration/pip_source.py

@@ -4,6 +4,7 @@ import pytest
  from buildstream import _yaml
  from tests.testutils import cli_integration as cli
 +from tests.testutils.python_repo import setup_pypi_repo
  from tests.testutils.integration import assert_contains
@@ -17,12 +18,21 @@ DATA_DIR = os.path.join(
  @pytest.mark.datafiles(DATA_DIR)
 -def test_pip_source_import(cli, tmpdir, datafiles):
 +def test_pip_source_import(cli, tmpdir, datafiles, setup_pypi_repo):
      project = os.path.join(datafiles.dirname, datafiles.basename)
      checkout = os.path.join(cli.directory, 'checkout')
      element_path = os.path.join(project, 'elements')
      element_name = 'pip/hello.bst'
 +    # check that exotically named packages are imported correctly
 +    myreqs_packages = ['hellolib']
 +    packages = ['app2', 'app.3', 'app-4', 'app_5', 'app.no.6', 'app-no-7', 'app_no_8']
++
 +    # create mock pypi repository
 +    pypi_repo = os.path.join(project, 'files', 'pypi-repo')
 +    os.makedirs(pypi_repo, exist_ok=True)
 +    setup_pypi_repo(myreqs_packages + packages, pypi_repo)
++
      element = {
          'kind': 'import',
          'sources': [
@@ -32,9 +42,9 @@ def test_pip_source_import(cli, tmpdir, datafiles):
              },
+             {
                  'kind': 'pip',
 -                'url': 'file://{}'.format(os.path.realpath(os.path.join(project, 'files', 'pypi-repo'))),
 +                'url': 'file://{}'.format(os.path.realpath(pypi_repo)),
                  'requirements-files': ['myreqs.txt'],
 -                'packages': ['app2']
 +                'packages': packages
+             }
+         ]
+     }
@@ -51,16 +61,31 @@ def test_pip_source_import(cli, tmpdir, datafiles):
      assert result.exit_code == 0
      assert_contains(checkout, ['/.bst_pip_downloads',
 -                               '/.bst_pip_downloads/HelloLib-0.1.tar.gz',
 -                               '/.bst_pip_downloads/App2-0.1.tar.gz'])
 +                               '/.bst_pip_downloads/hellolib-0.1.tar.gz',
 +                               '/.bst_pip_downloads/app2-0.1.tar.gz',
 +                               '/.bst_pip_downloads/app.3-0.1.tar.gz',
 +                               '/.bst_pip_downloads/app-4-0.1.tar.gz',
 +                               '/.bst_pip_downloads/app_5-0.1.tar.gz',
 +                               '/.bst_pip_downloads/app.no.6-0.1.tar.gz',
 +                               '/.bst_pip_downloads/app-no-7-0.1.tar.gz',
 +                               '/.bst_pip_downloads/app_no_8-0.1.tar.gz'])
  @pytest.mark.datafiles(DATA_DIR)
 -def test_pip_source_build(cli, tmpdir, datafiles):
 +def test_pip_source_build(cli, tmpdir, datafiles, setup_pypi_repo):
      project = os.path.join(datafiles.dirname, datafiles.basename)
      element_path = os.path.join(project, 'elements')
      element_name = 'pip/hello.bst'
 +    # check that exotically named packages are imported correctly
 +    myreqs_packages = ['hellolib']
 +    packages = ['app2', 'app.3', 'app-4', 'app_5', 'app.no.6', 'app-no-7', 'app_no_8']
++
 +    # create mock pypi repository
 +    pypi_repo = os.path.join(project, 'files', 'pypi-repo')
 +    os.makedirs(pypi_repo, exist_ok=True)
 +    setup_pypi_repo(myreqs_packages + packages, pypi_repo)
++
      element = {
          'kind': 'manual',
          'depends': ['base.bst'],
@@ -71,16 +96,15 @@ def test_pip_source_build(cli, tmpdir, datafiles):
              },
+             {
                  'kind': 'pip',
 -                'url': 'file://{}'.format(os.path.realpath(os.path.join(project, 'files', 'pypi-repo'))),
 +                'url': 'file://{}'.format(os.path.realpath(pypi_repo)),
                  'requirements-files': ['myreqs.txt'],
 -                'packages': ['app2']
 +                'packages': packages
+             }
          ],
          'config': {
              'install-commands': [
                  'pip3 install --no-index --prefix %{install-root}/usr .bst_pip_downloads/*.tar.gz',
 -                'chmod +x app1.py',
 -                'install app1.py  %{install-root}/usr/bin/'
 +                'install app1.py %{install-root}/usr/bin/'
+             ]
+         }
+     }
@@ -95,5 +119,4 @@ def test_pip_source_build(cli, tmpdir, datafiles):
      result = cli.run(project=project, args=['shell', element_name, '/usr/bin/app1.py'])
      assert result.exit_code == 0
 -    assert result.output == """Hello App1!
 -"""
 +    assert result.output == "Hello App1! This is hellolib\n"

tests/integration/project/files/pypi-repo/app2/App2-0.1.tar.gz deleted

No preview for this file type

tests/integration/project/files/pypi-repo/app2/index.html deleted

 -<html>
 -  <head>
 -    <title>Links for app1</title>
 -  </head>
 -  <body>
 -    <a href="">'App2-0.1.tar.gz'>App2-0.1.tar.gz</a><br />
 -  </body>
 -</html>

tests/integration/project/files/pypi-repo/hellolib/HelloLib-0.1.tar.gz deleted

No preview for this file type

tests/integration/project/files/pypi-repo/hellolib/index.html deleted

 -<html>
 -  <head>
 -    <title>Links for app1</title>
 -  </head>
 -  <body>
 -    <a href="">'HelloLib-0.1.tar.gz'>HelloLib-0.1.tar.gz</a><br />
 -  </body>
 -</html>

tests/integration/shell.py

@@ -278,7 +278,7 @@ def test_workspace_visible(cli, tmpdir, datafiles):
      # Open a workspace on our build failing element
+     #
 -    res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
 +    res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
      assert res.exit_code == 0
      # Ensure the dependencies of our build failing element are built

tests/integration/workspace.py

@@ -23,7 +23,7 @@ def test_workspace_mount(cli, tmpdir, datafiles):
      workspace = os.path.join(cli.directory, 'workspace')
      element_name = 'workspace/workspace-mount.bst'
 -    res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
 +    res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
      assert res.exit_code == 0
      res = cli.run(project=project, args=['build', element_name])
@@ -39,7 +39,7 @@ def test_workspace_commanddir(cli, tmpdir, datafiles):
      workspace = os.path.join(cli.directory, 'workspace')
      element_name = 'workspace/workspace-commanddir.bst'
 -    res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
 +    res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
      assert res.exit_code == 0
      res = cli.run(project=project, args=['build', element_name])
@@ -75,7 +75,7 @@ def test_workspace_updated_dependency(cli, tmpdir, datafiles):
      _yaml.dump(dependency, os.path.join(element_path, dep_name))
      # First open the workspace
 -    res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
 +    res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
      assert res.exit_code == 0
      # We build the workspaced element, so that we have an artifact
@@ -130,7 +130,7 @@ def test_workspace_update_dependency_failed(cli, tmpdir, datafiles):
      _yaml.dump(dependency, os.path.join(element_path, dep_name))
      # First open the workspace
 -    res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
 +    res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
      assert res.exit_code == 0
      # We build the workspaced element, so that we have an artifact
@@ -205,7 +205,7 @@ def test_updated_dependency_nested(cli, tmpdir, datafiles):
      _yaml.dump(dependency, os.path.join(element_path, dep_name))
      # First open the workspace
 -    res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
 +    res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
      assert res.exit_code == 0
      # We build the workspaced element, so that we have an artifact
@@ -258,7 +258,7 @@ def test_incremental_configure_commands_run_only_once(cli, tmpdir, datafiles):
      _yaml.dump(element, os.path.join(element_path, element_name))
      # We open a workspace on the above element
 -    res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
 +    res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
      res.assert_success()
      # Then we build, and check whether the configure step succeeded

tests/plugins/filter.py

@@ -108,7 +108,7 @@ def test_filter_forbid_also_rdep(datafiles, cli):
  def test_filter_workspace_open(datafiles, cli, tmpdir):
      project = os.path.join(datafiles.dirname, datafiles.basename)
      workspace_dir = os.path.join(tmpdir.dirname, tmpdir.basename, "workspace")
 -    result = cli.run(project=project, args=['workspace', 'open', 'deps-permitted.bst', workspace_dir])
 +    result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace_dir, 'deps-permitted.bst'])
      result.assert_success()
      assert os.path.exists(os.path.join(workspace_dir, "foo"))
      assert os.path.exists(os.path.join(workspace_dir, "bar"))
@@ -120,7 +120,7 @@ def test_filter_workspace_build(datafiles, cli, tmpdir):
      project = os.path.join(datafiles.dirname, datafiles.basename)
      tempdir = os.path.join(tmpdir.dirname, tmpdir.basename)
      workspace_dir = os.path.join(tempdir, "workspace")
 -    result = cli.run(project=project, args=['workspace', 'open', 'output-orphans.bst', workspace_dir])
 +    result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace_dir, 'output-orphans.bst'])
      result.assert_success()
      src = os.path.join(workspace_dir, "foo")
      dst = os.path.join(workspace_dir, "quux")
@@ -138,7 +138,7 @@ def test_filter_workspace_close(datafiles, cli, tmpdir):
      project = os.path.join(datafiles.dirname, datafiles.basename)
      tempdir = os.path.join(tmpdir.dirname, tmpdir.basename)
      workspace_dir = os.path.join(tempdir, "workspace")
 -    result = cli.run(project=project, args=['workspace', 'open', 'output-orphans.bst', workspace_dir])
 +    result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace_dir, 'output-orphans.bst'])
      result.assert_success()
      src = os.path.join(workspace_dir, "foo")
      dst = os.path.join(workspace_dir, "quux")
@@ -158,7 +158,7 @@ def test_filter_workspace_reset(datafiles, cli, tmpdir):
      project = os.path.join(datafiles.dirname, datafiles.basename)
      tempdir = os.path.join(tmpdir.dirname, tmpdir.basename)
      workspace_dir = os.path.join(tempdir, "workspace")
 -    result = cli.run(project=project, args=['workspace', 'open', 'output-orphans.bst', workspace_dir])
 +    result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace_dir, 'output-orphans.bst'])
      result.assert_success()
      src = os.path.join(workspace_dir, "foo")
      dst = os.path.join(workspace_dir, "quux")

tests/sources/pip.py

@@ -3,6 +3,7 @@ import pytest
  from buildstream._exceptions import ErrorDomain
  from buildstream import _yaml
 +from buildstream.plugins.sources.pip import _match_package_name
  from tests.testutils import cli
  DATA_DIR = os.path.join(
@@ -45,3 +46,22 @@ def test_no_packages(cli, tmpdir, datafiles):
          'show', 'target.bst'
      ])
      result.assert_main_error(ErrorDomain.SOURCE, None)
++
++
 +# Test that pip source parses tar ball names correctly for the ref
 +@pytest.mark.parametrize(
 +    'tarball, expected_name, expected_version',
 +    [
 +        ('dotted.package-0.9.8.tar.gz', 'dotted.package', '0.9.8'),
 +        ('hyphenated-package-2.6.0.tar.gz', 'hyphenated-package', '2.6.0'),
 +        ('underscore_pkg-3.1.0.tar.gz', 'underscore_pkg', '3.1.0'),
 +        ('numbers2and5-1.0.1.tar.gz', 'numbers2and5', '1.0.1'),
 +        ('multiple.dots.package-5.6.7.tar.gz', 'multiple.dots.package', '5.6.7'),
 +        ('multiple-hyphens-package-1.2.3.tar.gz', 'multiple-hyphens-package', '1.2.3'),
 +        ('multiple_underscore_pkg-3.4.5.tar.gz', 'multiple_underscore_pkg', '3.4.5'),
 +        ('shortversion-1.0.tar.gz', 'shortversion', '1.0'),
 +        ('longversion-1.2.3.4.tar.gz', 'longversion', '1.2.3.4')
 +    ])
 +def test_match_package_name(tarball, expected_name, expected_version):
 +    name, version = _match_package_name(tarball)
 +    assert (expected_name, expected_version) == (name, version)

tests/testutils/__init__.py

@@ -29,3 +29,4 @@ from .artifactshare import create_artifact_share
  from .element_generators import create_element_size, update_element_size
  from .junction import generate_junction
  from .runner_integration import wait_for_cache_granularity
 +from .python_repo import setup_pypi_repo

tests/testutils/python_repo.py

 +from setuptools.sandbox import run_setup
 +import os
 +import pytest
 +import re
 +import shutil
++
++
 +SETUP_TEMPLATE = '''\
 +from setuptools import setup
++
 +setup(
 +    name='{name}',
 +    version='{version}',
 +    description='{name}',
 +    packages=['{pkgdirname}'],
 +    entry_points={{
 +        'console_scripts': [
 +            '{pkgdirname}={pkgdirname}:main'
 +        ]
 +    }}
 +)
 +'''
++
 +# All packages generated via generate_pip_package will have the functions below
 +INIT_TEMPLATE = '''\
 +def main():
 +    print('This is {name}')
++
 +def hello(actor='world'):
 +    print('Hello {{}}! This is {name}'.format(actor))
 +'''
++
 +HTML_TEMPLATE = '''\
 +<html>
 +  <head>
 +    <title>Links for {name}</title>
 +  </head>
 +  <body>
 +    <a href=''>{name}-{version}.tar.gz</a><br />
 +  </body>
 +</html>
 +'''
++
++
 +# Creates a simple python source distribution and copies this into a specified
 +# directory which is to serve as a mock python repository
 +#
 +# Args:
 +#    tmpdir (str): Directory in which the source files will be created
 +#    pypi (str): Directory serving as a mock python repository
 +#    name (str): The name of the package to be created
 +#    version (str): The version of the package to be created
 +#
 +# Returns:
 +#    None
 +#
 +def generate_pip_package(tmpdir, pypi, name, version='0.1'):
 +    # check if package already exists in pypi
 +    pypi_package = os.path.join(pypi, re.sub('[^0-9a-zA-Z]+', '-', name))
 +    if os.path.exists(pypi_package):
 +        return
++
 +    # create the package source files in tmpdir resulting in a directory
 +    # tree resembling the following structure:
 +    #
 +    # tmpdir
 +    # |-- setup.py
 +    # `-- package
 +    #     `-- __init__.py
 +    #
 +    setup_file = os.path.join(tmpdir, 'setup.py')
 +    pkgdirname = re.sub('[^0-9a-zA-Z]+', '', name)
 +    with open(setup_file, 'w') as f:
 +        f.write(
 +            SETUP_TEMPLATE.format(
 +                name=name,
 +                version=version,
 +                pkgdirname=pkgdirname
 +            )
 +        )
 +    os.chmod(setup_file, 0o755)
++
 +    package = os.path.join(tmpdir, pkgdirname)
 +    os.makedirs(package)
++
 +    main_file = os.path.join(package, '__init__.py')
 +    with open(main_file, 'w') as f:
 +        f.write(INIT_TEMPLATE.format(name=name))
 +    os.chmod(main_file, 0o644)
++
 +    run_setup(setup_file, ['sdist'])
++
 +    # create directory for this package in pypi resulting in a directory
 +    # tree resembling the following structure:
 +    #
 +    # pypi
 +    # `-- pypi_package
 +    #     |-- index.html
 +    #     `-- foo-0.1.tar.gz
 +    #
 +    os.makedirs(pypi_package)
++
 +    # add an index html page
 +    index_html = os.path.join(pypi_package, 'index.html')
 +    with open(index_html, 'w') as f:
 +        f.write(HTML_TEMPLATE.format(name=name, version=version))
++
 +    # copy generated tarfile to pypi package
 +    dist_dir = os.path.join(tmpdir, 'dist')
 +    for tar in os.listdir(dist_dir):
 +        tarpath = os.path.join(dist_dir, tar)
 +        shutil.copy(tarpath, pypi_package)
++
++
 +@pytest.fixture
 +def setup_pypi_repo(tmpdir):
 +    def create_pkgdir(package):
 +        pkgdirname = re.sub('[^0-9a-zA-Z]+', '', package)
 +        pkgdir = os.path.join(str(tmpdir), pkgdirname)
 +        os.makedirs(pkgdir)
 +        return pkgdir
++
 +    def add_packages(packages, pypi_repo):
 +        for package in packages:
 +            pkgdir = create_pkgdir(package)
 +            generate_pip_package(pkgdir, pypi_repo, package)
++
 +    return add_packages

...	...	@@ -22,6 +22,9 @@ artifactdir: ${XDG_CACHE_HOME}/buildstream/artifacts
22	22	# Location to store build logs
23	23	logdir: ${XDG_CACHE_HOME}/buildstream/logs
24	24
	25	+# Default root location for workspaces, blank for no default set.
	26	+workspacedir: .
	27	+
25	28	#
26	29	# Cache
27	30	#

[Notes] [Git][BuildStream/buildstream][willsalmon/defaultWorkspaces] 9 commits: plugins/sources/pip.py: Accomodate characters '-', '.', '_' for packages

Will Salmon pushed to branch willsalmon/defaultWorkspaces at BuildStream / buildstream

Commits:

24 changed files:

Changes: