GNOME.org
 

[Notes] [Git][BuildStream/buildstream][tpollard/494] 17 commits: _frontend/app.py: Set correct element-path in interactive bst-init

Title: GitLab

Tom Pollard pushed to branch tpollard/494 at BuildStream / buildstream

Commits:

28 changed files:

Changes:

  • .gitlab-ci.yml
    ... ... @@ -166,6 +166,12 @@ docs:
    166 166 
         BST_EXT_REF: 1d6ab71151b93c8cbc0a91a36ffe9270f3b835f1 # 0.5.1
    167 167 
         FD_SDK_REF: 88d7c22c2281b987faa02edd57df80d430eecf1f # 18.08.11-35-g88d7c22c
    168 168 
       before_script:
    169 
    +  - |
    170 
    +    mkdir -p "${HOME}/.config"
    171 
    +    cat <<EOF >"${HOME}/.config/buildstream.conf"
    172 
    +    scheduler:
    173 
    +      fetchers: 2
    174 
    +    EOF
    169 175 
       - (cd dist && ./unpack.sh && cd buildstream && pip3 install .)
    170 176 
       - pip3 install --user -e ${BST_EXT_URL}@${BST_EXT_REF}#egg=bst_ext
    171 177 
       - git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk.git

  • NEWS
    ... ... @@ -2,6 +2,12 @@
    2 2 
     buildstream 1.3.1
    3 3 
     =================
    4 4
    5 
    +  o BREAKING CHANGE: The 'manual' element lost its default 'MAKEFLAGS' and 'V'
    6 
    +    environment variables. There is already a 'make' element with the same
    7 
    +    variables. Note that this is a breaking change, it will require users to
    8 
    +    make changes to their .bst files if they are expecting these environment
    9 
    +    variables to be set.
    10 
    +
    5 11 
       o Failed builds are included in the cache as well.
    6 12 
         `bst checkout` will provide anything in `%{install-root}`.
    7 13 
         A build including cached fails will cause any dependant elements
    ... ... @@ -32,13 +38,23 @@ buildstream 1.3.1
    32 38 
         a bug fix to workspaces so they can be build in workspaces too.
    33 39
    34 40 
       o Creating a build shell through the interactive mode or `bst shell --build`
    35 
    -    will now use the cached build tree. It is now easier to debug local build
    36 
    -    failures.
    41 
    +    will now use the cached buildtree if available locally. It is now easier to
    42 
    +    debug local build failures.
    37 43
    38 44 
       o `bst shell --sysroot` now takes any directory that contains a sysroot,
    39 45 
         instead of just a specially-formatted build-root with a `root` and `scratch`
    40 46 
         subdirectory.
    41 47
    48 
    +  o Due to the element `build tree` being cached in the respective artifact their
    49 
    +    size in some cases has significantly increased. In *most* cases the build tree
    50 
    +    is not utilised when building targets, as such by default bst 'pull' & 'build'
    51 
    +    will not fetch buildtrees from remotes. This behaviour can be overriden with
    52 
    +    the cli main option '--pull-build-trees', or the user configuration option
    53 
    +    'pullbuildtrees = True'. The override will also add the build tree to already
    54 
    +    cached artifacts. When attempting to populate an artifactcache server with
    55 
    +    cached artifacts, only 'complete' elements can be pushed. If the element is
    56 
    +    expected to have a populated build tree then it must be cached before pushing.
    57 
    +
    42 58
    43 59 
     =================
    44 60 
     buildstream 1.1.5

  • buildstream/_artifactcache/artifactcache.py
    ... ... @@ -428,6 +428,22 @@ class ArtifactCache():
    428 428 
             raise ImplError("Cache '{kind}' does not implement contains()"
    429 429 
                             .format(kind=type(self).__name__))
    430 430
    431 
    +    # contains_subdir_artifact():
    432 
    +    #
    433 
    +    # Check whether an artifact element contains a digest for a subdir
    434 
    +    # which is populated in the cache, i.e non dangling.
    435 
    +    #
    436 
    +    # Args:
    437 
    +    #     element (Element): The Element to check
    438 
    +    #     key (str): The cache key to use
    439 
    +    #     subdir (str): The subdir to check
    440 
    +    #
    441 
    +    # Returns: True if the subdir exists & is populated in the cache, False otherwise
    442 
    +    #
    443 
    +    def contains_subdir_artifact(self, element, key, subdir):
    444 
    +        raise ImplError("Cache '{kind}' does not implement contains_subdir_artifact()"
    445 
    +                        .format(kind=type(self).__name__))
    446 
    +
    431 447 
         # list_artifacts():
    432 448 
         #
    433 449 
         # List artifacts in this cache in LRU order.
    ... ... @@ -464,6 +480,7 @@ class ArtifactCache():
    464 480 
         # Args:
    465 481 
         #     element (Element): The Element to extract
    466 482 
         #     key (str): The cache key to use
    483 
    +    #     subdir (str): Optional specific dir to extract
    467 484 
         #
    468 485 
         # Raises:
    469 486 
         #     ArtifactError: In cases there was an OSError, or if the artifact
    ... ... @@ -471,7 +488,7 @@ class ArtifactCache():
    471 488 
         #
    472 489 
         # Returns: path to extracted artifact
    473 490 
         #
    474 
    -    def extract(self, element, key):
    491 
    +    def extract(self, element, key, subdir=None):
    475 492 
             raise ImplError("Cache '{kind}' does not implement extract()"
    476 493 
                             .format(kind=type(self).__name__))
    477 494
    ... ... @@ -553,11 +570,13 @@ class ArtifactCache():
    553 570 
         #     element (Element): The Element whose artifact is to be fetched
    554 571 
         #     key (str): The cache key to use
    555 572 
         #     progress (callable): The progress callback, if any
    573 
    +    #     subdir (str): The optional specific subdir to pull
    574 
    +    #     excluded_subdirs (list): The optional list of subdirs to not pull
    556 575 
         #
    557 576 
         # Returns:
    558 577 
         #   (bool): True if pull was successful, False if artifact was not available
    559 578 
         #
    560 
    -    def pull(self, element, key, *, progress=None):
    579 
    +    def pull(self, element, key, *, progress=None, subdir=None, excluded_subdirs=None):
    561 580 
             raise ImplError("Cache '{kind}' does not implement pull()"
    562 581 
                             .format(kind=type(self).__name__))
    563 582

  • buildstream/_artifactcache/cascache.py
    ... ... @@ -93,16 +93,36 @@ class CASCache(ArtifactCache):
    93 93 
             # This assumes that the repository doesn't have any dangling pointers
    94 94 
             return os.path.exists(refpath)
    95 95
    96 
    -    def extract(self, element, key):
    96 
    +    def contains_subdir_artifact(self, element, key, subdir):
    97 
    +        tree = self.resolve_ref(self.get_artifact_fullname(element, key))
    98 
    +
    99 
    +        # This assumes that the subdir digest is present in the element tree
    100 
    +        subdirdigest = self._get_subdir(tree, subdir)
    101 
    +        objpath = self.objpath(subdirdigest)
    102 
    +
    103 
    +        # True if subdir content is cached or if empty as expected
    104 
    +        return os.path.exists(objpath)
    105 
    +
    106 
    +    def extract(self, element, key, subdir=None):
    97 107 
             ref = self.get_artifact_fullname(element, key)
    98 108
    99 109 
             tree = self.resolve_ref(ref, update_mtime=True)
    100 110
    101 
    -        dest = os.path.join(self.extractdir, element._get_project().name,
    102 
    -                            element.normal_name, tree.hash)
    111 
    +        elementdest = dest = os.path.join(self.extractdir, element._get_project().name,
    112 
    +                                          element.normal_name, tree.hash)
    113 
    +
    114 
    +        # If artifact is already extracted, check if the optional subdir
    115 
    +        # has also been extracted. If the artifact has not been extracted
    116 
    +        # a full extraction would include the optional subdir
    103 117 
             if os.path.isdir(dest):
    104 
    -            # artifact has already been extracted
    105 
    -            return dest
    118 
    +            if subdir:
    119 
    +                if not os.path.isdir(os.path.join(dest, subdir)):
    120 
    +                    dest = os.path.join(dest, subdir)
    121 
    +                    tree = self._get_subdir(tree, subdir)
    122 
    +                else:
    123 
    +                    return dest
    124 
    +            else:
    125 
    +                return dest
    106 126
    107 127 
             with tempfile.TemporaryDirectory(prefix='tmp', dir=self.extractdir) as tmpdir:
    108 128 
                 checkoutdir = os.path.join(tmpdir, ref)
    ... ... @@ -121,7 +141,7 @@ class CASCache(ArtifactCache):
    121 141 
                         raise ArtifactError("Failed to extract artifact for ref '{}': {}"
    122 142 
                                             .format(ref, e)) from e
    123 143
    124 
    -        return dest
    144 
    +        return elementdest
    125 145
    126 146 
         def commit(self, element, content, keys):
    127 147 
             refs = [self.get_artifact_fullname(element, key) for key in keys]
    ... ... @@ -229,7 +249,7 @@ class CASCache(ArtifactCache):
    229 249 
                 remotes_for_project = self._remotes[element._get_project()]
    230 250 
                 return any(remote.spec.push for remote in remotes_for_project)
    231 251
    232 
    -    def pull(self, element, key, *, progress=None):
    252 
    +    def pull(self, element, key, *, progress=None, subdir=None, excluded_subdirs=None):
    233 253 
             ref = self.get_artifact_fullname(element, key)
    234 254
    235 255 
             project = element._get_project()
    ... ... @@ -248,8 +268,18 @@ class CASCache(ArtifactCache):
    248 268 
                     tree.hash = response.digest.hash
    249 269 
                     tree.size_bytes = response.digest.size_bytes
    250 270
    251 
    -                self._fetch_directory(remote, tree)
    271 
    +                # Check if the element artifact is present, if so just fetch the subdir
    272 
    +                # and extract into artifact extract dir if it already exists. If the
    273 
    +                # respective artifact extract dir does not exist a complete extraction
    274 
    +                # will complete
    275 
    +                if subdir and os.path.exists(self.objpath(tree)):
    276 
    +                    self._fetch_subdir(remote, tree, subdir)
    277 
    +                    self.extract(element, key, subdir)
    278 
    +                else:
    279 
    +                    # Fetch artifact, excluded_subdirs determined in pullqueue
    280 
    +                    self._fetch_directory(remote, tree, excluded_subdirs=excluded_subdirs)
    252 281
    282 
    +                # tree is the remote value, so is the same without or without dangling ref locally
    253 283 
                     self.set_ref(ref, tree)
    254 284
    255 285 
                     element.info("Pulled artifact {} <- {}".format(display_key, remote.spec.url))
    ... ... @@ -673,8 +703,10 @@ class CASCache(ArtifactCache):
    673 703 
                              stat.S_IRGRP | stat.S_IXGRP | stat.S_IROTH | stat.S_IXOTH)
    674 704
    675 705 
             for dirnode in directory.directories:
    676 
    -            fullpath = os.path.join(dest, dirnode.name)
    677 
    -            self._checkout(fullpath, dirnode.digest)
    706 
    +            # Don't try to checkout a dangling ref
    707 
    +            if os.path.exists(self.objpath(dirnode.digest)):
    708 
    +                fullpath = os.path.join(dest, dirnode.name)
    709 
    +                self._checkout(fullpath, dirnode.digest)
    678 710
    679 711 
             for symlinknode in directory.symlinks:
    680 712 
                 # symlink
    ... ... @@ -952,11 +984,14 @@ class CASCache(ArtifactCache):
    952 984 
         # Args:
    953 985 
         #     remote (Remote): The remote to use.
    954 986 
         #     dir_digest (Digest): Digest object for the directory to fetch.
    987 
    +    #     excluded_subdirs (list): The optional list of subdirs to not fetch
    955 988 
         #
    956 
    -    def _fetch_directory(self, remote, dir_digest):
    989 
    +    def _fetch_directory(self, remote, dir_digest, *, excluded_subdirs=None):
    957 990 
             fetch_queue = [dir_digest]
    958 991 
             fetch_next_queue = []
    959 992 
             batch = _CASBatchRead(remote)
    993 
    +        if not excluded_subdirs:
    994 
    +            excluded_subdirs = []
    960 995
    961 996 
             while len(fetch_queue) + len(fetch_next_queue) > 0:
    962 997 
                 if not fetch_queue:
    ... ... @@ -971,8 +1006,9 @@ class CASCache(ArtifactCache):
    971 1006 
                     directory.ParseFromString(f.read())
    972 1007
    973 1008 
                 for dirnode in directory.directories:
    974 
    -                batch = self._fetch_directory_node(remote, dirnode.digest, batch,
    975 
    -                                                   fetch_queue, fetch_next_queue, recursive=True)
    1009 
    +                if dirnode.name not in excluded_subdirs:
    1010 
    +                    batch = self._fetch_directory_node(remote, dirnode.digest, batch,
    1011 
    +                                                       fetch_queue, fetch_next_queue, recursive=True)
    976 1012
    977 1013 
                 for filenode in directory.files:
    978 1014 
                     batch = self._fetch_directory_node(remote, filenode.digest, batch,
    ... ... @@ -981,6 +1017,10 @@ class CASCache(ArtifactCache):
    981 1017 
             # Fetch final batch
    982 1018 
             self._fetch_directory_batch(remote, batch, fetch_queue, fetch_next_queue)
    983 1019
    1020 
    +    def _fetch_subdir(self, remote, tree, subdir):
    1021 
    +        subdirdigest = self._get_subdir(tree, subdir)
    1022 
    +        self._fetch_directory(remote, subdirdigest)
    1023 
    +
    984 1024 
         def _fetch_tree(self, remote, digest):
    985 1025 
             # download but do not store the Tree object
    986 1026 
             with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:

  • buildstream/_context.py
    ... ... @@ -105,6 +105,9 @@ class Context():
    105 105 
             # What to do when a build fails in non interactive mode
    106 106 
             self.sched_error_action = 'continue'
    107 107
    108 
    +        # Whether or not to attempt to pull build trees globally
    109 
    +        self.pull_build_trees = False
    110 
    +
    108 111 
             # Whether elements must be rebuilt when their dependencies have changed
    109 112 
             self._strict_build_plan = None
    110 113
    ... ... @@ -161,7 +164,7 @@ class Context():
    161 164 
             _yaml.node_validate(defaults, [
    162 165 
                 'sourcedir', 'builddir', 'artifactdir', 'logdir',
    163 166 
                 'scheduler', 'artifacts', 'logging', 'projects',
    164 
    -            'cache'
    167 
    +            'cache', 'pullbuildtrees'
    165 168 
             ])
    166 169
    167 170 
             for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir']:
    ... ... @@ -186,6 +189,9 @@ class Context():
    186 189 
             # Load artifact share configuration
    187 190 
             self.artifact_cache_specs = ArtifactCache.specs_from_config_node(defaults)
    188 191
    192 
    +        # Load pull build trees configuration
    193 
    +        self.pull_build_trees = _yaml.node_get(defaults, bool, 'pullbuildtrees', default_value='False')
    194 
    +
    189 195 
             # Load logging config
    190 196 
             logging = _yaml.node_get(defaults, Mapping, 'logging')
    191 197 
             _yaml.node_validate(logging, [

  • buildstream/_frontend/app.py
    ... ... @@ -182,7 +182,8 @@ class App():
    182 182 
                 'fetchers': 'sched_fetchers',
    183 183 
                 'builders': 'sched_builders',
    184 184 
                 'pushers': 'sched_pushers',
    185 
    -            'network_retries': 'sched_network_retries'
    185 
    +            'network_retries': 'sched_network_retries',
    186 
    +            'pull_build_trees': 'pull_build_trees'
    186 187 
             }
    187 188 
             for cli_option, context_attr in override_map.items():
    188 189 
                 option_value = self._main_options.get(cli_option)
    ... ... @@ -305,7 +306,6 @@ class App():
    305 306 
             directory = self._main_options['directory']
    306 307 
             directory = os.path.abspath(directory)
    307 308 
             project_path = os.path.join(directory, 'project.conf')
    308 
    -        elements_path = os.path.join(directory, element_path)
    309 309
    310 310 
             try:
    311 311 
                 # Abort if the project.conf already exists, unless `--force` was specified in `bst init`
    ... ... @@ -335,6 +335,7 @@ class App():
    335 335 
                     raise AppError("Error creating project directory {}: {}".format(directory, e)) from e
    336 336
    337 337 
                 # Create the elements sub-directory if it doesnt exist
    338 
    +            elements_path = os.path.join(directory, element_path)
    338 339 
                 try:
    339 340 
                     os.makedirs(elements_path, exist_ok=True)
    340 341 
                 except IOError as e:

  • buildstream/_frontend/cli.py
    ... ... @@ -219,6 +219,8 @@ def print_version(ctx, param, value):
    219 219 
                   help="Specify a project option")
    220 220 
     @click.option('--default-mirror', default=None,
    221 221 
                   help="The mirror to fetch from first, before attempting other mirrors")
    222 
    +@click.option('--pull-build-trees', is_flag=True, default=None,
    223 
    +              help="Include an element's build trees when pulling remote element artifacts")
    222 224 
     @click.pass_context
    223 225 
     def cli(context, **kwargs):
    224 226 
         """Build and manipulate BuildStream projects

  • buildstream/_scheduler/queues/pullqueue.py
    ... ... @@ -32,9 +32,20 @@ class PullQueue(Queue):
    32 32 
         complete_name = "Pulled"
    33 33 
         resources = [ResourceType.DOWNLOAD, ResourceType.CACHE]
    34 34
    35 
    +    def __init__(self, scheduler):
    36 
    +        super().__init__(scheduler)
    37 
    +
    38 
    +        # Current default exclusions on pull
    39 
    +        self._excluded_subdirs = ["buildtree"]
    40 
    +        self._subdir = None
    41 
    +        # If build trees are to be pulled, remove the value from exclusion list
    42 
    +        if self._scheduler.context.pull_build_trees:
    43 
    +            self._subdir = "buildtree"
    44 
    +            self._excluded_subdirs.remove(self._subdir)
    45 
    +
    35 46 
         def process(self, element):
    36 47 
             # returns whether an artifact was downloaded or not
    37 
    -        if not element._pull():
    48 
    +        if not element._pull(subdir=self._subdir, excluded_subdirs=self._excluded_subdirs):
    38 49 
                 raise SkipJob(self.action_name)
    39 50
    40 51 
         def status(self, element):
    ... ... @@ -49,7 +60,7 @@ class PullQueue(Queue):
    49 60 
             if not element._can_query_cache():
    50 61 
                 return QueueStatus.WAIT
    51 62
    52 
    -        if element._pull_pending():
    63 
    +        if element._pull_pending(subdir=self._subdir):
    53 64 
                 return QueueStatus.READY
    54 65 
             else:
    55 66 
                 return QueueStatus.SKIP

  • buildstream/_versions.py
    ... ... @@ -23,7 +23,7 @@
    23 23 
     # This version is bumped whenever enhancements are made
    24 24 
     # to the `project.conf` format or the core element format.
    25 25 
     #
    26 
    -BST_FORMAT_VERSION = 17
    26 
    +BST_FORMAT_VERSION = 18
    27 27
    28 28
    29 29 
     # The base BuildStream artifact version

  • buildstream/_yaml.py
    ... ... @@ -1049,6 +1049,12 @@ class ChainMap(collections.ChainMap):
    1049 1049 
             for key in clearable:
    1050 1050 
                 del self[key]
    1051 1051
    1052 
    +    def get(self, key, default=None):
    1053 
    +        try:
    1054 
    +            return self[key]
    1055 
    +        except KeyError:
    1056 
    +            return default
    1057 
    +
    1052 1058
    1053 1059 
     def node_chain_copy(source):
    1054 1060 
         copy = ChainMap({}, source)

  • buildstream/data/userconfig.yaml
    ... ... @@ -97,3 +97,5 @@ logging:
    97 97
    98 98 
         [%{elapsed}][%{key}][%{element}] %{action} %{message}
    99 99
    100 
    +# Whether to pull buildtrees when downloading element artifacts
    101 
    +pullbuildtrees: False

  • buildstream/element.py
    ... ... @@ -1399,9 +1399,18 @@ class Element(Plugin):
    1399 1399 
                                                      .format(workspace.get_absolute_path())):
    1400 1400 
                                 workspace.stage(temp_staging_directory)
    1401 1401 
                     elif self._cached():
    1402 
    -                    # We have a cached buildtree to use, instead
    1403 
    -                    artifact_base, _ = self.__extract()
    1404 
    -                    import_dir = os.path.join(artifact_base, 'buildtree')
    1402 
    +                    # Check if we have a cached buildtree to use
    1403 
    +                    context = self._get_context()
    1404 
    +                    if context.get_strict():
    1405 
    +                        if self.__artifacts.contains_subdir_artifact(self, self.__strict_cache_key, 'buildtree'):
    1406 
    +                            artifact_base, _ = self.__extract()
    1407 
    +                            import_dir = os.path.join(artifact_base, 'buildtree')
    1408 
    +                    elif self.__artifacts.contains_subdir_artifact(self, self.__weak_cache_key, 'buildtree'):
    1409 
    +                        artifact_base, _ = self.__extract()
    1410 
    +                        import_dir = os.path.join(artifact_base, 'buildtree')
    1411 
    +                    else:
    1412 
    +                        self.warn("{} is cached without a buildtree, the source will be staged instead"
    1413 
    +                                  .format(self.name))
    1405 1414 
                     else:
    1406 1415 
                         # No workspace, stage directly
    1407 1416 
                         for source in self.sources():
    ... ... @@ -1699,18 +1708,26 @@ class Element(Plugin):
    1699 1708
    1700 1709 
         # _pull_pending()
    1701 1710 
         #
    1702 
    -    # Check whether the artifact will be pulled.
    1711 
    +    # Check whether the artifact will be pulled. If the pull operation is to
    1712 
    +    # include a specific subdir of the element artifact (from cli or user conf)
    1713 
    +    # then the local cache is queried for the subdirs existence.
    1714 
    +    #
    1715 
    +    # Args:
    1716 
    +    #    subdir (str): Whether the pull has been invoked with a specific subdir set
    1703 1717 
         #
    1704 1718 
         # Returns:
    1705 1719 
         #   (bool): Whether a pull operation is pending
    1706 1720 
         #
    1707 
    -    def _pull_pending(self):
    1721 
    +    def _pull_pending(self, subdir=None):
    1708 1722 
             if self._get_workspace():
    1709 1723 
                 # Workspace builds are never pushed to artifact servers
    1710 1724 
                 return False
    1711 1725
    1712 
    -        if self.__strong_cached:
    1713 
    -            # Artifact already in local cache
    1726 
    +        if self.__strong_cached and subdir:
    1727 
    +            # If we've specified a subdir, check if the subdir is cached locally
    1728 
    +            if self.__artifacts.contains_subdir_artifact(self, self.__strict_cache_key, subdir):
    1729 
    +                return False
    1730 
    +        elif self.__strong_cached:
    1714 1731 
                 return False
    1715 1732
    1716 1733 
             # Pull is pending if artifact remote server available
    ... ... @@ -1732,50 +1749,27 @@ class Element(Plugin):
    1732 1749
    1733 1750 
             self._update_state()
    1734 1751
    1735 
    -    def _pull_strong(self, *, progress=None):
    1736 
    -        weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
    1737 
    -
    1738 
    -        key = self.__strict_cache_key
    1739 
    -        if not self.__artifacts.pull(self, key, progress=progress):
    1740 
    -            return False
    1741 
    -
    1742 
    -        # update weak ref by pointing it to this newly fetched artifact
    1743 
    -        self.__artifacts.link_key(self, key, weak_key)
    1744 
    -
    1745 
    -        return True
    1746 
    -
    1747 
    -    def _pull_weak(self, *, progress=None):
    1748 
    -        weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
    1749 
    -
    1750 
    -        if not self.__artifacts.pull(self, weak_key, progress=progress):
    1751 
    -            return False
    1752 
    -
    1753 
    -        # extract strong cache key from this newly fetched artifact
    1754 
    -        self._pull_done()
    1755 
    -
    1756 
    -        # create tag for strong cache key
    1757 
    -        key = self._get_cache_key(strength=_KeyStrength.STRONG)
    1758 
    -        self.__artifacts.link_key(self, weak_key, key)
    1759 
    -
    1760 
    -        return True
    1761 
    -
    1762 1752 
         # _pull():
    1763 1753 
         #
    1764 1754 
         # Pull artifact from remote artifact repository into local artifact cache.
    1765 1755 
         #
    1756 
    +    # Args:
    1757 
    +    #     subdir (str): The optional specific subdir to pull
    1758 
    +    #     excluded_subdirs (list): The optional list of subdirs to not pull
    1759 
    +    #
    1766 1760 
         # Returns: True if the artifact has been downloaded, False otherwise
    1767 1761 
         #
    1768 
    -    def _pull(self):
    1762 
    +    def _pull(self, subdir=None, excluded_subdirs=None):
    1769 1763 
             context = self._get_context()
    1770 1764
    1771 1765 
             def progress(percent, message):
    1772 1766 
                 self.status(message)
    1773 1767
    1774 1768 
             # Attempt to pull artifact without knowing whether it's available
    1775 
    -        pulled = self._pull_strong(progress=progress)
    1769 
    +        pulled = self.__pull_strong(progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs)
    1776 1770
    1777 1771 
             if not pulled and not self._cached() and not context.get_strict():
    1778 
    -            pulled = self._pull_weak(progress=progress)
    1772 
    +            pulled = self.__pull_weak(progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs)
    1779 1773
    1780 1774 
             if not pulled:
    1781 1775 
                 return False
    ... ... @@ -1798,10 +1792,21 @@ class Element(Plugin):
    1798 1792 
             if not self._cached():
    1799 1793 
                 return True
    1800 1794
    1801 
    -        # Do not push tained artifact
    1795 
    +        # Do not push tainted artifact
    1802 1796 
             if self.__get_tainted():
    1803 1797 
                 return True
    1804 1798
    1799 
    +        # strict_cache_key can't be relied on to be available when running in non strict mode
    1800 
    +        context = self._get_context()
    1801 
    +
    1802 
    +        # Do not push elements that have a dangling buildtree artifact unless element type is
    1803 
    +        # expected to have an empty buildtree directory
    1804 
    +        if context.get_strict():
    1805 
    +            if not self.__artifacts.contains_subdir_artifact(self, self.__strict_cache_key, 'buildtree'):
    1806 
    +                return True
    1807 
    +        elif not self.__artifacts.contains_subdir_artifact(self, self.__weak_cache_key, 'buildtree'):
    1808 
    +            return True
    1809 
    +
    1805 1810 
             return False
    1806 1811
    1807 1812 
         # _push():
    ... ... @@ -2180,6 +2185,7 @@ class Element(Plugin):
    2180 2185 
                                         stderr=stderr,
    2181 2186 
                                         config=config,
    2182 2187 
                                         server_url=self.__remote_execution_url,
    2188 
    +                                    bare_directory=bare_directory,
    2183 2189 
                                         allow_real_directory=False)
    2184 2190 
                 yield sandbox
    2185 2191
    ... ... @@ -2681,6 +2687,59 @@ class Element(Plugin):
    2681 2687
    2682 2688 
             return utils._deduplicate(keys)
    2683 2689
    2690 
    +    # __pull_strong():
    2691 
    +    #
    2692 
    +    # Attempt pulling given element from configured artifact caches with
    2693 
    +    # the strict cache key
    2694 
    +    #
    2695 
    +    # Args:
    2696 
    +    #     progress (callable): The progress callback, if any
    2697 
    +    #     subdir (str): The optional specific subdir to pull
    2698 
    +    #     excluded_subdirs (list): The optional list of subdirs to not pull
    2699 
    +    #
    2700 
    +    # Returns:
    2701 
    +    #     (bool): Whether or not the pull was successful
    2702 
    +    #
    2703 
    +    def __pull_strong(self, *, progress=None, subdir=None, excluded_subdirs=None):
    2704 
    +        weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
    2705 
    +        key = self.__strict_cache_key
    2706 
    +        if not self.__artifacts.pull(self, key, progress=progress, subdir=subdir,
    2707 
    +                                     excluded_subdirs=excluded_subdirs):
    2708 
    +            return False
    2709 
    +
    2710 
    +        # update weak ref by pointing it to this newly fetched artifact
    2711 
    +        self.__artifacts.link_key(self, key, weak_key)
    2712 
    +
    2713 
    +        return True
    2714 
    +
    2715 
    +    # __pull_weak():
    2716 
    +    #
    2717 
    +    # Attempt pulling given element from configured artifact caches with
    2718 
    +    # the weak cache key
    2719 
    +    #
    2720 
    +    # Args:
    2721 
    +    #     progress (callable): The progress callback, if any
    2722 
    +    #     subdir (str): The optional specific subdir to pull
    2723 
    +    #     excluded_subdirs (list): The optional list of subdirs to not pull
    2724 
    +    #
    2725 
    +    # Returns:
    2726 
    +    #     (bool): Whether or not the pull was successful
    2727 
    +    #
    2728 
    +    def __pull_weak(self, *, progress=None, subdir=None, excluded_subdirs=None):
    2729 
    +        weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
    2730 
    +        if not self.__artifacts.pull(self, weak_key, progress=progress, subdir=subdir,
    2731 
    +                                     excluded_subdirs=excluded_subdirs):
    2732 
    +            return False
    2733 
    +
    2734 
    +        # extract strong cache key from this newly fetched artifact
    2735 
    +        self._pull_done()
    2736 
    +
    2737 
    +        # create tag for strong cache key
    2738 
    +        key = self._get_cache_key(strength=_KeyStrength.STRONG)
    2739 
    +        self.__artifacts.link_key(self, weak_key, key)
    2740 
    +
    2741 
    +        return True
    2742 
    +
    2684 2743
    2685 2744 
     def _overlap_error_detail(f, forbidden_overlap_elements, elements):
    2686 2745 
         if forbidden_overlap_elements:

  • buildstream/plugins/elements/manual.yaml
    1 
    -# No variables added for the manual element by default, set
    2 
    -# this if you plan to use make, and the sources cannot handle
    3 
    -# parallelization.
    4 
    -#
    5 
    -# variables:
    6 
    -#
    7 
    -#   notparallel: True
    8 
    -
    9 1 
     # Manual build element does not provide any default
    10 2 
     # build commands
    11 3 
     config:
    ... ... @@ -28,14 +20,3 @@ config:
    28 20 
       strip-commands:
    29 21 
       - |
    30 22 
         %{strip-binaries}
    31 
    -
    32 
    -# Use max-jobs CPUs for building and enable verbosity
    33 
    -environment:
    34 
    -  MAKEFLAGS: -j%{max-jobs}
    35 
    -  V: 1
    36 
    -
    37 
    -# And dont consider MAKEFLAGS or V as something which may
    38 
    -# affect build output.
    39 
    -environment-nocache:
    40 
    -- MAKEFLAGS
    41 
    -- V

  • buildstream/plugins/sources/pip.py
    ... ... @@ -96,7 +96,7 @@ _PYTHON_VERSIONS = [
    96 96 
     # Names of source distribution archives must be of the form
    97 97 
     # '%{package-name}-%{version}.%{extension}'.
    98 98 
     _SDIST_RE = re.compile(
    99 
    -    r'^([a-zA-Z0-9]+?)-(.+).(?:tar|tar.bz2|tar.gz|tar.xz|tar.Z|zip)$',
    99 
    +    r'^([\w.-]+?)-((?:[\d.]+){2,})\.(?:tar|tar.bz2|tar.gz|tar.xz|tar.Z|zip)$',
    100 100 
         re.IGNORECASE)
    101 101
    102 102
    ... ... @@ -225,12 +225,27 @@ class PipSource(Source):
    225 225 
         def _parse_sdist_names(self, basedir):
    226 226 
             reqs = []
    227 227 
             for f in os.listdir(basedir):
    228 
    -            pkg_match = _SDIST_RE.match(f)
    229 
    -            if pkg_match:
    230 
    -                reqs.append(pkg_match.groups())
    228 
    +            pkg = _match_package_name(f)
    229 
    +            if pkg is not None:
    230 
    +                reqs.append(pkg)
    231 231
    232 232 
             return sorted(reqs)
    233 233
    234 234
    235 
    +# Extract the package name and version of a source distribution
    236 
    +#
    237 
    +# Args:
    238 
    +#    filename (str): Filename of the source distribution
    239 
    +#
    240 
    +# Returns:
    241 
    +#    (tuple): A tuple of (package_name, version)
    242 
    +#
    243 
    +def _match_package_name(filename):
    244 
    +    pkg_match = _SDIST_RE.match(filename)
    245 
    +    if pkg_match is None:
    246 
    +        return None
    247 
    +    return pkg_match.groups()
    248 
    +
    249 
    +
    235 250 
     def setup():
    236 251 
         return PipSource

  • setup.py
    ... ... @@ -39,6 +39,7 @@ if sys.version_info[0] != REQUIRED_PYTHON_MAJOR or sys.version_info[1] < REQUIRE
    39 39 
     try:
    40 40 
         from setuptools import setup, find_packages, Command
    41 41 
         from setuptools.command.easy_install import ScriptWriter
    42 
    +    from setuptools.command.test import test as TestCommand
    42 43 
     except ImportError:
    43 44 
         print("BuildStream requires setuptools in order to build. Install it using"
    44 45 
               " your package manager (usually python3-setuptools) or via pip (pip3"
    ... ... @@ -219,9 +220,48 @@ class BuildGRPC(Command):
    219 220 
                             f.write(code)
    220 221
    221 222
    223 
    +#####################################################
    224 
    +#                   Pytest command                  #
    225 
    +#####################################################
    226 
    +class PyTest(TestCommand):
    227 
    +    """Defines a pytest command class to run tests from setup.py"""
    228 
    +
    229 
    +    user_options = TestCommand.user_options + [
    230 
    +        ("addopts=", None, "Arguments to pass to pytest"),
    231 
    +        ('index-url=''build_grpc': BuildGRPC,
    264 
    +        'pytest': PyTest,
    225 265 
         }
    226 266 
         cmdclass.update(versioneer.get_cmdclass())
    227 267 
         return cmdclass
    ... ... @@ -305,6 +345,5 @@ setup(name='BuildStream',
    305 345 
               'grpcio >= 1.10',
    306 346 
           ],
    307 347 
           entry_points=bst_install_entry_points,
    308 
    -      setup_requires=['pytest-runner'],
    309 348 
           tests_require=dev_requires,
    310 349 
           zip_safe=False)

  • tests/completions/completions.py
    ... ... @@ -42,6 +42,7 @@ MAIN_OPTIONS = [
    42 42 
         "-o ",
    43 43 
         "--option ",
    44 44 
         "--on-error ",
    45 
    +    "--pull-build-trees ",
    45 46 
         "--pushers ",
    46 47 
         "--strict ",
    47 48 
         "--verbose ",

  • tests/frontend/init.py
    ... ... @@ -3,6 +3,7 @@ import pytest
    3 3 
     from tests.testutils import cli
    4 4
    5 5 
     from buildstream import _yaml
    6 
    +from buildstream._frontend.app import App
    6 7 
     from buildstream._exceptions import ErrorDomain, LoadErrorReason
    7 8 
     from buildstream._versions import BST_FORMAT_VERSION
    8 9
    ... ... @@ -98,3 +99,34 @@ def test_bad_element_path(cli, tmpdir, element_path):
    98 99 
             'init', '--project-name', 'foo', '--element-path', element_path
    99 100 
         ])
    100 101 
         result.assert_main_error(ErrorDomain.APP, 'invalid-element-path')
    102 
    +
    103 
    +
    104 
    +@pytest.mark.parametrize("element_path", [('foo'), ('foo/bar')])
    105 
    +def test_element_path_interactive(cli, tmp_path, monkeypatch, element_path):
    106 
    +    project = tmp_path
    107 
    +    project_conf_path = project.joinpath('project.conf')
    108 
    +
    109 
    +    class DummyInteractiveApp(App):
    110 
    +        def __init__(self, *args, **kwargs):
    111 
    +            super().__init__(*args, **kwargs)
    112 
    +            self.interactive = True
    113 
    +
    114 
    +        @classmethod
    115 
    +        def create(cls, *args, **kwargs):
    116 
    +            return DummyInteractiveApp(*args, **kwargs)
    117 
    +
    118 
    +        def _init_project_interactive(self, *args, **kwargs):
    119 
    +            return ('project_name', '0', element_path)
    120 
    +
    121 
    +    monkeypatch.setattr(App, 'create', DummyInteractiveApp.create)
    122 
    +
    123 
    +    result = cli.run(project=str(project), args=['init'])
    124 
    +    result.assert_success()
    125 
    +
    126 
    +    full_element_path = project.joinpath(element_path)
    127 
    +    assert full_element_path.exists()
    128 
    +
    129 
    +    project_conf = _yaml.load(str(project_conf_path))
    130 
    +    assert project_conf['name'] == 'project_name'
    131 
    +    assert project_conf['format-version'] == '0'
    132 
    +    assert project_conf['element-path'] == element_path

  • tests/integration/build-tree.py
    ... ... @@ -70,8 +70,8 @@ def test_buildtree_pulled(cli, tmpdir, datafiles):
    70 70 
             })
    71 71 
             assert cli.get_element_state(project, element_name) != 'cached'
    72 72
    73 
    -        # Pull from cache
    74 
    -        result = cli.run(project=project, args=['pull', '--deps', 'all', element_name])
    73 
    +        # Pull from cache, ensuring cli options is set to pull the buildtree
    74 
    +        result = cli.run(project=project, args=['--pull-build-trees', 'pull', '--deps', 'all', element_name])
    75 75 
             result.assert_success()
    76 76
    77 77 
             # Check it's using the cached build tree

  • tests/integration/pip_source.py
    ... ... @@ -4,6 +4,7 @@ import pytest
    4 4 
     from buildstream import _yaml
    5 5
    6 6 
     from tests.testutils import cli_integration as cli
    7 
    +from tests.testutils.python_repo import setup_pypi_repo
    7 8 
     from tests.testutils.integration import assert_contains
    8 9
    9 10
    ... ... @@ -17,12 +18,21 @@ DATA_DIR = os.path.join(
    17 18
    18 19
    19 20 
     @pytest.mark.datafiles(DATA_DIR)
    20 
    -def test_pip_source_import(cli, tmpdir, datafiles):
    21 
    +def test_pip_source_import(cli, tmpdir, datafiles, setup_pypi_repo):
    21 22 
         project = os.path.join(datafiles.dirname, datafiles.basename)
    22 23 
         checkout = os.path.join(cli.directory, 'checkout')
    23 24 
         element_path = os.path.join(project, 'elements')
    24 25 
         element_name = 'pip/hello.bst'
    25 26
    27 
    +    # check that exotically named packages are imported correctly
    28 
    +    myreqs_packages = ['hellolib']
    29 
    +    packages = ['app2', 'app.3', 'app-4', 'app_5', 'app.no.6', 'app-no-7', 'app_no_8']
    30 
    +
    31 
    +    # create mock pypi repository
    32 
    +    pypi_repo = os.path.join(project, 'files', 'pypi-repo')
    33 
    +    os.makedirs(pypi_repo, exist_ok=True)
    34 
    +    setup_pypi_repo(myreqs_packages + packages, pypi_repo)
    35 
    +
    26 36 
         element = {
    27 37 
             'kind': 'import',
    28 38 
             'sources': [
    ... ... @@ -32,9 +42,9 @@ def test_pip_source_import(cli, tmpdir, datafiles):
    32 42 
                 },
    33 43 
                 {
    34 44 
                     'kind': 'pip',
    35 
    -                'url': 'file://{}'.format(os.path.realpath(os.path.join(project, 'files', 'pypi-repo'))),
    45 
    +                'url': 'file://{}'.format(os.path.realpath(pypi_repo)),
    36 46 
                     'requirements-files': ['myreqs.txt'],
    37 
    -                'packages': ['app2']
    47 
    +                'packages': packages
    38 48 
                 }
    39 49 
             ]
    40 50 
         }
    ... ... @@ -51,16 +61,31 @@ def test_pip_source_import(cli, tmpdir, datafiles):
    51 61 
         assert result.exit_code == 0
    52 62
    53 63 
         assert_contains(checkout, ['/.bst_pip_downloads',
    54 
    -                               '/.bst_pip_downloads/HelloLib-0.1.tar.gz',
    55 
    -                               '/.bst_pip_downloads/App2-0.1.tar.gz'])
    64 
    +                               '/.bst_pip_downloads/hellolib-0.1.tar.gz',
    65 
    +                               '/.bst_pip_downloads/app2-0.1.tar.gz',
    66 
    +                               '/.bst_pip_downloads/app.3-0.1.tar.gz',
    67 
    +                               '/.bst_pip_downloads/app-4-0.1.tar.gz',
    68 
    +                               '/.bst_pip_downloads/app_5-0.1.tar.gz',
    69 
    +                               '/.bst_pip_downloads/app.no.6-0.1.tar.gz',
    70 
    +                               '/.bst_pip_downloads/app-no-7-0.1.tar.gz',
    71 
    +                               '/.bst_pip_downloads/app_no_8-0.1.tar.gz'])
    56 72
    57 73
    58 74 
     @pytest.mark.datafiles(DATA_DIR)
    59 
    -def test_pip_source_build(cli, tmpdir, datafiles):
    75 
    +def test_pip_source_build(cli, tmpdir, datafiles, setup_pypi_repo):
    60 76 
         project = os.path.join(datafiles.dirname, datafiles.basename)
    61 77 
         element_path = os.path.join(project, 'elements')
    62 78 
         element_name = 'pip/hello.bst'
    63 79
    80 
    +    # check that exotically named packages are imported correctly
    81 
    +    myreqs_packages = ['hellolib']
    82 
    +    packages = ['app2', 'app.3', 'app-4', 'app_5', 'app.no.6', 'app-no-7', 'app_no_8']
    83 
    +
    84 
    +    # create mock pypi repository
    85 
    +    pypi_repo = os.path.join(project, 'files', 'pypi-repo')
    86 
    +    os.makedirs(pypi_repo, exist_ok=True)
    87 
    +    setup_pypi_repo(myreqs_packages + packages, pypi_repo)
    88 
    +
    64 89 
         element = {
    65 90 
             'kind': 'manual',
    66 91 
             'depends': ['base.bst'],
    ... ... @@ -71,16 +96,15 @@ def test_pip_source_build(cli, tmpdir, datafiles):
    71 96 
                 },
    72 97 
                 {
    73 98 
                     'kind': 'pip',
    74 
    -                'url': 'file://{}'.format(os.path.realpath(os.path.join(project, 'files', 'pypi-repo'))),
    99 
    +                'url': 'file://{}'.format(os.path.realpath(pypi_repo)),
    75 100 
                     'requirements-files': ['myreqs.txt'],
    76 
    -                'packages': ['app2']
    101 
    +                'packages': packages
    77 102 
                 }
    78 103 
             ],
    79 104 
             'config': {
    80 105 
                 'install-commands': [
    81 106 
                     'pip3 install --no-index --prefix %{install-root}/usr .bst_pip_downloads/*.tar.gz',
    82 
    -                'chmod +x app1.py',
    83 
    -                'install app1.py  %{install-root}/usr/bin/'
    107 
    +                'install app1.py %{install-root}/usr/bin/'
    84 108 
                 ]
    85 109 
             }
    86 110 
         }
    ... ... @@ -95,5 +119,4 @@ def test_pip_source_build(cli, tmpdir, datafiles):
    95 119
    96 120 
         result = cli.run(project=project, args=['shell', element_name, '/usr/bin/app1.py'])
    97 121 
         assert result.exit_code == 0
    98 
    -    assert result.output == """Hello App1!
    99 
    -"""
    122 
    +    assert result.output == "Hello App1! This is hellolib\n"

  • tests/integration/project/files/pypi-repo/app2/App2-0.1.tar.gz deleted
    No preview for this file type
  • tests/integration/project/files/pypi-repo/app2/index.html deleted
    1 
    -<html>
    2 
    -  <head>
    3 
    -    <title>Links for app1</title>
    4 
    -  </head>
    5 
    -  <body>
    6 
    -    <a href="">'App2-0.1.tar.gz'>App2-0.1.tar.gz</a><br />
    7 
    -  </body>
    8 
    -</html>

  • tests/integration/project/files/pypi-repo/hellolib/HelloLib-0.1.tar.gz deleted
    No preview for this file type
  • tests/integration/project/files/pypi-repo/hellolib/index.html deleted
    1 
    -<html>
    2 
    -  <head>
    3 
    -    <title>Links for app1</title>
    4 
    -  </head>
    5 
    -  <body>
    6 
    -    <a href="">'HelloLib-0.1.tar.gz'>HelloLib-0.1.tar.gz</a><br />
    7 
    -  </body>
    8 
    -</html>

  • tests/integration/pullbuildtrees.py
    1 
    +import os
    2 
    +import shutil
    3 
    +import pytest
    4 
    +
    5 
    +from tests.testutils import cli_integration as cli, create_artifact_share
    6 
    +from tests.testutils.integration import assert_contains
    7 
    +
    8 
    +
    9 
    +DATA_DIR = os.path.join(
    10 
    +    os.path.dirname(os.path.realpath(__file__)),
    11 
    +    "project"
    12 
    +)
    13 
    +
    14 
    +
    15 
    +# Remove artifact cache & set cli.config value of pullbuildtrees
    16 
    +# to false, which is the default user context. The cache has to be
    17 
    +# cleared as just forcefully removing the refpath leaves dangling objects.
    18 
    +def default_state(cli, tmpdir, share):
    19 
    +    shutil.rmtree(os.path.join(str(tmpdir), 'artifacts'))
    20 
    +    cli.configure({
    21 
    +        'pullbuildtrees': False,
    22 
    +        'artifacts': {'url': share.repo, 'push': False},
    23 
    +        'artifactdir': os.path.join(str(tmpdir), 'artifacts')
    24 
    +    })
    25 
    +
    26 
    +
    27 
    +# A test to capture the integration of the pullbuildtrees
    28 
    +# behaviour, which by default is to not include the buildtree
    29 
    +# directory of an element.
    30 
    +@pytest.mark.integration
    31 
    +@pytest.mark.datafiles(DATA_DIR)
    32 
    +def test_pullbuildtrees(cli, tmpdir, datafiles, integration_cache):
    33 
    +
    34 
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    35 
    +    element_name = 'autotools/amhello.bst'
    36 
    +
    37 
    +    # Create artifact shares for pull & push testing
    38 
    +    with create_artifact_share(os.path.join(str(tmpdir), 'share1')) as share1,\
    39 
    +        create_artifact_share(os.path.join(str(tmpdir), 'share2')) as share2:
    40 
    +        cli.configure({
    41 
    +            'artifacts': {'url': share1.repo, 'push': True},
    42 
    +            'artifactdir': os.path.join(str(tmpdir), 'artifacts')
    43 
    +        })
    44 
    +
    45 
    +        # Build autotools element, checked pushed, delete local
    46 
    +        result = cli.run(project=project, args=['build', element_name])
    47 
    +        assert result.exit_code == 0
    48 
    +        assert cli.get_element_state(project, element_name) == 'cached'
    49 
    +        assert share1.has_artifact('test', element_name, cli.get_element_key(project, element_name))
    50 
    +        default_state(cli, tmpdir, share1)
    51 
    +
    52 
    +        # Pull artifact with default config, assert that pulling again
    53 
    +        # doesn't create a pull job, then assert with buildtrees user
    54 
    +        # config set creates a pull job.
    55 
    +        result = cli.run(project=project, args=['pull', element_name])
    56 
    +        assert element_name in result.get_pulled_elements()
    57 
    +        result = cli.run(project=project, args=['pull', element_name])
    58 
    +        assert element_name not in result.get_pulled_elements()
    59 
    +        cli.configure({'pullbuildtrees': True})
    60 
    +        result = cli.run(project=project, args=['pull', element_name])
    61 
    +        assert element_name in result.get_pulled_elements()
    62 
    +        default_state(cli, tmpdir, share1)
    63 
    +
    64 
    +        # Pull artifact with default config, then assert that pulling
    65 
    +        # with buildtrees cli flag set creates a pull job.
    66 
    +        # Also assert that the buildtree is added to the artifact's
    67 
    +        # extract dir
    68 
    +        result = cli.run(project=project, args=['pull', element_name])
    69 
    +        assert element_name in result.get_pulled_elements()
    70 
    +        elementdigest = share1.has_artifact('test', element_name, cli.get_element_key(project, element_name))
    71 
    +        buildtreedir = os.path.join(str(tmpdir), 'artifacts', 'extract', 'test', 'autotools-amhello',
    72 
    +                                    elementdigest.hash, 'buildtree')
    73 
    +        assert not os.path.isdir(buildtreedir)
    74 
    +        result = cli.run(project=project, args=['--pull-build-trees', 'pull', element_name])
    75 
    +        assert element_name in result.get_pulled_elements()
    76 
    +        assert os.path.isdir(buildtreedir)
    77 
    +        default_state(cli, tmpdir, share1)
    78 
    +
    79 
    +        # Pull artifact with pullbuildtrees set in user config, then assert
    80 
    +        # that pulling with the same user config doesn't creates a pull job,
    81 
    +        # or when buildtrees cli flag is set.
    82 
    +        cli.configure({'pullbuildtrees': True})
    83 
    +        result = cli.run(project=project, args=['pull', element_name])
    84 
    +        assert element_name in result.get_pulled_elements()
    85 
    +        result = cli.run(project=project, args=['pull', element_name])
    86 
    +        assert element_name not in result.get_pulled_elements()
    87 
    +        result = cli.run(project=project, args=['--pull-build-trees', 'pull', element_name])
    88 
    +        assert element_name not in result.get_pulled_elements()
    89 
    +        default_state(cli, tmpdir, share1)
    90 
    +
    91 
    +        # Pull artifact with default config and buildtrees cli flag set, then assert
    92 
    +        # that pulling with pullbuildtrees set in user config doesn't create a pull
    93 
    +        # job.
    94 
    +        result = cli.run(project=project, args=['--pull-build-trees', 'pull', element_name])
    95 
    +        assert element_name in result.get_pulled_elements()
    96 
    +        cli.configure({'pullbuildtrees': True})
    97 
    +        result = cli.run(project=project, args=['pull', element_name])
    98 
    +        assert element_name not in result.get_pulled_elements()
    99 
    +        default_state(cli, tmpdir, share1)
    100 
    +
    101 
    +        # Assert that a partial build element (not containing a populated buildtree dir)
    102 
    +        # can't be pushed to an artifact share, then assert that a complete build element
    103 
    +        # can be. This will attempt a partial pull from share1 and then a partial push
    104 
    +        # to share2
    105 
    +        result = cli.run(project=project, args=['pull', element_name])
    106 
    +        assert element_name in result.get_pulled_elements()
    107 
    +        cli.configure({'artifacts': {'url': share2.repo, 'push': True}})
    108 
    +        result = cli.run(project=project, args=['push', element_name])
    109 
    +        assert element_name not in result.get_pushed_elements()
    110 
    +        assert not share2.has_artifact('test', element_name, cli.get_element_key(project, element_name))
    111 
    +
    112 
    +        # Assert that after pulling the missing buildtree the element artifact can be
    113 
    +        # successfully pushed to the remote. This will attempt to pull the buildtree
    114 
    +        # from share1 and then a 'complete' push to share2
    115 
    +        cli.configure({'artifacts': {'url': share1.repo, 'push': False}})
    116 
    +        result = cli.run(project=project, args=['--pull-build-trees', 'pull', element_name])
    117 
    +        assert element_name in result.get_pulled_elements()
    118 
    +        cli.configure({'artifacts': {'url': share2.repo, 'push': True}})
    119 
    +        result = cli.run(project=project, args=['push', element_name])
    120 
    +        assert element_name in result.get_pushed_elements()
    121 
    +        assert share2.has_artifact('test', element_name, cli.get_element_key(project, element_name))
    122 
    +        default_state(cli, tmpdir, share1)

  • tests/sources/pip.py
    ... ... @@ -3,6 +3,7 @@ import pytest
    3 3
    4 4 
     from buildstream._exceptions import ErrorDomain
    5 5 
     from buildstream import _yaml
    6 
    +from buildstream.plugins.sources.pip import _match_package_name
    6 7 
     from tests.testutils import cli
    7 8
    8 9 
     DATA_DIR = os.path.join(
    ... ... @@ -45,3 +46,22 @@ def test_no_packages(cli, tmpdir, datafiles):
    45 46 
             'show', 'target.bst'
    46 47 
         ])
    47 48 
         result.assert_main_error(ErrorDomain.SOURCE, None)
    49 
    +
    50 
    +
    51 
    +# Test that pip source parses tar ball names correctly for the ref
    52 
    +@pytest.mark.parametrize(
    53 
    +    'tarball, expected_name, expected_version',
    54 
    +    [
    55 
    +        ('dotted.package-0.9.8.tar.gz', 'dotted.package', '0.9.8'),
    56 
    +        ('hyphenated-package-2.6.0.tar.gz', 'hyphenated-package', '2.6.0'),
    57 
    +        ('underscore_pkg-3.1.0.tar.gz', 'underscore_pkg', '3.1.0'),
    58 
    +        ('numbers2and5-1.0.1.tar.gz', 'numbers2and5', '1.0.1'),
    59 
    +        ('multiple.dots.package-5.6.7.tar.gz', 'multiple.dots.package', '5.6.7'),
    60 
    +        ('multiple-hyphens-package-1.2.3.tar.gz', 'multiple-hyphens-package', '1.2.3'),
    61 
    +        ('multiple_underscore_pkg-3.4.5.tar.gz', 'multiple_underscore_pkg', '3.4.5'),
    62 
    +        ('shortversion-1.0.tar.gz', 'shortversion', '1.0'),
    63 
    +        ('longversion-1.2.3.4.tar.gz', 'longversion', '1.2.3.4')
    64 
    +    ])
    65 
    +def test_match_package_name(tarball, expected_name, expected_version):
    66 
    +    name, version = _match_package_name(tarball)
    67 
    +    assert (expected_name, expected_version) == (name, version)

  • tests/testutils/__init__.py
    ... ... @@ -29,3 +29,4 @@ from .artifactshare import create_artifact_share
    29 29 
     from .element_generators import create_element_size, update_element_size
    30 30 
     from .junction import generate_junction
    31 31 
     from .runner_integration import wait_for_cache_granularity
    32 
    +from .python_repo import setup_pypi_repo

  • tests/testutils/artifactshare.py
    ... ... @@ -114,7 +114,7 @@ class ArtifactShare():
    114 114 
         #    cache_key (str): The cache key
    115 115 
         #
    116 116 
         # Returns:
    117 
    -    #    (bool): True if the artifact exists in the share, otherwise false.
    117 
    +    #    (str): artifact digest if the artifact exists in the share, otherwise None.
    118 118 
         def has_artifact(self, project_name, element_name, cache_key):
    119 119
    120 120 
             # NOTE: This should be kept in line with our
    ... ... @@ -134,9 +134,9 @@ class ArtifactShare():
    134 134
    135 135 
             try:
    136 136 
                 tree = self.cas.resolve_ref(artifact_key)
    137 
    -            return True
    137 
    +            return tree
    138 138 
             except ArtifactError:
    139 
    -            return False
    139 
    +            return None
    140 140
    141 141 
         # close():
    142 142 
         #

  • tests/testutils/python_repo.py
    1 
    +from setuptools.sandbox import run_setup
    2 
    +import os
    3 
    +import pytest
    4 
    +import re
    5 
    +import shutil
    6 
    +
    7 
    +
    8 
    +SETUP_TEMPLATE = '''\
    9 
    +from setuptools import setup
    10 
    +
    11 
    +setup(
    12 
    +    name='{name}',
    13 
    +    version='{version}',
    14 
    +    description='{name}',
    15 
    +    packages=['{pkgdirname}'],
    16 
    +    entry_points={{
    17 
    +        'console_scripts': [
    18 
    +            '{pkgdirname}={pkgdirname}:main'
    19 
    +        ]
    20 
    +    }}
    21 
    +)
    22 
    +'''
    23 
    +
    24 
    +# All packages generated via generate_pip_package will have the functions below
    25 
    +INIT_TEMPLATE = '''\
    26 
    +def main():
    27 
    +    print('This is {name}')
    28 
    +
    29 
    +def hello(actor='world'):
    30 
    +    print('Hello {{}}! This is {name}'.format(actor))
    31 
    +'''
    32 
    +
    33 
    +HTML_TEMPLATE = '''\
    34 
    +<html>
    35 
    +  <head>
    36 
    +    <title>Links for {name}</title>
    37 
    +  </head>
    38 
    +  <body>
    39 
    +    <a href=''>{name}-{version}.tar.gz</a><br />
    40 
    +  </body>
    41 
    +</html>
    42 
    +'''
    43 
    +
    44 
    +
    45 
    +# Creates a simple python source distribution and copies this into a specified
    46 
    +# directory which is to serve as a mock python repository
    47 
    +#
    48 
    +# Args:
    49 
    +#    tmpdir (str): Directory in which the source files will be created
    50 
    +#    pypi (str): Directory serving as a mock python repository
    51 
    +#    name (str): The name of the package to be created
    52 
    +#    version (str): The version of the package to be created
    53 
    +#
    54 
    +# Returns:
    55 
    +#    None
    56 
    +#
    57 
    +def generate_pip_package(tmpdir, pypi, name, version='0.1'):
    58 
    +    # check if package already exists in pypi
    59 
    +    pypi_package = os.path.join(pypi, re.sub('[^0-9a-zA-Z]+', '-', name))
    60 
    +    if os.path.exists(pypi_package):
    61 
    +        return
    62 
    +
    63 
    +    # create the package source files in tmpdir resulting in a directory
    64 
    +    # tree resembling the following structure:
    65 
    +    #
    66 
    +    # tmpdir
    67 
    +    # |-- setup.py
    68 
    +    # `-- package
    69 
    +    #     `-- __init__.py
    70 
    +    #
    71 
    +    setup_file = os.path.join(tmpdir, 'setup.py')
    72 
    +    pkgdirname = re.sub('[^0-9a-zA-Z]+', '', name)
    73 
    +    with open(setup_file, 'w') as f:
    74 
    +        f.write(
    75 
    +            SETUP_TEMPLATE.format(
    76 
    +                name=name,
    77 
    +                version=version,
    78 
    +                pkgdirname=pkgdirname
    79 
    +            )
    80 
    +        )
    81 
    +    os.chmod(setup_file, 0o755)
    82 
    +
    83 
    +    package = os.path.join(tmpdir, pkgdirname)
    84 
    +    os.makedirs(package)
    85 
    +
    86 
    +    main_file = os.path.join(package, '__init__.py')
    87 
    +    with open(main_file, 'w') as f:
    88 
    +        f.write(INIT_TEMPLATE.format(name=name))
    89 
    +    os.chmod(main_file, 0o644)
    90 
    +
    91 
    +    run_setup(setup_file, ['sdist'])
    92 
    +
    93 
    +    # create directory for this package in pypi resulting in a directory
    94 
    +    # tree resembling the following structure:
    95 
    +    #
    96 
    +    # pypi
    97 
    +    # `-- pypi_package
    98 
    +    #     |-- index.html
    99 
    +    #     `-- foo-0.1.tar.gz
    100 
    +    #
    101 
    +    os.makedirs(pypi_package)
    102 
    +
    103 
    +    # add an index html page
    104 
    +    index_html = os.path.join(pypi_package, 'index.html')
    105 
    +    with open(index_html, 'w') as f:
    106 
    +        f.write(HTML_TEMPLATE.format(name=name, version=version))
    107 
    +
    108 
    +    # copy generated tarfile to pypi package
    109 
    +    dist_dir = os.path.join(tmpdir, 'dist')
    110 
    +    for tar in os.listdir(dist_dir):
    111 
    +        tarpath = os.path.join(dist_dir, tar)
    112 
    +        shutil.copy(tarpath, pypi_package)
    113 
    +
    114 
    +
    115 
    +@pytest.fixture
    116 
    +def setup_pypi_repo(tmpdir):
    117 
    +    def create_pkgdir(package):
    118 
    +        pkgdirname = re.sub('[^0-9a-zA-Z]+', '', package)
    119 
    +        pkgdir = os.path.join(str(tmpdir), pkgdirname)
    120 
    +        os.makedirs(pkgdir)
    121 
    +        return pkgdir
    122 
    +
    123 
    +    def add_packages(packages, pypi_repo):
    124 
    +        for package in packages:
    125 
    +            pkgdir = create_pkgdir(package)
    126 
    +            generate_pip_package(pkgdir, pypi_repo, package)
    127 
    +
    128 
    +    return add_packages

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.

    [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]