[Notes] [Git][BuildStream/buildstream][jonathan/source-mirror-project-re

Jonathan Maw pushed to branch jonathan/source-mirror-project-refs at BuildStream / buildstream

Commits:

d5b396e0

by Phillip Smyth at 2018-10-02T16:40:18Z

Forcing CI to install Sphinx 1.7.9

.gitlab-ci.yml: forced pip3 install Sphinx 1.7.9
The theme we use in our docs "sphinx_rtd_theme"
Does not support the latest version of Sphinx (>1.8)
This results in search functionality failing

dae842fd

by Phillip Smyth at 2018-10-02T17:14:09Z

Merge branch 'docs_Search_not_working' into 'master'

Fixing: Search functionality in the Documentation has stopped working

Closes #681

See merge request BuildStream/buildstream!848

8dc16d3f

by Jürg Billeter at 2018-10-03T05:08:21Z

_platform/linux.py: Accept all configs for dummy sandbox

If Sandbox.run() is never called, sandbox config is irrelevant. If it is
called, the build will fail anyway with the dummy sandbox.

This matches the logic in the Darwin platform.

66446fc3

by Jürg Billeter at 2018-10-03T05:36:38Z

Merge branch 'juerg/dummy-sandbox' into 'master'

_platform/linux.py: Accept all configs for dummy sandbox

See merge request BuildStream/buildstream!843

29c19bea

by Tristan Van Berkom at 2018-10-03T07:33:48Z

git.py: Added timed activity around fetching, displaying the URL being fetched.

This will take care of silencing the status messages while checking submodules.

b645881c

by Tristan Van Berkom at 2018-10-03T07:33:48Z

source.py: Silence messages while consuming source fetchers

The source fetchers might be a list or a generator, when it is
a generator (like the git source does), then we want to ensure that
we silence the status messages which might occur as a result of
consuming a source fetcher from the generator.

This fixes the logs to be less verbose.

c9437616

by Tristan Van Berkom at 2018-10-03T08:07:15Z

Merge branch 'tristan/fix-status-messages' into 'master'

fix status messages

See merge request BuildStream/buildstream!845

11320fe2

by Tristan Van Berkom at 2018-10-03T09:33:39Z

_scheduler: Fix bookkeeping of terminated jobs

 * Enhanced the base Job class to bookkeep which jobs have
   been terminated, and to consider them as `skipped` when
   asked via the `skipped` property.

 * Enhanced the base Queue class to bookkeep the job statuses
   more carefully.

This fixes #479

91271964

by Tristan Van Berkom at 2018-10-03T09:59:40Z

Merge branch 'tristan/fix-terminated-jobs' into 'master'

_scheduler: Fix bookkeeping of terminated jobs

Closes #479

See merge request BuildStream/buildstream!850

3bf895d2

by Jonathan Maw at 2018-10-03T11:48:48Z

tests: Add regression test for mirroring with project.refs

Changes:

.gitlab-ci.yml

@@ -145,7 +145,8 @@ docs:
    stage: test
    script:
    - export BST_SOURCE_CACHE="$(pwd)/cache/integration-cache/sources"
 -  - pip3 install sphinx
 +  # Currently sphinx_rtd_theme does not support Sphinx >1.8, this breaks search functionality
 +  - pip3 install sphinx==1.7.9
    - pip3 install sphinx-click
    - pip3 install sphinx_rtd_theme
    - cd dist && ./unpack.sh && cd buildstream

buildstream/_platform/linux.py

@@ -55,6 +55,10 @@ class Linux(Platform):
              return SandboxBwrap(*args, **kwargs)
      def check_sandbox_config(self, config):
 +        if not self._local_sandbox_available():
 +            # Accept all sandbox configs as it's irrelevant with the dummy sandbox (no Sandbox.run).
 +            return True
++
          if self._user_ns_available:
              # User namespace support allows arbitrary build UID/GID settings.
              return True

buildstream/_scheduler/jobs/job.py

@@ -119,6 +119,8 @@ class Job():
          self._result = None                    # Return value of child action in the parent
          self._tries = 0                        # Try count, for retryable jobs
          self._skipped_flag = False             # Indicate whether the job was skipped.
 +        self._terminated = False               # Whether this job has been explicitly terminated
++
          # If False, a retry will not be attempted regardless of whether _tries is less than _max_retries.
+         #
          self._retry_flag = True
@@ -190,6 +192,8 @@ class Job():
          # Terminate the process using multiprocessing API pathway
          self._process.terminate()
 +        self._terminated = True
++
      # terminate_wait()
+     #
      # Wait for terminated jobs to complete
@@ -273,18 +277,22 @@ class Job():
      # running the integration commands).
+     #
      # Args:
 -    #     (int): The plugin identifier for this task
 +    #     task_id (int): The plugin identifier for this task
+     #
      def set_task_id(self, task_id):
          self._task_id = task_id
      # skipped
+     #
 +    # This will evaluate to True if the job was skipped
 +    # during processing, or if it was forcefully terminated.
 +    #
      # Returns:
 -    #    bool: True if the job was skipped while processing.
 +    #    (bool): Whether the job should appear as skipped
 +    #
      @property
      def skipped(self):
 -        return self._skipped_flag
 +        return self._skipped_flag or self._terminated
      #######################################################
      #                  Abstract Methods                   #

buildstream/_scheduler/queues/queue.py

@@ -326,16 +326,20 @@ class Queue():
                            detail=traceback.format_exc())
              self.failed_elements.append(element)
          else:
+-
 -            # No exception occured, handle the success/failure state in the normal way
+             #
 +            # No exception occured in post processing
 +            #
++
 +            # All jobs get placed on the done queue for later processing.
              self._done_queue.append(job)
 -            if success:
 -                if not job.skipped:
 -                    self.processed_elements.append(element)
 -                else:
 -                    self.skipped_elements.append(element)
 +            # A Job can be skipped whether or not it has failed,
 +            # we want to only bookkeep them as processed or failed
 +            # if they are not skipped.
 +            if job.skipped:
 +                self.skipped_elements.append(element)
 +            elif success:
 +                self.processed_elements.append(element)
              else:
                  self.failed_elements.append(element)

buildstream/plugins/sources/git.py

@@ -184,10 +184,18 @@ class GitMirror(SourceFetcher):
                           cwd=self.mirror)
      def fetch(self, alias_override=None):
 -        self.ensure(alias_override)
 -        if not self.has_ref():
 -            self._fetch(alias_override)
 -        self.assert_ref()
 +        # Resolve the URL for the message
 +        resolved_url = self.source.translate_url(self.url,
 +                                                 alias_override=alias_override,
 +                                                 primary=self.primary)
++
 +        with self.source.timed_activity("Fetching from {}"
 +                                        .format(resolved_url),
 +                                        silent_nested=True):
 +            self.ensure(alias_override)
 +            if not self.has_ref():
 +                self._fetch(alias_override)
 +            self.assert_ref()
      def has_ref(self):
          if not self.ref:

buildstream/source.py

@@ -965,28 +965,48 @@ class Source(Plugin):
      # Tries to call fetch for every mirror, stopping once it succeeds
      def __do_fetch(self, **kwargs):
          project = self._get_project()
 -        source_fetchers = self.get_source_fetchers()
 +        context = self._get_context()
++
 +        # Silence the STATUS messages which might happen as a result
 +        # of checking the source fetchers.
 +        with context.silence():
 +            source_fetchers = self.get_source_fetchers()
          # Use the source fetchers if they are provided
+         #
          if source_fetchers:
 -            for fetcher in source_fetchers:
 -                alias = fetcher._get_alias()
 -                for uri in project.get_alias_uris(alias, first_pass=self.__first_pass):
 -                    try:
 -                        fetcher.fetch(uri)
 -                    # FIXME: Need to consider temporary vs. permanent failures,
 -                    #        and how this works with retries.
 -                    except BstError as e:
 -                        last_error = e
 -                        continue
+-
 -                    # No error, we're done with this fetcher
 -                    break
 -                else:
 -                    # No break occurred, raise the last detected error
 -                    raise last_error
 +            # Use a contorted loop here, this is to allow us to
 +            # silence the messages which can result from consuming
 +            # the items of source_fetchers, if it happens to be a generator.
 +            #
 +            source_fetchers = iter(source_fetchers)
 +            try:
++
 +                while True:
++
 +                    with context.silence():
 +                        fetcher = next(source_fetchers)
++
 +                    alias = fetcher._get_alias()
 +                    for uri in project.get_alias_uris(alias, first_pass=self.__first_pass):
 +                        try:
 +                            fetcher.fetch(uri)
 +                        # FIXME: Need to consider temporary vs. permanent failures,
 +                        #        and how this works with retries.
 +                        except BstError as e:
 +                            last_error = e
 +                            continue
++
 +                        # No error, we're done with this fetcher
 +                        break
++
 +                    else:
 +                        # No break occurred, raise the last detected error
 +                        raise last_error
++
 +            except StopIteration:
 +                pass
          # Default codepath is to reinstantiate the Source
+         #

tests/frontend/mirror.py

@@ -139,6 +139,82 @@ def test_mirror_fetch(cli, tmpdir, datafiles, kind):
      result.assert_success()
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("ref_storage", [("inline"), ("project.refs")])
 +@pytest.mark.parametrize("mirror", [("no-mirror"), ("mirror"), ("unrelated-mirror")])
 +def test_mirror_fetch_ref_storage(cli, tmpdir, datafiles, ref_storage, mirror):
 +    bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
 +    dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
 +    upstream_repodir = os.path.join(str(tmpdir), 'upstream')
 +    mirror_repodir = os.path.join(str(tmpdir), 'mirror')
 +    project_dir = os.path.join(str(tmpdir), 'project')
 +    os.makedirs(project_dir)
 +    element_dir = os.path.join(project_dir, 'elements')
++
 +    # Create repo objects of the upstream and mirror
 +    upstream_repo = create_repo('tar', upstream_repodir)
 +    upstream_ref = upstream_repo.create(bin_files_path)
 +    mirror_repo = upstream_repo.copy(mirror_repodir)
 +    mirror_ref = upstream_ref
 +    upstream_ref = upstream_repo.create(dev_files_path)
++
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            upstream_repo.source_config(ref=upstream_ref if ref_storage == 'inline' else None)
 +        ]
 +    }
 +    element_name = 'test.bst'
 +    element_path = os.path.join(element_dir, element_name)
 +    full_repo = element['sources'][0]['url']
 +    upstream_map, repo_name = os.path.split(full_repo)
 +    alias = 'foo'
 +    aliased_repo = alias + ':' + repo_name
 +    element['sources'][0]['url'] = aliased_repo
 +    full_mirror = mirror_repo.source_config()['url']
 +    mirror_map, _ = os.path.split(full_mirror)
 +    os.makedirs(element_dir)
 +    _yaml.dump(element, element_path)
++
 +    if ref_storage == 'project.refs':
 +        # Manually set project.refs to avoid caching the repo prematurely
 +        project_refs = {'projects': {
 +            'test': {
 +                element_name: [
 +                    {'ref': upstream_ref}
 +                ]
 +            }
 +        }}
 +        project_refs_path = os.path.join(project_dir, 'project.refs')
 +        _yaml.dump(project_refs, project_refs_path)
++
 +    project = {
 +        'name': 'test',
 +        'element-path': 'elements',
 +        'aliases': {
 +            alias: upstream_map + "/"
 +        },
 +        'ref-storage': ref_storage
 +    }
 +    if mirror != 'no-mirror':
 +        mirror_data = [{
 +            'name': 'middle-earth',
 +            'aliases': {alias: [mirror_map + '/']}
 +        }]
 +        if mirror == 'unrelated-mirror':
 +            mirror_data.insert(0, {
 +                'name': 'narnia',
 +                'aliases': {'frob': ['http://www.example.com/repo']}
 +            })
 +        project['mirrors'] = mirror_data
++
 +    project_file = os.path.join(project_dir, 'project.conf')
 +    _yaml.dump(project, project_file)
++
 +    result = cli.run(project=project_dir, args=['fetch', element_name])
 +    result.assert_success()
++
++
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
  def test_mirror_fetch_upstream_absent(cli, tmpdir, datafiles, kind):

[Notes] [Git][BuildStream/buildstream][jonathan/source-mirror-project-refs] 10 commits: Forcing CI to install Sphinx 1.7.9

Jonathan Maw pushed to branch jonathan/source-mirror-project-refs at BuildStream / buildstream

Commits:

7 changed files:

Changes: