[Notes] [Git][BuildStream/buildstream][bschubert/pipeline] 10 commits: d

Benjamin Schubert pushed to branch bschubert/pipeline at BuildStream / buildstream

Commits:

fc3dcec8

by Angelos Evripiotis at 2019-01-11T16:45:00Z

dev-requirements: pytest-cov==2.6.1 for '--no-cov'

Bump the version of pytest-cov, so that we pick up the fix for
the '--no-cov' option breaking in pytest 4.0:

    https://github.com/pytest-dev/pytest-cov/pull/230

This was the error you would get prior to this version:

    _pytest.warning_types.RemovedInPytest4Warning: config.warn has been
    deprecated, use warnings.warn instead

Although it says 'deprecated', it did result in
termination with stack trace.

f86b7ff3

by Angelos Evripiotis at 2019-01-11T16:45:00Z

contributing: '--last-failed' and '--no-cov' tips

To help new folks get to grips with pytest, add some tips on other
frequently used features.

d983f231

by Angelos Evripiotis at 2019-01-11T17:19:06Z

Merge branch 'aevri/tox_no_cov' into 'master'

pytest-cov==2.6.1, contributing: '--last-failed' and '--no-cov' tips

See merge request BuildStream/buildstream!1059

829a2b93

by Tristan Van Berkom at 2019-01-12T21:32:13Z

_scheduler/queues/queue.py: Put elements in the skip list, not jobs

Minor correction, looks like we're not observing this queue
otherwise we'd be seeing crashes.

993e30ae

by Tristan Van Berkom at 2019-01-12T22:02:55Z

Merge branch 'tristan/fix-skipped-elements' into 'master'

_scheduler/queues/queue.py: Put elements in the skip list, not jobs

See merge request BuildStream/buildstream!1063

32732e01

by Javier Jardón at 2019-01-14T09:04:01Z

.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25

4f5f1184

by Valentin David at 2019-01-14T09:40:56Z

Merge branch 'jjardon/freedesktop-sdk_latest' into 'master'

.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25

Closes #858

See merge request BuildStream/buildstream!1064

bb80a2b8

by Chandan Singh at 2019-01-14T14:30:15Z

.gitlab-ci.yml: Use newer image for overning aarch tests

During recent reorganizing of `.gitlab-ci.yml` to work with `tox`, seems
like we missed to update the image used by the overnight aarch tests,
meaning that they currently fail due to `tox` being missing from them.

While these tests will be skipped on MRs usually, here is an example of
what the tests look like if they are actually run with this change:
https://gitlab.com/BuildStream/buildstream/-/jobs/145449561.

Fixes #859.

10b3ee62

by Chandan Singh at 2019-01-14T14:57:08Z

Merge branch 'chandan/fix-overnight-aarch' into 'master'

Chandan/fix overnight aarch

Closes #859

See merge request BuildStream/buildstream!1065

db424ec3
by Benjamin Schubert at 2019-01-14T23:19:29Z
```
Blah
```

6 changed files:

.gitlab-ci.yml
CONTRIBUTING.rst
buildstream/_scheduler/queues/fetchqueue.py
buildstream/_scheduler/queues/queue.py
buildstream/element.py
requirements/dev-requirements.txt

Changes:

.gitlab-ci.yml

@@ -61,7 +61,7 @@ tests-ubuntu-18.04:
    <<: *tests
  overnight-fedora-28-aarch64:
 -  image: buildstream/testsuite-fedora:aarch64-28-06bab030-32a101f6
 +  image: buildstream/testsuite-fedora:aarch64-28-5da27168-32c47d1c
    tags:
      - aarch64
    <<: *tests
@@ -70,6 +70,12 @@ overnight-fedora-28-aarch64:
    except: []
    only:
    - schedules
 +  before_script:
 +    # grpcio needs to be compiled from source on aarch64 so we additionally
 +    # need a C++ compiler here.
 +    # FIXME: Ideally this would be provided by the base image. This will be
 +    # unblocked by https://gitlab.com/BuildStream/buildstream-docker-images/issues/34
 +    - dnf install -y gcc-c++
  tests-unix:
    # Use fedora here, to a) run a test on fedora and b) ensure that we
@@ -141,7 +147,7 @@ docs:
    variables:
      BST_EXT_URL: git+https://gitlab.com/BuildStream/bst-external.git
      BST_EXT_REF: 573843768f4d297f85dc3067465b3c7519a8dcc3 # 0.7.0
 -    FD_SDK_REF: 612f66e218445eee2b1a9d7dd27c9caba571612e # freedesktop-sdk-18.08.19-54-g612f66e2
 +    FD_SDK_REF: freedesktop-sdk-18.08.25-0-g250939d465d6dd7768a215f1fa59c4a3412fc337
    before_script:
    - |
      mkdir -p "${HOME}/.config"

CONTRIBUTING.rst

@@ -1534,6 +1534,10 @@ You can always abort on the first failure by running::
    tox -- -x
 +Similarly, you may also be interested in the ``--last-failed`` and
 +``--failed-first`` options as per the
 +`pytest cache <https://docs.pytest.org/en/latest/cache.html>`_ documentation.
++
  If you want to run a specific test or a group of tests, you
  can specify a prefix to match. E.g. if you want to run all of
  the frontend tests you can do::
@@ -1545,6 +1549,12 @@ If you wanted to run the test_build_track test within frontend/buildtrack.py you
    tox -- tests/frontend/buildtrack.py::test_build_track
 +When running only a few tests, you may find the coverage and timing output
 +excessive, there are options to trim them. Note that coverage step will fail.
 +Here is an example::
++
 +  tox -- --no-cov --durations=1 tests/frontend/buildtrack.py::test_build_track
++
  We also have a set of slow integration tests that are disabled by
  default - you will notice most of them marked with SKIP in the pytest
  output. To run them, you can use::

buildstream/_scheduler/queues/fetchqueue.py

@@ -44,9 +44,6 @@ class FetchQueue(Queue):
          element._fetch()
      def status(self, element):
 -        # state of dependencies may have changed, recalculate element state
 -        element._update_state()
+-
          if not element._is_required():
              # Artifact is not currently required but it may be requested later.
              # Keep it in the queue.

buildstream/_scheduler/queues/queue.py

@@ -170,9 +170,9 @@ class Queue():
          skip = [job for job in jobs if self.status(job.element) == QueueStatus.SKIP]
          wait = [job for job in jobs if job not in skip]
 +        self.skipped_elements.extend([job.element for job in skip])
          self._wait_queue.extend(wait)
          self._done_queue.extend(skip)
 -        self.skipped_elements.extend(skip)
      # dequeue()
+     #

buildstream/element.py

@@ -197,6 +197,7 @@ class Element(Plugin):
          self.__runtime_dependencies = []        # Direct runtime dependency Elements
          self.__build_dependencies = []          # Direct build dependency Elements
 +        self.__reverse_build_dependencies = []  # Direct reverse dependency Elements
          self.__sources = []                     # List of Sources
          self.__weak_cache_key = None            # Our cached weak cache key
          self.__strict_cache_key = None          # Our cached cache key for strict builds
@@ -439,6 +440,16 @@ class Element(Plugin):
          if should_yield and (recurse or recursed) and scope != Scope.BUILD:
              yield self
 +    def reverse_build_dependencies(self):
 +        def recurse_rdeps(element):
 +            yield element
++
 +            for rdep in element.__reverse_build_dependencies:
 +                yield from recurse_rdeps(rdep)
++
 +        for rdep in self.__reverse_build_dependencies:
 +            yield from recurse_rdeps(rdep)
++
      def search(self, scope, name):
          """Search for a dependency by name
@@ -930,6 +941,7 @@ class Element(Plugin):
          for meta_dep in meta.build_dependencies:
              dependency = Element._new_from_meta(meta_dep)
              element.__build_dependencies.append(dependency)
 +            dependency.__reverse_build_dependencies.append(element)
          return element
@@ -1306,6 +1318,9 @@ class Element(Plugin):
          self._update_state()
 +        for reverse_dep in self.reverse_build_dependencies():
 +            reverse_dep._update_state()
++
      # _track():
+     #
      # Calls track() on the Element sources
@@ -1503,6 +1518,9 @@ class Element(Plugin):
          self._update_state()
          if self._get_workspace() and self._cached_success():
 +            for rdep in self.reverse_build_dependencies():
 +                rdep._update_state()
++
              assert utils._is_main_process(), \
                  "Attempted to save workspace configuration from child process"
+             #

requirements/dev-requirements.txt

@@ -2,7 +2,7 @@ coverage==4.4
  pylint==2.2.2
  pycodestyle==2.4.0
  pytest==4.0.2
 -pytest-cov==2.6.0
 +pytest-cov==2.6.1
  pytest-datafiles==2.0
  pytest-env==0.6.2
  pytest-xdist==1.25.0

[Notes] [Git][BuildStream/buildstream][bschubert/pipeline] 10 commits: dev-requirements: pytest-cov==2.6.1 for '--no-cov'

Benjamin Schubert pushed to branch bschubert/pipeline at BuildStream / buildstream

Commits:

6 changed files:

Changes: