[Notes] [Git][BuildStream/buildstream][willsalmon/580-backport] 45 commits: buildstream/_project.py:

[Will Salmon <gitlab mg gitlab com>]

Title: GitLab

Will Salmon pushed to branch willsalmon/580-backport at BuildStream / buildstream

Commits:

• 6c5a3f0a
  by Javier Jardón at 2018-08-09T16:32:39Z

  buildstream/_project.py:
  
  Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for
  systems where not all CPU can be used (in particular, Docker, CI etc).
  
  As suggested in the multiprocessing docs
  (http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count)
  using len(os.sched_getaffinity(0)) might be better.
  

• 5adf32f1
  by Javier Jardón at 2018-08-09T16:32:40Z

  buildstream/_project.py: Restrict max-jobs
  
  even if the machine have mores cores available
  
  Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227
  

• cd59993d
  by Javier Jardón at 2018-08-09T17:19:44Z

  Merge branch 'jjardon/max-jobs_1.2' into 'bst-1.2'
  
  [1.2] Backport of !620: Restrict max-jobs
  
  See merge request BuildStream/buildstream!629

• 9f507407
  by Jim MacArthur at 2018-08-10T04:04:29Z

  setup.py: Pin ruamel.yaml version to <= 0.15
  

• 1b652035
  by Tristan Van Berkom at 2018-08-10T05:29:21Z

  Merge branch 'tristan/pin-ruamel-version-1.2' into 'bst-1.2'
  
  Backport fix for ruamel.yaml version to 1.2
  
  See merge request BuildStream/buildstream!631

• 13e21035
  by Valentin David at 2018-08-10T10:15:04Z

  Set environment in bwrap command line instead of its environment
  
  Fixes #498
  

• 7bc4a817
  by Valentin David at 2018-08-10T13:34:17Z

  Merge branch 'valentindavid/498_bwrap_environment-1.2' into 'bst-1.2'
  
  Set environment in bwrap command line instead of its environment
  
  See merge request BuildStream/buildstream!632

• f03df3ce
  by Chandan Singh at 2018-08-11T23:44:19Z

  Move development reqirements to dev-requirements.txt
  
  This is backport of !637 to bst-1.2.
  
  There were a couple of differences between the requirements listed in
  the `master` branch and the `bst-1.2` branch so I have left them
  untouched.
  
  For reference, here are the differences between them:
  
  ```diff
  --- a/dev-requirements.txt
  +++ b/dev-requirements.txt
  @@ -2,8 +2,9 @@
   # random crashes with 4.4.2
   coverage == 4.4.0
   pep8
  +pylint >= 1.8 , < 2
   pytest >= 3.1.0
  -pytest-cov >= 2.5.0
  +pytest-cov
   pytest-datafiles
   pytest-env
   pytest-pep8
  ```
  

• ba58532d
  by Javier Jardón at 2018-08-12T01:37:18Z

  Merge branch 'chandan/dev-reqs-1.2' into 'bst-1.2'
  
  Move development reqirements to dev-requirements.txt
  
  See merge request BuildStream/buildstream!642

• bc4c62b8
  by Phil Dawson at 2018-08-12T01:41:38Z

  .gitlab-ci-yml: Add ubuntu 18.04 test
  

• 3e5a684b
  by Javier Jardón at 2018-08-12T01:42:06Z

  .gitlab-ci.yml: Run tests in fedora28 image
  

• e5032b7c
  by Javier Jardón at 2018-08-12T01:42:14Z

  .gitlab-ci.yml: Use the same version of the docker image for all jobs
  

• a42d5c5f
  by Jürg Billeter at 2018-08-12T02:05:05Z

  Revert "Restrict version of pylint"
  
  This reverts commit 4f168b9b6a02216e2fae24d758ae6b778e545869.
  
  The latest version of pytest_pylint works fine with pylint 2, which
  means there is no longer a reason to restrict the pylint version.
  
  pylint 2 is required for Python 3.7.
  

• f2f572cb
  by Tristan Van Berkom at 2018-08-12T02:07:38Z

  setup.py: Specify minimum required version of pytest-cov plugin
  
  This causes the new artifact tests to pass (unless you happened
  to already have a recent enough version of pytest-cov, in which case
  you didn't notice the breakage).
  

• dee0e9bf
  by Javier Jardón at 2018-08-12T03:20:25Z

  dev-requirements.txt: Remove obsolete comments
  
  This also fixes an error with pylint in setup.py:
  setup.py:226:19: R1718: Consider using a set comprehension (consider-using-set-comprehension)
  

• 37742ac7
  by Javier Jardón at 2018-08-12T03:35:45Z

  setup.py: fix previous commit
  

• b9f6b1d7
  by Valentin David at 2018-08-12T05:57:44Z

  buildstream/plugins/sources/local.py: Make staging deterministic.
  
  Instead of copying metadata on files staged by local, we manually set
  mode to 0755 or 0644 depending on whether user execution was enabled
  on source file.
  
  This makes file modes deterministic independently on the way source
  was distributed.
  
  Non-deterministic mode copying all metadata can still be enabled by
  disable 'deterministic' Boolean configuration on the plugin.
  
  Fixes #527.
  

• 465909dd
  by Valentin David at 2018-08-12T05:57:44Z

  Use deterministic umask when staging sources.
  
  This fix is applied to plugins bzr, git, patch.
  
  Fixes #543 #544 #555.
  

• e8a3297b
  by Valentin David at 2018-08-12T05:57:44Z

  buildstream/plugins/sources/zip.py: Fix non-determism in staging.
  
  Staging could end-up with file with different rights depending on the
  umask. The extracted files need to get their access rights fixed.
  

• 9a0259b9
  by Valentin David at 2018-08-12T05:57:44Z

  buildstream/plugins/sources/remote.py: Make staging deterministic.
  

• 4c15d049
  by Valentin David at 2018-08-12T05:57:44Z

  Add some integration tests for source plugin determinism.
  

• b7b4b718
  by Valentin David at 2018-08-12T05:57:44Z

  Bump BST_CORE_ARTIFACT_VERSION for deterministic source plugins
  

• e5dfbb54
  by Tristan Van Berkom at 2018-08-12T07:00:17Z

  Merge branch 'valentindavid/deterministic-source-1.2' into 'bst-1.2'
  
  Deterministic staging
  
  See merge request BuildStream/buildstream!634

• 2b5c63d0
  by Valentin David at 2018-08-13T09:34:59Z

  Fix tracking of junctions used in project.conf.
  
  Stream._load() now returns early without resolving build pipeline when
  only tracking. Resolving track pipelines does not require to fully
  load project configurations when when elements to track are only
  junctions.
  
  However build pipelines require to fully load project
  configurations. This might not be possible in the case a project
  configuration includes a file from a junction that yet needs to be
  tracked.
  
  Fixes #565.
  

• 650c4d8d
  by Valentin David at 2018-08-13T11:03:12Z

  Merge branch 'valentindavid/fix_included_junction_track-1.2' into 'bst-1.2'
  
  Fix tracking of junctions used in project.conf.
  
  See merge request BuildStream/buildstream!645

• f53e12c2
  by Valentin David at 2018-08-13T12:39:42Z

  Set version of ruaml.yaml to strictly less than 0.15.52.
  
  This allows to use version 0.15.51 rather than 0.15.0 which is
  required for Python 3.
  
  Fixes #571.
  

• 01850d55
  by Valentin David at 2018-08-13T13:48:09Z

  Merge branch 'valentindavid/ruamel-version-1.2' into 'bst-1.2'
  
  Set version of ruaml.yaml to at least 0.15.41 but strictly less than 0.15.52.
  
  See merge request BuildStream/buildstream!647

• 064abe66
  by Sam Thursfield at 2018-08-13T13:50:08Z

  Fix crash when --debug is passed
  
  I hit the following backtrace running `bst --debug push` and `bst
  --debug build`:
  
      pid:16736 id:000[--:--:--][][] START   Push
      pid:16736 id:000[--:--:--][][] START   Loading pipeline
      pid:16736 id:000[00:00:00][][] SUCCESS Loading pipeline
      pid:16736 id:000[--:--:--][][] START   Resolving pipeline
      pid:16736 id:000[--:--:--][][] BUG     'MesonElement' object has no attribute '_Element__cache_key'
  
          Traceback (most recent call last):
            File "/home/sam/.local/bin/bst", line 8, in <module>
              sys.exit(cli())
            File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
              return self.main(*args, **kwargs)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/cli.py", line 162, in override_main
              standalone_mode=standalone_mode, **extra)
            File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 697, in main
              rv = self.invoke(ctx)
            File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
              return _process_result(sub_ctx.command.invoke(sub_ctx))
            File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
              return ctx.invoke(self.callback, **ctx.params)
            File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
              return callback(*args, **kwargs)
            File "/home/sam/.local/lib/python3.6/site-packages/click/decorators.py", line 27, in new_func
              return f(get_current_context().obj, *args, **kwargs)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/cli.py", line 471, in push
              app.stream.push(elements, selection=deps, remote=remote)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_stream.py", line 341, in push
              fetch_subprojects=True)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_stream.py", line 814, in _load
              fetch_subprojects=fetch_subprojects)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_pipeline.py", line 119, in load
              for meta in meta_elements
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_pipeline.py", line 119, in <listcomp>
              for meta in meta_elements
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 894, in _new_from_meta
              element = project.create_element(artifacts, meta)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_project.py", line 184, in create_element
              element = self._element_factory.create(self._context, self, artifacts, meta)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_elementfactory.py", line 57, in create
              return element_type(context, project, artifacts, meta, default_config)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 196, in __init__
              super().__init__(meta.name, context, project, meta.provenance, "element")
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 171, in __init__
              self.debug("Created: {}".format(self))
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 407, in debug
              self.__message(MessageType.DEBUG, brief, detail=detail)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 652, in __message
              self.__context.message(message)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_context.py", line 338, in message
              self._message_handler(message, context=self)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/app.py", line 665, in _message_handler
              text = self.logger.render(message)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 585, in render
              return self._render(message)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 620, in _render
              text += widget.render(message)
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 228, in render
              _, key, missing = plugin._get_display_key()
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 1138, in _get_display_key
              cache_key = self._get_cache_key()
            File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 991, in _get_cache_key
              return self.__cache_key
          AttributeError: 'MesonElement' object has no attribute '_Element__cache_key'
  
  The issue here is that when `--debug` is enabled, Plugin.__init__()
  tries to log a message like this:
  
          pid:16929 id:001[--:--:--][????????][ main:tracker.bst                   ] DEBUG   Created: meson element at tracker.bst [line 1 column 0]
  
  The log formatter is trying to get the cache key of the element in
  order to show it in the log, but the attribute hasn't yet been set
  because we've not got to the Element() constructor.
  

• 6a2f3b59
  by Valentin David at 2018-08-13T15:25:42Z

  Merge branch 'valentindavid/fix-debug-crash-1.2' into 'bst-1.2'
  
  Fix crash when --debug is passed
  
  See merge request BuildStream/buildstream!648

• 002749b5
  by Jürg Billeter at 2018-08-13T15:39:22Z

  _artifactcache/cascache.py: Fix for PEP 479 / Python 3.7
  
  Do not rely on `StopIteration` bubbling up.
  
  https://www.python.org/dev/peps/pep-0479/
  

• 60df233d
  by Valentin David at 2018-08-13T17:44:03Z

  Merge branch 'valentindavid/python3.7-1.2' into 'bst-1.2'
  
  Python 3.7 support
  
  See merge request BuildStream/buildstream!649

• 60c817c0
  by Phillip Smyth at 2018-08-13T19:04:16Z

  _stream.py: Added functionality for workspace open -f
  
  tests/frontend/workspace.py: Added tests
  

• 2315fff0
  by Valentin David at 2018-08-13T20:04:28Z

  Merge branch 'valentindavid/bst_workspace_open_force_does_nothing-1.2' into 'bst-1.2'
  
  _stream.py: Added functionality for workspace open -f
  
  See merge request BuildStream/buildstream!651

• b3fac258
  by Francisco Redondo Marchena at 2018-08-13T20:23:50Z

  source-bundle: Enable --except option
  
  Before this option was listed in help but was ignored when
  creating the source-bundle.
  
  Issue #468
  

• 41d97b7e
  by Valentin David at 2018-08-13T21:18:08Z

  Merge branch 'valentindavid/fix-except-argument-in-source-bundle-1.2' into 'bst-1.2'
  
  source-bundle: Enable --except option
  
  See merge request BuildStream/buildstream!650

• 83710050
  by Martin Blanchard at 2018-08-13T21:37:28Z

  complete.py: Ensure paths get completed from 'element-path'
  
  Element paths should always be completed from the root element folder
  defined by the element-path key in project.conf. Fix complete_path() to
  always search into its given base_directory argument.
  
  See issue BuildStream/buildstream#448
  

• d6714dc2
  by Valentin David at 2018-08-13T22:51:26Z

  Merge branch 'valentindavid/448-autocompletion-broken-defaults' into 'bst-1.2'
  
  Fix autocompletion for elements in sub-folders
  
  See merge request BuildStream/buildstream!652

• b82d6793
  by Phil Dawson at 2018-08-14T05:33:23Z

  tests/frontend/show.py: Add test case for maximum recursion depth being exceeded
  
  Add test to ensure gracefull handling of exception thrown while loading
  a pipeline due the python's max recursion depth being exceeded.
  
  This is part of the work for issue #203
  

• 9e2b66cc
  by Phil Dawson at 2018-08-14T05:33:23Z

  app.py: Handle exception thrown when recursion limit is exceeded
  
  Because the RecursionError exception was introduced in Python 3.5, until we
  drop support for for Python 3.4, we must use RuntimeError.
  

• cbdeba7b
  by Tristan Van Berkom at 2018-08-14T06:45:42Z

  Merge branch 'valentindavid/203-BuildStream-crashes-when-dependency-tree-too-deep' into 'bst-1.2'
  
  203 build stream crashes when dependency tree too deep
  
  See merge request BuildStream/buildstream!654

• 3088aca2
  by Martin Blanchard at 2018-08-14T07:12:45Z

  element.py: Raise an exception on unbuilt element checkout
  
  Trying to checkout an element that has no cached artifacts should be and
  handled failure. See BuildStream/buildstream#447.
  

• 7a9cd5ff
  by Martin Blanchard at 2018-08-14T07:12:45Z

  _stream.py: Print details on checkout failure
  

• 3c692ee7
  by Tristan Van Berkom at 2018-08-14T08:17:23Z

  Merge branch 'valentindavid/447-stack-trace-checkout' into 'bst-1.2'
  
  Handle checkout failure for unbuilt elements
  
  See merge request BuildStream/buildstream!653

• 0155d52d
  by William Salmon at 2018-08-14T10:07:58Z

  Add Error to git source configure if track and ref are not present
  
  This is to address https://gitlab.com/BuildStream/buildstream/issues/471
  that documented unhelpful behaviour when tracking git sources.
  

• af4fa28e
  by William Salmon at 2018-08-14T10:08:46Z

  Improve error message for build if there are refs missing
  

30 changed files:

• .gitlab-ci.yml
• MANIFEST.in
• buildstream/_artifactcache/cascache.py
• buildstream/_frontend/app.py
• buildstream/_frontend/cli.py
• buildstream/_frontend/complete.py
• buildstream/_pipeline.py
• buildstream/_project.py
• buildstream/_stream.py
• buildstream/_versions.py
• buildstream/element.py
• buildstream/plugins/sources/git.py
• buildstream/plugins/sources/local.py
• buildstream/plugins/sources/remote.py
• buildstream/plugins/sources/zip.py
• buildstream/sandbox/_sandboxbwrap.py
• buildstream/utils.py
• + dev-requirements.txt
• setup.py
• tests/cachekey/project/elements/build1.expected
• tests/cachekey/project/elements/build2.expected
• tests/cachekey/project/elements/compose1.expected
• tests/cachekey/project/elements/compose2.expected
• tests/cachekey/project/elements/compose3.expected
• tests/cachekey/project/elements/compose4.expected
• tests/cachekey/project/elements/compose5.expected
• tests/cachekey/project/elements/import1.expected
• tests/cachekey/project/elements/import2.expected
• tests/cachekey/project/elements/import3.expected
• tests/cachekey/project/elements/script1.expected

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.