[Notes] [Git][BuildStream/buildstream][725-job-cancellation-on-remote-builds] 46 commits: Run tests on aarch64

[Jürg Billeter <gitlab mg gitlab com>]

Title: GitLab

Jürg Billeter pushed to branch 725-job-cancellation-on-remote-builds at BuildStream / buildstream

Commits:

• c87d6e45
  by Valentin David at 2018-11-14T11:43:33Z

  Run tests on aarch64
  
  Fixes #755
  

• 16b01489
  by Valentin David at 2018-11-14T11:43:33Z

  Disable cachekey tests on other architectures than x86_64
  

• 85046b29
  by Valentin David at 2018-11-14T11:43:33Z

  Disable tests on example on other architectures than x86_64
  

• dd5a073b
  by Valentin David at 2018-11-14T11:43:33Z

  Add support for aarch64 in a test
  

• c5d72ae7
  by Valentin David at 2018-11-14T11:52:53Z

  Test aarch64 only overnight
  

• 264a57f6
  by Javier Jardón at 2018-11-14T12:22:03Z

  Merge branch 'valentindavid/integration-tests-aarch64' into 'master'
  
  Run tests on aarch64
  
  Closes #369 and #755
  
  See merge request BuildStream/buildstream!948

• da735e56
  by Richard Maw at 2018-11-14T13:30:34Z

  _platform/linux.py: Move get_bwrap_version into _site.py
  
  It's inconvenient to have to create a Linux platform to parse the bwrap version
  and we want to get the version in a consistent manner.
  

• e7633500
  by Richard Maw at 2018-11-14T13:30:34Z

  buildstream/sandbox/_sandboxbwrap.py: Distinguish sandbox failure from command failure
  
  If `bwrap` fails to set up the sandbox and start the payload command
  it won't write an exit-code in --json-status-fd,
  so we can report if it was a sandboxing failure if we don't get exit-code status
  and a payload command failure if we do and it's non-zero.
  
  Closes https://gitlab.com/BuildStream/buildstream/issues/286
  

• e9e08823
  by Richard Maw at 2018-11-14T13:30:34Z

  tests/testutils/site.py: Check for bwrap supporting --json-status-fd
  

• 90ca007e
  by Richard Maw at 2018-11-14T13:30:34Z

  tests/integration/sandbox-bwrap.py: Test distinguishing sandbox exit code from command
  

• 327b19dd
  by richardmaw-codethink at 2018-11-14T13:59:16Z

  Merge branch 'richardmaw/distinguish-sandboxing-build-fail' into 'master'
  
  Distinguish between bubblewrap sandboxing failure and command failure
  
  Closes #286
  
  See merge request BuildStream/buildstream!868

• 082b3811
  by Angelos Evripiotis at 2018-11-15T19:48:12Z

  contributing: add guidance on unit tests
  
  Decrease uncertainty around whether unit tests are welcome in the
  project or not.
  

• b5b79056
  by Jürg Billeter at 2018-11-15T20:24:58Z

  Merge branch 'aevri/unit_tests' into 'master'
  
  contributing: add guidance on unit tests
  
  See merge request BuildStream/buildstream!943

• 199bfff1
  by Tom Pollard at 2018-11-17T13:07:59Z

  Add cli main and user config option for 'pull-buildtrees' context.
  
  _context.py: Add pull_buildtrees global user context, the default
  of which is set to False via the addition of pull-buildtrees to
  userconfig.yaml cache group.
  
  _frontend/app.py & cli.py: Add --pull-buildtrees as a bst main
  option, which when passed will override the default or user defined
  context for pull_buildtrees.
  
  tests/completions/completions.py: Update for the added flag.
  

• 454dcaf1
  by Tom Pollard at 2018-11-17T13:07:59Z

  Don't pull artifact build trees by default.
  
  The addition of cached build trees being included in element
  artifacts has led to mostly redundant download overheads when
  pulling from a remote artifact server. As such the default behaviour
  of pull shouldn't fetch the build tree object if available.
  
  element.py: extend relevant pull logic for specified subdir
  consideration and ensure push logic does not lead to partial
  artifact pushes. Change assumption that the buildtree can be
  extracted if artifact is cached. __cached_buildtree() and
  __pull_directories() helpers added.
  
  _artifactcache/: artifactcache.py & cascache.py inclusion of
  helper functions for subdir artifact checking & fetching, fetch
  logic extended to only pull required artifact directories.
  extract & checkout updated to handle full/partial operation.
  
  tests/: addition of integration test pullbuildtrees.py,
  buildtrees.py adapted cli options, testutils/artifactshare.py
  has_artifact changed to return artifact digest on true condition.
  

• 0d309478
  by Jürg Billeter at 2018-11-17T13:49:58Z

  Merge branch 'tpollard/494' into 'master'
  
  Don't pull artifact buildtrees by default
  
  See merge request BuildStream/buildstream!786

• 27932739
  by Benjamin Schubert at 2018-11-19T09:46:39Z

  plugins/sources/pip.py: also look for python version named "python"
  
  In some virtual environments, we might not have the binary with the
  exact python version, which would lead the current pip plugin to
  fail to find a working pip version.
  
  This adds "python" at the start of the list of valid python versions
  

• ea2de561
  by Benjamin Schubert at 2018-11-19T10:19:24Z

  Merge branch 'bschubert/fix-pip-python' into 'master'
  
  plugins/sources/pip.py: also look for python version named "python"
  
  Closes #758
  
  See merge request BuildStream/buildstream!942

• f23b6031
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  tests/frontend/buildtrack.py: check for success after invocation
  
  Previously the result of the first call in test_build_track was not
  checked to be successful, which would lead to potential erros later on
  that were hard to debug.
  
  Let's check it everytime
  

• a6defc0b
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  Fix os.rename in git source element to correctly handle error codes
  
  According to the documentation
  (https://www.unix.com/man-page/POSIX/3posix/rename/), when the directory
  already is there, either EEXIST or ENOTEMPTY could be thrown.
  
  Previously only ENOTEMPTY was checked.
  
  Done:
    - Separated the move into its own function
    - Check for both errors
    - Create unit tests for it, covering most test cases
  

• 88089d2d
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  Extract atomic move function to utils.py
  
  Moving atomically a file/directory can be tricky since different
  errors might be raised for the same underlying problem.
  
  Having a utility function to reduce this discrepancies will help
  in ensuring we have correct behavior
  

• fd9e46be
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  cascache.py: use move_atomic instead of manual error checking
  

• d32e0b83
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  sources/pip.py: Use move_atomic instead of manual os.rename
  
  This uses move_atomic insteand of the manual os.rename and manual error
  checking and throws a SourceError for consistency with other modules.
  

• 6f837118
  by Benjamin Schubert at 2018-11-19T12:22:40Z

  Merge branch 'bschubert/fix-atomic-move-git-repo' into 'master'
  
  Fix os.rename in git source element to correctly handle error codes
  
  See merge request BuildStream/buildstream!938

• 30b72244
  by Chandan Singh at 2018-11-19T12:35:01Z

  buildstream/_pipeline.py: Simplify error message for assert_consistent()
  
  Currently, `Pipeline.assert_consistent()` prints an error message like
  the one below when certain elements have inconsistent sources:
  
  ```
  ...
        Element: hello.bst is inconsistent
          Source tar source at hello.bst [line 16 column 2] is missing ref
  ...
  ```
  
  Drop the word "Source" from the beginning of the message as
  "Source <source-kind> source ..." is kind of awkward to read.
  

• 76c5d2f8
  by Chandan Singh at 2018-11-19T12:35:01Z

  Add `bst source-checkout` command
  
  As discussed in
  https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
  add `bst source-checkout` command. This will allow users to checkout
  sources for a given target.
  
  * _frontend/cli.py: Add source-checkout command
  * _pipeline.py: Add assert_sources_cached() method
  * _stream.py: Add source_checkout method, abstract out
    __check_location_writable() method that used to part of checkout()
  

• 9f629638
  by Chandan Singh at 2018-11-19T13:03:36Z

  Merge branch 'chandan/source-checkout' into 'master'
  
  Add `bst source-checkout` command
  
  See merge request BuildStream/buildstream!820

• bc827cc8
  by Abderrahim Kitouni at 2018-11-19T14:55:15Z

  plugins/elements/cmake.yaml: always specify variable types
  
  cmake sometimes misinterprets relative paths as relative to the current directory
  if this is not specified. See freedesktop-sdk/freedesktop-sdk#431
  
  adjust tests/format/variables.py accordingly.
  

• 5fbc5f41
  by Valentin David at 2018-11-19T15:31:05Z

  Merge branch 'abderrahim/cmake-variable-types' into 'master'
  
  plugins/elements/cmake.yaml: always specify variable types
  
  Closes #761
  
  See merge request BuildStream/buildstream!947

• 06e28860
  by Benjamin Schubert at 2018-11-19T15:52:24Z

  Don't cache sandbox errors
  
  Sandbox errors (like missing host tools) are dependent on the host
  system and rarely on what is actually done.
  
  It is therefore better to not cache them as they are subject to
  change between two runs.
  
  Also add test to ensure sandbox failure are not cached
  

• dd36cfbc
  by Benjamin Schubert at 2018-11-19T15:52:24Z

  element.py: Only cache on element error or success
  
  Reverse the way we were handling caching by only caching
  when we know it's a problem with the build or the build is a
  success in order not to cache transient errors.
  

• eac7274d
  by Benjamin Schubert at 2018-11-19T16:29:07Z

  Merge branch 'bschubert/dont-cache-errors-from-host-tools' into 'master'
  
  Don't cache sandbox failures
  
  Closes #727
  
  See merge request BuildStream/buildstream!895

• 625dfe1f
  by Chandan Singh at 2018-11-19T16:41:17Z

  NEWS: Add entry for the new source-checkout command
  
  `bst source-checkout` command was recently added in
  https://gitlab.com/BuildStream/buildstream/merge_requests/820. Add a
  NEWS entry for the new command.
  

• d4f12184
  by Chandan Singh at 2018-11-19T17:15:10Z

  Merge branch 'chandan/source-checkout-news' into 'master'
  
  NEWS: Add entry for the new source-checkout command
  
  See merge request BuildStream/buildstream!960

• ba6c96e8
  by Richard Maw at 2018-11-19T19:54:30Z

  CasBasedDirectory: Change constructor to take a CASCache instead of a Context
  
  The Context was only used to obtain a reference to the CASCache
  and set the unused cas_directory field.
  

• 8a0dc3a3
  by Jürg Billeter at 2018-11-19T19:54:30Z

  tests/testutils/artifactshare.py: Do not create a fake context
  
  The fake context did not set the cache quota, triggering an error. With
  CASCache now separate from ArtifactCache, we can instantiate a CASCache
  without context.
  

• 8722aced
  by Jürg Billeter at 2018-11-19T19:54:30Z

  tests/plugins/pipeline.py: Load context default values
  

• fc56ffa4
  by Jürg Billeter at 2018-11-19T19:54:30Z

  _context.py: Drop duplicated default values for user configuration
  
  The default values are in userconfig.yaml, together with the
  documentation. The default values should not be duplicated in
  _context.py.
  

• 3f663d82
  by Jürg Billeter at 2018-11-19T20:36:57Z

  Merge branch 'juerg/context-default-values' into 'master'
  
  _context.py: Drop duplicated default values for user configuration
  
  See merge request BuildStream/buildstream!953

• b498cce7
  by Angelos Evripiotis at 2018-11-20T11:17:38Z

  _yaml: document node_get()'s default_value arg
  

• eb2d376f
  by Angelos Evripiotis at 2018-11-20T11:17:38Z

  _context: allow 'terminate' for scheduler.on-error
  
  Enable this option of 'terminate', which is mentioned in userconfig.yaml
  and handled in _frontend/app.py:_handle_failure(). It appears to have
  been left out of the valid_actions as an oversight.
  
  Originally introduced in
  https://gitlab.com/BuildStream/buildstream/commit/2622d5da9bd9fefd87436613d6e3e9770fdd0f28
  

• b81c4333
  by Angelos Evripiotis at 2018-11-20T11:17:38Z

  _context: refactor, extract _node_get_option_str
  
  Use a new helper function to simplify working with nodes that can only
  accept certain strings. This will be used when adding the prompt.*
  config options.
  
  In later work we can see if this function would be useful elsewhere, and
  could be added to '_yaml.py'.
  

• 27ca6593
  by Angelos Evripiotis at 2018-11-20T11:19:33Z

  Add prompt.auto-init buildstream.conf option
  
  Provide an option in buildstream.conf to disable the 'Would you like to
  ...' prompt when we cannot resolve a project.
  
  Some users prefer not to be interrupted by such prompts, so pave the way
  to creating options to disable all those that might get in the way.
  
  Follow the example of the advice.* options 'git-config', and create a
  namespace for these UI options grouped by behaviour, rather than an
  over-reaching 'ui.*' namespace. In later work perhaps we'll also add
  'advice.*' options.
  
  Add a NEWS item for this.
  

• 7ae3a3d2
  by Angelos Evripiotis at 2018-11-20T11:43:49Z

  Add prompt.workspace-... options
  
  Provide options in project.conf to disable the 'Are you sure ...'
  prompts when making destructive changes:
  
  - Add prompt.really-workspace-close-remove-dir
  - Add prompt.really-workspace-reset-hard
  
  Add a NEWS item for these.
  

• 8071c00c
  by Angelos Evripiotis at 2018-11-20T12:12:11Z

  Merge branch 'aevri/prompt-config' into 'master'
  
  Add 'prompt.*' config options to buildstream.conf
  
  See merge request BuildStream/buildstream!887

• fd41b0b5
  by Raoul Hidalgo Charman at 2018-11-20T16:32:10Z

  _sandboxremote.py: Add sigterm handler that sends CancelOperation
  

30 changed files:

• .gitlab-ci.yml
• CONTRIBUTING.rst
• NEWS
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_frontend/cli.py
• buildstream/_pipeline.py
• buildstream/_platform/linux.py
• buildstream/_site.py
• buildstream/_stream.py
• buildstream/_yaml.py
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/plugins/elements/cmake.yaml
• buildstream/plugins/sources/git.py
• buildstream/plugins/sources/pip.py
• buildstream/sandbox/_sandboxbwrap.py
• buildstream/sandbox/_sandboxremote.py
• buildstream/sandbox/sandbox.py
• buildstream/storage/_casbaseddirectory.py
• buildstream/utils.py
• conftest.py
• tests/artifactcache/push.py
• tests/cachekey/cachekey.py
• tests/completions/completions.py
• tests/examples/autotools.py
• tests/examples/developing.py
• tests/examples/flatpak-autotools.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.