[Notes] [Git][BuildStream/buildstream][adamjones/labels] 736 commits: _frontend/app.py: Set correct element-path in interactive bst-init

[Angelos Evripiotis <gitlab mg gitlab com>]

Title: GitLab

Angelos Evripiotis pushed to branch adamjones/labels at BuildStream / buildstream

Commits:

• 516e990e
  by ctolentino8 at 2018-10-31T11:36:46Z

  _frontend/app.py: Set correct element-path in interactive bst-init
  

• b8a37a63
  by Tristan Van Berkom at 2018-11-01T10:16:25Z

  Merge branch 'chiaratolentino/bst-init-interactive-elementpath' into 'master'
  
  _frontend/app.py: Set correct element-path in interactive bst-init
  
  See merge request BuildStream/buildstream!912

• b27b592a
  by Benjamin Schubert at 2018-11-01T10:49:57Z

  Remove dependency on pytest-runner
  
  This includes a new command mimicking pytest-runner so that we
  can drop this dependency
  
  This was the only setup_requires dependency that we had and
  will make like easier for people behind proxies
  

• 89ace5d7
  by Benjamin Schubert at 2018-11-01T11:16:36Z

  Merge branch 'bschubert/remove-pytest-runner' into 'master'
  
  Remove dependency on pytest-runner
  
  See merge request BuildStream/buildstream!886

• 4cfabce8
  by Angelos Evripiotis at 2018-11-01T11:35:02Z

  BREAK: manual.yaml: don't set any default env vars
  
  Remove the 'MAKEFLAGS' and 'V' defaults.
  
  Now that there is a specialised 'make' element, these make-specific
  defaults don't make sense here. This element is meant to be a blank
  slate for folks to build on.
  
  Note that this is a breaking change, that will require some users to
  make changes to their .bst files if they are expecting these environment
  variable to be set.
  
  _versions.py: BST_FORMAT_VERSION bumped to 18.
  
  This fixes issue #718
  

• 48860aac
  by Tristan Van Berkom at 2018-11-01T12:01:04Z

  Merge branch 'tristan/empty-manual-element' into 'master'
  
  BREAK: manual.yaml: don't set any default env vars
  
  Closes #718
  
  See merge request BuildStream/buildstream!916

• d868b409
  by Daniel Silverstone at 2018-11-01T13:40:24Z

  _yaml.py: Implement `get()` for `ChainMap`
  
  Since the core Python `ChainMap.get()` implements with:
  
      self[key] if key in self else default
  
  The double-chain-lookup is expensive.  This simple change solves that
  for our ChainMap derived structure.  As such it improves matters
  for #466 somewhat.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 7f79b9ce
  by Tristan Van Berkom at 2018-11-01T14:25:57Z

  Merge branch 'danielsilverstone-ct/improve-chainmap-get' into 'master'
  
  _yaml.py: Implement `get()` for `ChainMap`
  
  See merge request BuildStream/buildstream!919

• de59ebdb
  by ctolentino8 at 2018-11-02T16:41:54Z

  plugins/sources/pip.py: Accomodate characters '-','.','_' for packages
  

• 8d7cf806
  by ctolentino8 at 2018-11-02T16:41:54Z

  tests/integration/project/files: Remove pypi-repo
  

• 9c2f9bf7
  by Chandan Singh at 2018-11-02T17:09:46Z

  Merge branch 'chiaratolentino/fix-pip-source-regex' into 'master'
  
  plugins/sources/pip.py: Accomodate characters '-','.','_' for packages
  
  See merge request BuildStream/buildstream!914

• 3788e701
  by Jürg Billeter at 2018-11-03T11:52:00Z

  element.py: Pass bare_directory also to SandboxRemote()
  
  Fixes: 825963b5 ("element: Make "--sysroot" take a bare directory")
  

• 82e971ef
  by Jürg Billeter at 2018-11-05T11:33:20Z

  Merge branch 'juerg/remote-execution' into 'master'
  
  element.py: Pass bare_directory also to SandboxRemote()
  
  See merge request BuildStream/buildstream!928

• 62942bfd
  by Valentin David at 2018-11-05T12:14:20Z

  Use less fetchers in overnight build to work-around limitations from servers.
  
  Issue happens with sourceware.org server.
  
  Fixes #742
  

• 442da2f9
  by Javier Jardón at 2018-11-05T12:41:54Z

  Merge branch 'valentindavid/limit_fetchers_in_overnight_build' into 'master'
  
  Use less fetchers in overnight build to work-around limitations from servers.
  
  Closes #742
  
  See merge request BuildStream/buildstream!929

• 0993de53
  by Richard Maw at 2018-11-05T16:41:05Z

  dev-requirements: Update pytest dependency to 3.8
  
  "_frontend/app.py: Set correct element-path in interactive bst-init"
  added a dependency on the tmp_path test fixture, which was added by 3.8.
  

• be8f0a54
  by richardmaw-codethink at 2018-11-05T17:08:43Z

  Merge branch 'richardmaw/pytest-dep-update' into 'master'
  
  dev-requirements: Update pytest dependency to 3.8
  
  See merge request BuildStream/buildstream!930

• bfb639bf
  by Jürg Billeter at 2018-11-05T17:18:12Z

  element.py: Remove artifacts parameter from constructor
  
  Get the artifact cache from the context instead.
  

• ca855f91
  by Jürg Billeter at 2018-11-05T17:18:12Z

  _elementfactory.py: Remove unused artifacts parameter from create()
  

• 15fed21c
  by Jürg Billeter at 2018-11-05T17:18:12Z

  _project.py: Remove unused artifacts parameter from create_element()
  

• 0085d2aa
  by Jürg Billeter at 2018-11-05T17:18:12Z

  element.py: Remove unused artifacts parameter from _new_from_meta()
  

• f69b1117
  by Jürg Billeter at 2018-11-05T17:18:12Z

  _project.py: Remove unused artifacts parameter from load_elements()
  

• e398f877
  by Jürg Billeter at 2018-11-05T17:18:12Z

  element.py: Do not include type name of artifact cache in cache key
  
  The artifact cache backend does not affect build outputs and we anyway
  no longer have pluggable artifact cache backends. This hardcodes
  CASCache instead of removing the entry completely to avoid cache key
  changes.
  

• 626d20ae
  by Jürg Billeter at 2018-11-05T17:18:12Z

  Split up artifact cache and CAS cache
  
  This changes CASCache from a subclass to a delegate object of
  ArtifactCache. As the lower layer, CASCache no longer deals with
  elements or projects.
  
  Fixes #659.
  

• ec04446b
  by Jürg Billeter at 2018-11-05T17:51:39Z

  Merge branch 'juerg/cas' into 'master'
  
  Split up artifact cache and CAS cache
  
  Closes #659
  
  See merge request BuildStream/buildstream!922

• 9a045080
  by Josh Smith at 2018-11-06T13:23:19Z

  utils: Fix _pretty_size() for sizes > 1024T
  

• d3a07e6b
  by Josh Smith at 2018-11-06T13:23:19Z

  Add regression test for _pretty_size
  

• 0c09fb9c
  by richardmaw-codethink at 2018-11-06T14:12:20Z

  Merge branch 'Qinusty/unit-test-utils' into 'master'
  
  Fix issue with _pretty_size with large numbers of bytes
  
  See merge request BuildStream/buildstream!799

• b4eec489
  by Jim MacArthur at 2018-11-06T16:44:14Z

  _sandboxremote.py: Use the standard SandboxError.
  
  Replaces the custom one which was erroneously added during development.
  Fixes #746.
  

• 1f7acf74
  by Jim MacArthur at 2018-11-06T17:16:31Z

  Merge branch 'jmac/remote-sandbox-errors' into 'master'
  
  _sandboxremote.py: Use the standard SandboxError
  
  Closes #746
  
  See merge request BuildStream/buildstream!931

• 8e64ccef
  by James Ennis at 2018-11-07T11:06:11Z

  using_config.rst: Update our instructions for how we declare caches in user config
  

• 09ab676d
  by James Ennis at 2018-11-07T11:06:11Z

  format_project.rst: Update instructions regarding how we declare remote caches in project.conf
  

• f514124f
  by James Ennis at 2018-11-07T11:06:11Z

  using_configuring_artifact_server.rst: Remove examples and redirect to user config/project config
  

• 029ba17d
  by richardmaw-codethink at 2018-11-07T11:32:20Z

  Merge branch 'jennis/docs_patch' into 'master'
  
  Update our instructions/examples regarding how we declare remote caches
  
  See merge request BuildStream/buildstream!932

• 261e2cd3
  by Jim MacArthur at 2018-11-07T12:02:43Z

  element.py: Use _force_rmtree instead of custom code.
  
  There are arguments for and against using onerror - see issue #735
  for details.
  

• 8931e42c
  by Jim MacArthur at 2018-11-07T12:26:06Z

  Merge branch 'jmac/make_writable_combination' into 'master'
  
  Use _force_rmtree instead of custom code.
  
  Closes #735
  
  See merge request BuildStream/buildstream!927

• 6ccfab0b
  by Valentin David at 2018-11-07T16:28:47Z

  Fix cache corruption by scripts when layout and integration commands are used
  
  Root directory was marked as a non-artifact mount, so not using
  SafeHardLink.  However integration commands executed with write access
  to the root directory.
  
  Fixes #749
  

• f24e20e9
  by richardmaw-codethink at 2018-11-07T16:59:50Z

  Merge branch 'valentindavid/script-artifact-corruption' into 'master'
  
  Fix cache corruption by scripts when layout and integration commands are used
  
  Closes #749
  
  See merge request BuildStream/buildstream!934

• e28f668a
  by Valentin David at 2018-11-08T09:34:11Z

  Fix infinite recursion in default strip debug command
  
  Fixes #645.
  

• 83d15350
  by Valentin David at 2018-11-08T10:01:39Z

  Merge branch 'valentindavid/fix-recursive-strip-debug' into 'master'
  
  Fix infinite recursion in default strip debug command
  
  Closes #645
  
  See merge request BuildStream/buildstream!918

• 9f0e12f1
  by Benjamin Schubert at 2018-11-08T10:21:12Z

  Move bwrap checks in platform/linux.py
  
  Remove the bwraps checks from _site.py and put them in platform.linux
  which is the only place where they are run.
  
  This allows the removal of a double level of caching, making
  reasoning about tests easier
  

• cf2e0059
  by Benjamin Schubert at 2018-11-08T10:21:12Z

  conftest.py: Ensure platform is not maintained between tests
  
  This removes the `_instance` on the platform object that we use
  for caching and not recreating the object everytime at the start
  of every test.
  
  This is to ensure our tests share the least amount of state.
  
  The performance penalty is from 5 to 10% accross the whole test suite.
  The readings were done 5 times for each before and after the change
  and on the same computer.
  

• c51ba01b
  by Benjamin Schubert at 2018-11-08T10:21:12Z

  Test that helpful messages are raised when missing dependencies
  
  This adds a `reason` to the SandboxEror thrown in sandboxdummy
  to be able to understand where the error comes from
  

• f116b9b7
  by Benjamin Schubert at 2018-11-08T11:07:12Z

  Merge branch 'bschubert/tests-no-chroot-linux' into 'master'
  
  test behavior when there is no sandbox on Linux
  
  Closes #696 and #736
  
  See merge request BuildStream/buildstream!921

• cc2e6ae5
  by Valentin David at 2018-11-08T12:41:36Z

  Fix bug with root mounted as non-artifact in script plugin.
  
  The issue was introduced by 6ccfab0b1b25990e406446d5cbe5aee83a5e158a.
  

• e578a89f
  by Valentin David at 2018-11-08T13:11:10Z

  Merge branch 'valentindavid/fix-script-install-root' into 'master'
  
  Fix bug with root mounted as non-artifact in script plugin.
  
  Closes #756 and #749
  
  See merge request BuildStream/buildstream!941

• c0a8bb66
  by Angelos Evripiotis at 2018-11-08T15:49:16Z

  contributing: WIP again to be kind to reviewers
  
  As someone coming from GitHub to GitLab, I was pleasantly surprised by
  the 'filter by WIP status' option.
  
  To make sure we get the most out of it, add a guideline to keep the
  filter clean, to reduce reviewer burden.
  

• f7231e90
  by Angelos Evripiotis at 2018-11-08T15:49:16Z

  contributing: non-WIP should always be landable
  
  I was recently surprised that we don't prefer addressing review comments
  in 'fixup!' commits.
  
  Coming from GitHub, I've found that fixup commits make it easier for
  reviewers to see what has changed since their last review. The idea is
  to use '--autosquash' before landing to clean up the history again.
  
  It's a pleasant surprise that it's easy to keep track of what changed in
  merge-requests between pushes, so we can always keep the history clean.
  
  Document this, so folks like me can see the light sooner.
  

• f7643440
  by Angelos Evripiotis at 2018-11-08T15:49:16Z

  contributing: emphasise the 'why' in commits
  
  Explain why it's useful to mention the decisions made for a change and
  provide links for background, e.g. issue numbers.
  
  My general experience of git histories is that folks don't do enough
  explaining, so it's great we have a contributing section for it.
  
  I've found it's easier for folks to stick to rules when they know the
  practical reasons for them, so provide some.
  

• dd5e7b04
  by Angelos Evripiotis at 2018-11-08T16:17:04Z

  Merge branch 'aevri/contributing_gitlab' into 'master'
  
  Add more to GitLab-relevant parts of contributing
  
  See merge request BuildStream/buildstream!935

• fe33e328
  by James Ennis at 2018-11-08T17:54:18Z

  using_config.rst: Add documentation to showing how to impose quotas on the local cache
  
  This patch partially resolves #700
  

• 09faf002
  by James Ennis at 2018-11-08T17:54:18Z

  artifactcache.py: Fix misleading error message when using % cache quota
  
  Due to the changed Exception message, this patch also changes
  the test_parse_size_over_1024T test in misc.py
  

• d153453c
  by Javier Jardón at 2018-11-08T18:22:54Z

  Merge branch 'jennis/quota_declaration_fix' into 'master'
  
  Add local cache expiry documentation and fix misleading error message when specifying a percentage cache quota
  
  Closes #700
  
  See merge request BuildStream/buildstream!939

• 7738f6df
  by Daniel Silverstone at 2018-11-09T11:25:44Z

  plugin.py: Redirect DEBUG from `__del__` to `sys.stderr`
  
  Previously, debug messages from `Plugin.__del__()` went to stdout
  which was a problem for the test suite in extremely rare and hard to
  predict circumstances.  This corrects that by sending the message to
  stderr where all the rest of the messages tend to go.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 62f59eaa
  by Valentin David at 2018-11-09T13:30:06Z

  Merge branch 'danielsilverstone-ct/plugin-destroy-debug-to-stderr' into 'master'
  
  plugin.py: Redirect DEBUG from `__del__` to `sys.stderr`
  
  See merge request BuildStream/buildstream!944

• 35ec6b6a
  by Jim MacArthur at 2018-11-09T14:58:22Z

  _casbaseddirectory.py: Enable direct CAS-to-CAS import.
  
  Fixes issue #574.
  

• b93100ec
  by Jim MacArthur at 2018-11-09T14:58:22Z

  _casbaseddirectory: Fix order of list_modified_paths
  
  This will now return paths in the same order as list_relative_paths.
  

• 78691fa0
  by Jim MacArthur at 2018-11-09T14:58:22Z

  virtual_directory_import.py: New test
  

• 7ce6581b
  by Jim MacArthur at 2018-11-09T15:25:46Z

  Merge branch 'jmac/cas_to_cas_v2' into 'master'
  
  Direct CAS-to-CAS import
  
  Closes #574
  
  See merge request BuildStream/buildstream!911

• 6f4351ec
  by Benjamin Schubert at 2018-11-09T16:53:42Z

  source.py: don't let StopIteration propagate to silence() contextmanager
  
  As per PEP 0479 (https://www.python.org/dev/peps/pep-0479/),
  StopIteration thrown in context managers are not valid starting from
  Python 3.7.
  

• e6e03451
  by richardmaw-codethink at 2018-11-12T11:05:15Z

  Merge branch 'bschubert/fix-silence-stopiteration-pep-0479' into 'master'
  
  source.py: don't let StopIteration propagate to silence() contextmanager
  
  See merge request BuildStream/buildstream!945

• ae5ccd76
  by Phillip Smyth at 2018-11-12T14:57:51Z

  _project.py: Validate nodes early in Project._load
  
  _project.py: Added validate_nodes() helper function to prevent duplicate lists
  
  element-path was being used before node validation resulting in uncaught errors
  

• 56a54161
  by Phillip Smyth at 2018-11-12T14:57:51Z

  tests/frontend: Add test for invalid element-path
  

• e55a9703
  by Jürg Billeter at 2018-11-12T15:44:45Z

  Merge branch 'element-path_not_validated' into 'master'
  
  Element path not validated before use
  
  See merge request BuildStream/buildstream!937

• c87d6e45
  by Valentin David at 2018-11-14T11:43:33Z

  Run tests on aarch64
  
  Fixes #755
  

• 16b01489
  by Valentin David at 2018-11-14T11:43:33Z

  Disable cachekey tests on other architectures than x86_64
  

• 85046b29
  by Valentin David at 2018-11-14T11:43:33Z

  Disable tests on example on other architectures than x86_64
  

• dd5a073b
  by Valentin David at 2018-11-14T11:43:33Z

  Add support for aarch64 in a test
  

• c5d72ae7
  by Valentin David at 2018-11-14T11:52:53Z

  Test aarch64 only overnight
  

• 264a57f6
  by Javier Jardón at 2018-11-14T12:22:03Z

  Merge branch 'valentindavid/integration-tests-aarch64' into 'master'
  
  Run tests on aarch64
  
  Closes #369 and #755
  
  See merge request BuildStream/buildstream!948

• da735e56
  by Richard Maw at 2018-11-14T13:30:34Z

  _platform/linux.py: Move get_bwrap_version into _site.py
  
  It's inconvenient to have to create a Linux platform to parse the bwrap version
  and we want to get the version in a consistent manner.
  

• e7633500
  by Richard Maw at 2018-11-14T13:30:34Z

  buildstream/sandbox/_sandboxbwrap.py: Distinguish sandbox failure from command failure
  
  If `bwrap` fails to set up the sandbox and start the payload command
  it won't write an exit-code in --json-status-fd,
  so we can report if it was a sandboxing failure if we don't get exit-code status
  and a payload command failure if we do and it's non-zero.
  
  Closes https://gitlab.com/BuildStream/buildstream/issues/286
  

• e9e08823
  by Richard Maw at 2018-11-14T13:30:34Z

  tests/testutils/site.py: Check for bwrap supporting --json-status-fd
  

• 90ca007e
  by Richard Maw at 2018-11-14T13:30:34Z

  tests/integration/sandbox-bwrap.py: Test distinguishing sandbox exit code from command
  

• 327b19dd
  by richardmaw-codethink at 2018-11-14T13:59:16Z

  Merge branch 'richardmaw/distinguish-sandboxing-build-fail' into 'master'
  
  Distinguish between bubblewrap sandboxing failure and command failure
  
  Closes #286
  
  See merge request BuildStream/buildstream!868

• 082b3811
  by Angelos Evripiotis at 2018-11-15T19:48:12Z

  contributing: add guidance on unit tests
  
  Decrease uncertainty around whether unit tests are welcome in the
  project or not.
  

• b5b79056
  by Jürg Billeter at 2018-11-15T20:24:58Z

  Merge branch 'aevri/unit_tests' into 'master'
  
  contributing: add guidance on unit tests
  
  See merge request BuildStream/buildstream!943

• 199bfff1
  by Tom Pollard at 2018-11-17T13:07:59Z

  Add cli main and user config option for 'pull-buildtrees' context.
  
  _context.py: Add pull_buildtrees global user context, the default
  of which is set to False via the addition of pull-buildtrees to
  userconfig.yaml cache group.
  
  _frontend/app.py & cli.py: Add --pull-buildtrees as a bst main
  option, which when passed will override the default or user defined
  context for pull_buildtrees.
  
  tests/completions/completions.py: Update for the added flag.
  

• 454dcaf1
  by Tom Pollard at 2018-11-17T13:07:59Z

  Don't pull artifact build trees by default.
  
  The addition of cached build trees being included in element
  artifacts has led to mostly redundant download overheads when
  pulling from a remote artifact server. As such the default behaviour
  of pull shouldn't fetch the build tree object if available.
  
  element.py: extend relevant pull logic for specified subdir
  consideration and ensure push logic does not lead to partial
  artifact pushes. Change assumption that the buildtree can be
  extracted if artifact is cached. __cached_buildtree() and
  __pull_directories() helpers added.
  
  _artifactcache/: artifactcache.py & cascache.py inclusion of
  helper functions for subdir artifact checking & fetching, fetch
  logic extended to only pull required artifact directories.
  extract & checkout updated to handle full/partial operation.
  
  tests/: addition of integration test pullbuildtrees.py,
  buildtrees.py adapted cli options, testutils/artifactshare.py
  has_artifact changed to return artifact digest on true condition.
  

• 0d309478
  by Jürg Billeter at 2018-11-17T13:49:58Z

  Merge branch 'tpollard/494' into 'master'
  
  Don't pull artifact buildtrees by default
  
  See merge request BuildStream/buildstream!786

• 27932739
  by Benjamin Schubert at 2018-11-19T09:46:39Z

  plugins/sources/pip.py: also look for python version named "python"
  
  In some virtual environments, we might not have the binary with the
  exact python version, which would lead the current pip plugin to
  fail to find a working pip version.
  
  This adds "python" at the start of the list of valid python versions
  

• ea2de561
  by Benjamin Schubert at 2018-11-19T10:19:24Z

  Merge branch 'bschubert/fix-pip-python' into 'master'
  
  plugins/sources/pip.py: also look for python version named "python"
  
  Closes #758
  
  See merge request BuildStream/buildstream!942

• f23b6031
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  tests/frontend/buildtrack.py: check for success after invocation
  
  Previously the result of the first call in test_build_track was not
  checked to be successful, which would lead to potential erros later on
  that were hard to debug.
  
  Let's check it everytime
  

• a6defc0b
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  Fix os.rename in git source element to correctly handle error codes
  
  According to the documentation
  (https://www.unix.com/man-page/POSIX/3posix/rename/), when the directory
  already is there, either EEXIST or ENOTEMPTY could be thrown.
  
  Previously only ENOTEMPTY was checked.
  
  Done:
    - Separated the move into its own function
    - Check for both errors
    - Create unit tests for it, covering most test cases
  

• 88089d2d
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  Extract atomic move function to utils.py
  
  Moving atomically a file/directory can be tricky since different
  errors might be raised for the same underlying problem.
  
  Having a utility function to reduce this discrepancies will help
  in ensuring we have correct behavior
  

• fd9e46be
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  cascache.py: use move_atomic instead of manual error checking
  

• d32e0b83
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  sources/pip.py: Use move_atomic instead of manual os.rename
  
  This uses move_atomic insteand of the manual os.rename and manual error
  checking and throws a SourceError for consistency with other modules.
  

• 6f837118
  by Benjamin Schubert at 2018-11-19T12:22:40Z

  Merge branch 'bschubert/fix-atomic-move-git-repo' into 'master'
  
  Fix os.rename in git source element to correctly handle error codes
  
  See merge request BuildStream/buildstream!938

• 30b72244
  by Chandan Singh at 2018-11-19T12:35:01Z

  buildstream/_pipeline.py: Simplify error message for assert_consistent()
  
  Currently, `Pipeline.assert_consistent()` prints an error message like
  the one below when certain elements have inconsistent sources:
  
  ```
  ...
        Element: hello.bst is inconsistent
          Source tar source at hello.bst [line 16 column 2] is missing ref
  ...
  ```
  
  Drop the word "Source" from the beginning of the message as
  "Source <source-kind> source ..." is kind of awkward to read.
  

• 76c5d2f8
  by Chandan Singh at 2018-11-19T12:35:01Z

  Add `bst source-checkout` command
  
  As discussed in
  https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
  add `bst source-checkout` command. This will allow users to checkout
  sources for a given target.
  
  * _frontend/cli.py: Add source-checkout command
  * _pipeline.py: Add assert_sources_cached() method
  * _stream.py: Add source_checkout method, abstract out
    __check_location_writable() method that used to part of checkout()
  

• 9f629638
  by Chandan Singh at 2018-11-19T13:03:36Z

  Merge branch 'chandan/source-checkout' into 'master'
  
  Add `bst source-checkout` command
  
  See merge request BuildStream/buildstream!820

• bc827cc8
  by Abderrahim Kitouni at 2018-11-19T14:55:15Z

  plugins/elements/cmake.yaml: always specify variable types
  
  cmake sometimes misinterprets relative paths as relative to the current directory
  if this is not specified. See freedesktop-sdk/freedesktop-sdk#431
  
  adjust tests/format/variables.py accordingly.
  

• 5fbc5f41
  by Valentin David at 2018-11-19T15:31:05Z

  Merge branch 'abderrahim/cmake-variable-types' into 'master'
  
  plugins/elements/cmake.yaml: always specify variable types
  
  Closes #761
  
  See merge request BuildStream/buildstream!947

• 06e28860
  by Benjamin Schubert at 2018-11-19T15:52:24Z

  Don't cache sandbox errors
  
  Sandbox errors (like missing host tools) are dependent on the host
  system and rarely on what is actually done.
  
  It is therefore better to not cache them as they are subject to
  change between two runs.
  
  Also add test to ensure sandbox failure are not cached
  

• dd36cfbc
  by Benjamin Schubert at 2018-11-19T15:52:24Z

  element.py: Only cache on element error or success
  
  Reverse the way we were handling caching by only caching
  when we know it's a problem with the build or the build is a
  success in order not to cache transient errors.
  

• eac7274d
  by Benjamin Schubert at 2018-11-19T16:29:07Z

  Merge branch 'bschubert/dont-cache-errors-from-host-tools' into 'master'
  
  Don't cache sandbox failures
  
  Closes #727
  
  See merge request BuildStream/buildstream!895

• 625dfe1f
  by Chandan Singh at 2018-11-19T16:41:17Z

  NEWS: Add entry for the new source-checkout command
  
  `bst source-checkout` command was recently added in
  https://gitlab.com/BuildStream/buildstream/merge_requests/820. Add a
  NEWS entry for the new command.
  

• d4f12184
  by Chandan Singh at 2018-11-19T17:15:10Z

  Merge branch 'chandan/source-checkout-news' into 'master'
  
  NEWS: Add entry for the new source-checkout command
  
  See merge request BuildStream/buildstream!960

• ba6c96e8
  by Richard Maw at 2018-11-19T19:54:30Z

  CasBasedDirectory: Change constructor to take a CASCache instead of a Context
  
  The Context was only used to obtain a reference to the CASCache
  and set the unused cas_directory field.
  

• 8a0dc3a3
  by Jürg Billeter at 2018-11-19T19:54:30Z

  tests/testutils/artifactshare.py: Do not create a fake context
  
  The fake context did not set the cache quota, triggering an error. With
  CASCache now separate from ArtifactCache, we can instantiate a CASCache
  without context.
  

• 8722aced
  by Jürg Billeter at 2018-11-19T19:54:30Z

  tests/plugins/pipeline.py: Load context default values
  

• fc56ffa4
  by Jürg Billeter at 2018-11-19T19:54:30Z

  _context.py: Drop duplicated default values for user configuration
  
  The default values are in userconfig.yaml, together with the
  documentation. The default values should not be duplicated in
  _context.py.
  

• 3f663d82
  by Jürg Billeter at 2018-11-19T20:36:57Z

  Merge branch 'juerg/context-default-values' into 'master'
  
  _context.py: Drop duplicated default values for user configuration
  
  See merge request BuildStream/buildstream!953

• b498cce7
  by Angelos Evripiotis at 2018-11-20T11:17:38Z

  _yaml: document node_get()'s default_value arg
  

• eb2d376f
  by Angelos Evripiotis at 2018-11-20T11:17:38Z

  _context: allow 'terminate' for scheduler.on-error
  
  Enable this option of 'terminate', which is mentioned in userconfig.yaml
  and handled in _frontend/app.py:_handle_failure(). It appears to have
  been left out of the valid_actions as an oversight.
  
  Originally introduced in
  https://gitlab.com/BuildStream/buildstream/commit/2622d5da9bd9fefd87436613d6e3e9770fdd0f28
  

• b81c4333
  by Angelos Evripiotis at 2018-11-20T11:17:38Z

  _context: refactor, extract _node_get_option_str
  
  Use a new helper function to simplify working with nodes that can only
  accept certain strings. This will be used when adding the prompt.*
  config options.
  
  In later work we can see if this function would be useful elsewhere, and
  could be added to '_yaml.py'.
  

• 27ca6593
  by Angelos Evripiotis at 2018-11-20T11:19:33Z

  Add prompt.auto-init buildstream.conf option
  
  Provide an option in buildstream.conf to disable the 'Would you like to
  ...' prompt when we cannot resolve a project.
  
  Some users prefer not to be interrupted by such prompts, so pave the way
  to creating options to disable all those that might get in the way.
  
  Follow the example of the advice.* options 'git-config', and create a
  namespace for these UI options grouped by behaviour, rather than an
  over-reaching 'ui.*' namespace. In later work perhaps we'll also add
  'advice.*' options.
  
  Add a NEWS item for this.
  

• 7ae3a3d2
  by Angelos Evripiotis at 2018-11-20T11:43:49Z

  Add prompt.workspace-... options
  
  Provide options in project.conf to disable the 'Are you sure ...'
  prompts when making destructive changes:
  
  - Add prompt.really-workspace-close-remove-dir
  - Add prompt.really-workspace-reset-hard
  
  Add a NEWS item for these.
  

• 8071c00c
  by Angelos Evripiotis at 2018-11-20T12:12:11Z

  Merge branch 'aevri/prompt-config' into 'master'
  
  Add 'prompt.*' config options to buildstream.conf
  
  See merge request BuildStream/buildstream!887

• fd41b0b5
  by Raoul Hidalgo Charman at 2018-11-20T16:32:10Z

  _sandboxremote.py: Add sigterm handler that sends CancelOperation
  

• c6306b88
  by Jürg Billeter at 2018-11-20T16:59:45Z

  Merge branch '725-job-cancellation-on-remote-builds' into 'master'
  
  _sandboxremote.py: Add sigterm handler that sends CancelOperation
  
  Closes #725
  
  See merge request BuildStream/buildstream!900

• 9fa8a881
  by James Ennis at 2018-11-20T17:00:42Z

  _yaml.py: Fix incorrect error message
  
  This patch ensures that we receive an appropriate error
  message if we specify an absolute path that leads within
  the project.
  

• fb8860db
  by Jürg Billeter at 2018-11-20T17:32:58Z

  Merge branch 'jennis/less_restrictive_pathways' into 'master'
  
  Fix warning when using an absolute path which exists inside the project
  
  See merge request BuildStream/buildstream!956

• 87751350
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  gitlab-ci.yml: Use CI_PROJECT_DIR instead of $(pwd) for cache
  

• 8668df9f
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  Export variables globally for less repetition.
  
  This doesn't change the test behavior and enable for better
  composability of tests
  

• cc6a7f0d
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  gitlab-cy.yml: make linux-tests-template more composable.
  
  This will allow for further de-duplication of work.
  

• eb37dd89
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  gitlab-ci.yml: remove unused artifacts for unix tests
  
  The unix-logs is always empty, there is therefore no reasons of keeping
  it there
  

• e030b48e
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  gitlab-ci.yml: use template for unix tests
  
  This makes all tests use the same template, which makes reasoning
  about them simpler
  

• 57e6358f
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  Ensure tests requiring bubblewrap are correctly marked
  

• e70dd1f8
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  gitlab-ci.yml: Add linux without brwap/ostree to ensure tests behave
  
  This allows us to make sure that tests behave nicely while missing brwap
  and ostree.
  

• f7cb4386
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  gitlab-ci.yml: Extract test command as a variable
  
  This removes the need of having it synchronized in multiple places
  

• 7f316a76
  by Benjamin Schubert at 2018-11-20T18:41:03Z

  Merge branch 'bschubert/add-fedora-nobwrap-tests' into 'master'
  
  Add tests for fedora with no bubblewrap available
  
  See merge request BuildStream/buildstream!949

• a50c598e
  by Benjamin Schubert at 2018-11-21T09:24:15Z

  utils.py: Ensure move_atomic booleans are keyword only arguments
  
  This is to be coherent with the rest of the codebase
  

• b2f604b8
  by Benjamin Schubert at 2018-11-21T09:24:15Z

  utils.py: improve documentation of move_atomic
  

• daca0c01
  by Benjamin Schubert at 2018-11-21T09:52:42Z

  Merge branch 'bschubert/mr938-comments' into 'master'
  
  Followup on MR 938, addressing additional comments
  
  See merge request BuildStream/buildstream!958

• 050cd1bd
  by Tom Pollard at 2018-11-21T11:37:48Z

  tests/plugin/pipeline.py: Avoid using host user conf
  
  create_pipeline was passing no config parameter to context.load,
  leading to _context.py loading in the host's buildstream user conf
  when running tests locally, potentially leading to unwanted errors.
  

• 58353aad
  by Jürg Billeter at 2018-11-21T14:29:49Z

  Merge branch 'tpollard/pipelinehostconfig' into 'master'
  
  tests/plugin/pipeline.py: Avoid using host user conf
  
  See merge request BuildStream/buildstream!961

• 2022c3e8
  by William Salmon at 2018-11-21T15:53:03Z

  tests/frontend/workspace.py: Refactor support function
  
  The Refactor now makes it possible to test for the forth coming multi
  element workspace open commands and to test for failure.
  

• 7bdd44b2
  by William Salmon at 2018-11-21T15:53:03Z

  Workspace CLI update
  
  This is to update the workspace CLI to as agreed on the mailing list
  https://mail.gnome.org/archives/buildstream-list/2018-September/msg00046.html
  
  This patch also introduces the default workspace directory.
  

• 6612e708
  by William Salmon at 2018-11-21T15:53:03Z

  Added a news item for workspace open changes
  

• 4d6e2cd3
  by Will Salmon at 2018-11-21T16:20:26Z

  Merge branch 'willsalmon/defaultWorkspaces' into 'master'
  
  Updated Workspace CLI
  
  See merge request BuildStream/buildstream!897

• dddf0eaa
  by Valentin David at 2018-11-22T11:12:53Z

  .gitlab-ci.yml: Bump version of fd.o SDK and bst_external in overnight tests
  

• abef70fe
  by Javier Jardón at 2018-11-22T11:41:31Z

  Merge branch 'valentindavid/overnight-tests-bump-fdo-sdk-version' into 'master'
  
  Bump version of fd.o SDK and bst_external in overnight tests
  
  See merge request BuildStream/buildstream!962

• 5c208698
  by Jim MacArthur at 2018-11-26T10:23:31Z

  Docs: Add remote execution architecture documentation.
  

• 5af455a4
  by Jim MacArthur at 2018-11-26T10:55:22Z

  Merge branch 'jmac/execution-architecture-docs' into 'master'
  
  Docs: Add remote execution architecture documentation.
  
  See merge request BuildStream/buildstream!923

• 4ca892e3
  by Jim MacArthur at 2018-11-27T09:30:13Z

  tests: Avoid hangs due to exceptions in subprocesses
  

• d09a18b8
  by Jim MacArthur at 2018-11-27T10:31:07Z

  Merge branch 'jmac/fix-test-hangs-2' into 'master'
  
  Avoid hanging artifact cache tests
  
  See merge request BuildStream/buildstream!964

• df60258a
  by Jürg Billeter at 2018-11-27T12:11:24Z

  source.py: Add optional _get_local_path() method
  

• 42809933
  by Jürg Billeter at 2018-11-27T12:11:24Z

  local.py: Implement _get_local_path()
  

• eb5b10f7
  by Jürg Billeter at 2018-11-27T12:11:24Z

  _loader.py: Use Source._get_local_path() for subprojects
  
  This eliminates unneeded copies of the project directory for junctions
  with a single local source.
  

• b2ea208c
  by Jürg Billeter at 2018-11-27T12:50:46Z

  Merge branch 'juerg/local-junctions' into 'master'
  
  Optimization for local junctions
  
  See merge request BuildStream/buildstream!290

• 223c0073
  by Jürg Billeter at 2018-11-27T13:41:09Z

  tests/integration/manual.py: Add test for command logging
  

• 38d16c62
  by Jürg Billeter at 2018-11-27T13:41:09Z

  sandbox: Deduplicate code to process cwd, env, and command arguments
  

• 85f5b574
  by Jürg Billeter at 2018-11-27T13:41:09Z

  sandbox/sandbox.py: Add SandboxFlags.NONE
  
  This improves readability.
  

• bdd6925f
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Use SandboxFlags.NONE instead of 0
  

• 69005c76
  by Jürg Billeter at 2018-11-27T13:41:09Z

  scriptelement.py: Use SandboxFlags.NONE instead of 0
  

• f1767de2
  by Jürg Billeter at 2018-11-27T13:41:09Z

  sandbox/sandbox.py: Add command batching API
  
  This adds the batch() context manager.
  

• 87f7c99a
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Pass element object to sandbox
  
  This allows the sandbox to output messages using the plugin id.
  

• 024d584d
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Cache artifact also on SandboxCommandError
  

• bb7c3213
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Defer workspace.prepared setting in case of pending commands
  

• 04f05b1a
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Support command batching for integration commands
  

• 7827ac50
  by Jürg Billeter at 2018-11-27T13:44:49Z

  element.py: Add batch_prepare_assemble() method
  
  This allows batching of all sandbox commands aross prepare() and
  assemble().
  

• 63db90b2
  by Jürg Billeter at 2018-11-27T13:44:49Z

  buildelement.py: Support batching for integration and build commands
  

• 73dec86e
  by Jürg Billeter at 2018-11-27T13:44:49Z

  scriptelement.py: Support batching for integration and build commands
  

• 79442e0a
  by Jürg Billeter at 2018-11-27T13:44:49Z

  compose.py: Support batching for integration commands
  

• 6018b3bd
  by Jürg Billeter at 2018-11-27T13:44:49Z

  sandbox/_sandboxremote.py: Implement command batching
  
  Execute batched commands in a single shell script.
  

• ec3d4486
  by Jürg Billeter at 2018-11-27T13:44:49Z

  NEWS: Add entry for command batching
  

• 127d332f
  by Jürg Billeter at 2018-11-27T14:16:50Z

  Merge branch 'juerg/command-batching' into 'master'
  
  Command batching
  
  Closes #675
  
  See merge request BuildStream/buildstream!915

• d5b5538d
  by Jim MacArthur at 2018-11-27T15:32:19Z

  artifactcache: Move ArtifactCacheSpec code into CASRemoteSpec.
  
  There is nothing in ArtifactCacheSpec that's actually specific to
  artifacts, so I've made it a CAS class so we can use the same
  spec for remote execution.
  

• f224d4cb
  by Jim MacArthur at 2018-11-27T15:32:19Z

  artifactcache.py, _context.py: Move CASCache object into context
  
  Since the artifact cache and remote execution share the same
  local CAS store, they should share the same CASCache object. Moving
  this into context allows us to do this.
  

• 926c1446
  by Jim MacArthur at 2018-11-27T15:33:34Z

  _sandboxremote: Remove references to artifact cache.
  
  Uses the new config options introduced earlier to specify remote
  exec storage and execution service.
  

• d7cdb03e
  by Jim MacArthur at 2018-11-27T15:33:36Z

  Documentation: Update to show new config options
  

• 95d8244f
  by Jim MacArthur at 2018-11-27T15:33:36Z

  tests/sandboxes/remote-exec-config.py: New test.
  
  Tests that we get a sensible error message when we specify incomplete
  or wrong information in the configuration for remote execution.
  

• 7f2f62eb
  by Jim MacArthur at 2018-11-27T15:33:36Z

  artifactcache.py: Remove unused verify_digest_pushed
  

• c55ca3a5
  by Jim MacArthur at 2018-11-27T15:33:36Z

  Docs: Correct URLs in artifact cache example
  
  The port number must come after the hostname and before the path.
  The path is ignored by CASCache anyway, so including it is misleading.
  

• 2a8a3b19
  by Jim MacArthur at 2018-11-27T18:17:03Z

  Merge branch 'jmac/remote_execution_split' into 'master'
  
  Split remote execution from artifact cache
  
  Closes #750
  
  See merge request BuildStream/buildstream!946

• 499c70fd
  by Valentin David at 2018-11-28T09:11:21Z

  setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52
  

• 3513580c
  by Valentin David at 2018-11-28T14:02:32Z

  Merge branch 'valentindavid/ruamel_0_15_41' into 'master'
  
  setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52
  
  See merge request BuildStream/buildstream!975

• 26cdee08
  by Valentin David at 2018-11-28T14:29:52Z

  Use f_bavail to query available space. Not f_bfree.
  
  f_bfree space might not be usable. In practice we see failures in big
  disks because f_bfree is over 2GB and f_bavail is 0. We get ENOSPC if
  writing on disk then.
  

• 58ca298f
  by Valentin David at 2018-11-28T14:29:52Z

  Make cache clients not fail when a blob is not available.
  
  We plan to make cache incomplete. That is some blobs are missing.  For
  most of cases we will delete references when requested if they are
  incomplete. But there will be corner cases where objects are removed
  after the reference is requested.
  

• 227fa26d
  by Valentin David at 2018-11-28T14:29:52Z

  Fix type error in RPC messages
  

• 5ef19a0b
  by Valentin David at 2018-11-28T14:29:52Z

  Avoid copying temporary file when adding object to CAS in server.
  
  The file is already a temporary file and does not need copy.  ENOSPC
  is thrown during that copy in issue #609.
  
  Fixes #678.
  

• 8d2946ff
  by Valentin David at 2018-11-28T14:29:52Z

  "Fallocate" object disk space to avoid getting NOSPC error when writing
  
  This locks the temporary object file so that cleanup does not need to
  be done for every write.
  

• b587953a
  by Valentin David at 2018-11-28T14:29:52Z

  Update mtimes of objects for requested references.
  
  This also remove references when some objects are missing. This is in
  preparation for the move from reference to object garbage collection.
  

• a64f667d
  by Valentin David at 2018-11-28T14:29:52Z

  Move cas server from ref-based to object-based garbage collection.
  

• 353b90dd
  by Valentin David at 2018-11-28T14:29:52Z

  Cleanup cache in cas server more agressively
  
  When there is less than 2GB left, it cleans up have 10GB available.
  These values are configurable.
  

• ba9afa98
  by Valentin David at 2018-11-28T14:29:52Z

  Lock cache cleanup in cas server
  
  Cleaning up in parallel might slow down the cleaning process
  

• 9a458402
  by Valentin David at 2018-11-29T08:47:40Z

  Merge branch 'valentindavid/cache_server_fill_up' into 'master'
  
  Fix cleanup of cache in server when disk is full
  
  Closes #678
  
  See merge request BuildStream/buildstream!830

• 47b7a9ba
  by Jim MacArthur at 2018-11-29T09:58:48Z

  _sandboxremote.py: Remove unnecessary tests.
  
  push_message and push_directory will both raise assertion failures if
  they fail to send the digest to the server. Checking the digest
  returned by each only tests that the content was hashed locally, which
  cannot reasonably fail.
  

• 4a8d0565
  by Jim MacArthur at 2018-11-29T10:42:31Z

  Merge branch 'jmac/no-verify-digests' into 'master'
  
  _sandboxremote.py: Remove unnecessary tests.
  
  Closes #786
  
  See merge request BuildStream/buildstream!976

• 2e78e0d1
  by Valentin David at 2018-11-29T13:18:17Z

  Add support for .netrc in remote/tar/zip plugins
  
  Fixes #723.
  

• a6144100
  by Valentin David at 2018-11-29T13:18:17Z

  .gitlab-ci.yml: Update image versions where forgotten
  

• f6c184f5
  by Valentin David at 2018-11-29T16:15:56Z

  Merge branch 'valentindavid/netrc' into 'master'
  
  Add support for .netrc in remote/tar/zip plugins
  
  Closes #723
  
  See merge request BuildStream/buildstream!908

• 1f31aefc
  by Angelos Evripiotis at 2018-11-30T09:34:38Z

  man/: update with changes since Apr 2018
  
  It seems that the regenerated man pages haven't been committed for some
  time. Add them all in one lump now.
  
  Had to manually strip the version number from the generated files.
  
  Had to manually fix the NAME field of bst.1 to not be 'bst - None'.
  
  Oddly had to invoke click-man directly in order to get the changes to
  be applied, the usual setup.py route mentioned in contributing.rst
  didn't work.
  
  Eyeballed the result of invoking man on each of the commands, seems
  good.
  

• 033a5ad9
  by Jürg Billeter at 2018-11-30T10:09:11Z

  Merge branch 'aevri/update_man' into 'master'
  
  man/: update with changes since Apr 2018
  
  See merge request BuildStream/buildstream!981

• 1123b9a1
  by Tom Pollard at 2018-11-30T13:30:35Z

  _stream.py: Ability to pull missing buildtrees outside of pull/build
  
  Adds helper function _buildtree_pull_required() to determine if a
  pullqueue should be constructed, for commands outside of bst pull
  and build where it is determined that an element's buildtree
  artifact is to be required given the respective semantics and
  config. Utilised in push() to attempt to mitigate skipping the push
  of partial elements without the user having to have preceded it with
  an explicit pull.
  
  cli.py: Add new behaviour to push command description
  
  element.py: Move _cached_buildtree() to be non local private method,
  use _KeyStrength types to reduce duplication.
  
  tests/integration/pullbuildtrees.py also updated to cover this
  use-case.
  

• ad293ed3
  by Jürg Billeter at 2018-11-30T14:10:06Z

  Merge branch 'tpollard/774' into 'master'
  
  _stream.py: Ability to pull missing buildtrees outside of pull/build
  
  Closes #774
  
  See merge request BuildStream/buildstream!978

• 0d634c21
  by Phillip Smyth at 2018-11-30T14:16:36Z

  plugin.py: Add BAD_ELEMENT_SUFFIX warning
  

• d97ef822
  by Phillip Smyth at 2018-11-30T14:16:36Z

  _project.py: Check for fatal_warnings earlier
  

• 59d442e5
  by Phillip Smyth at 2018-11-30T15:41:57Z

  loader: Added file suffix check and helper function
  
  _loader/loader.py: Added Suffix check and warning for named elements and add helper function
  

• 2d6ec07d
  by Phillip Smyth at 2018-11-30T15:41:57Z

  cli.py: no autocomplete on files without .bst
  

• ac0bc4c0
  by Phillip Smyth at 2018-11-30T16:50:35Z

  Added tests for .bst suffix and completions
  
  buildcheckout.py: Add mandatory .bst suffix tests
  completions.py: Add test for fail on invalid suffix
  Added required files for testing
  integration/source-determinism.py: renamed test elements to end with .bst
  loader/__init__.py: initialised context properly with load and a message handler
  

• d4bc662a
  by Phillip Smyth at 2018-11-30T16:50:50Z

  NEWS: Added update about the Mandatory .bst change
  

• 7cea464a
  by Jürg Billeter at 2018-11-30T17:21:37Z

  Merge branch 'mandatory_suffix' into 'master'
  
  Mandatory .bst suffix
  
  See merge request BuildStream/buildstream!967

• cc5ef68c
  by Tristan Van Berkom at 2018-12-03T08:41:42Z

  MANIFEST.in: Include the sources for rebuilding the documentation session HTML
  
  This was previously missing, causing our CI to forget to rebuild the sessions
  when updating the documentation.
  
  This fixes issue #778
  

• 87ba19d7
  by Tristan Van Berkom at 2018-12-03T09:19:54Z

  Merge branch 'tristan/fix-doc-session-rebuilds' into 'master'
  
  MANIFEST.in: Include the sources for rebuilding the documentation session HTML
  
  Closes #778
  
  See merge request BuildStream/buildstream!984

• 1b24148f
  by Chandan Singh at 2018-12-03T09:46:13Z

  Ensure `--deps=none` option works for `bst checkout`
  
  Currently, `bst checkout --deps none` command always produces empty
  output. Fix this issue and add regression test for the same.
  
  - element_enums.py: Add Scope.NONE.
  
  - element.py: Ensure Scope.NONE works correctly in addition to
    Scope.RUN/Scope.ALL in Element.dependencies() and Element.search().
  
  - tests/frontend/buildcheckout.py: Fix tests for `--deps none`.
  
  Fixes #670.
  

• 8ac1e757
  by Chandan Singh at 2018-12-03T09:46:14Z

  Add `--deps build` option to `bst checkout`
  
  As discussed in
  https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
  add `--deps build` option to `bst checkout`. This will allow users to
  checkout the all build dependencies of a given element using a single
  command.
  
  - _frontend/cli.py: Add `--deps build` option for `bst checkout`.
  
  - element.py: Support `deps='build'` in Element._prepare_sandbox().
  
  - tests/frontend/buildcheckout.py: Ensure `--deps build` works as
    expected.
  

• ac60f47a
  by Benjamin Schubert at 2018-12-03T09:46:14Z

  Refactor and simplify _prepare_sandbox for elements
  
  Before we would have a intricate logics with multiple arguments
  that might get ignored.
  
  This simplifies the design and introduces a bool `shell` instead
  of having two different variables concerned about scope
  

• b1430fb5
  by Tristan Van Berkom at 2018-12-03T11:01:54Z

  Merge branch 'chandan/bst-checkout-build' into 'master'
  
  Add `--deps build` option to `bst checkout`
  
  Closes #670
  
  See merge request BuildStream/buildstream!819

• 9022a778
  by Javier Jardón at 2018-12-03T11:14:55Z

  buildstream/plugins/sources/git.py: warn about checkout by default
  
  See #783
  

• 4462892e
  by Tristan Van Berkom at 2018-12-03T11:45:07Z

  Merge branch 'jjardon/doc_git_submodules' into 'master'
  
  git: warn about checkout submodules  by default
  
  See merge request BuildStream/buildstream!983

• 6fcfa452
  by Jonathan Maw at 2018-12-03T13:21:05Z

  _yamlcache.py: Use a project's junction name if present
  
  This is required because if there are projects with the same name
  loaded, that have elements in the same path, with the same contents,
  they are considered the same, despite actually belonging to different
  Projects.
  
  This fixes buildstream issue #795
  

• d0f9c724
  by James Ennis at 2018-12-03T13:21:05Z

  tests/loader/junctions.py: Test that we can build a junction that is referred differently, twice.
  
  Test required to close #795 - We should be able to build a project which refers
  to the same junction twice, but with two different filenames.
  

• f7581026
  by Jürg Billeter at 2018-12-03T16:27:03Z

  Merge branch 'jonathan/fix-identical-element' into 'master'
  
  _yamlcache.py: Use a project's junction name if present
  
  Closes #795
  
  See merge request BuildStream/buildstream!980

• 98c15463
  by Tristan Van Berkom at 2018-12-05T07:04:56Z

  _scheduler/queues: Don't call update state outside of error handling harness
  
  Commit 3fa79d8da, part of an initiative for caching of the failed builds,
  introduced a call to Element._update_state() after a job completes and before
  entering the error handling harness intended for handling plugin raised errors.
  
  Element._update_state() can result in triggering plugin code to
  run, so this is incorrect, and causes raised errors to crash BuildStream
  if they happen here.
  
  After analyzing the code, it appears that this additional call to
  Element._update_state() is unneeded, and was only added because the
  state needs to be updated for a failure as well as a success.
  
  Instead, we now have the BuildQueue call Element._assemble_done()
  unconditionally, regardless of whether the build was successful
  or not, which has the same effect and also reads better.
  
  In addition, added a FIXME comment that we are still conditionally
  updating the artifact cache size from BuildQueue.done() only
  if the build is successful, which is incorrect because failed builds
  also increase the local artifact cache size - to fix this we need
  to communicate the added artifact size through Element._assemble()
  regardless of whether the build succeeded or failed.
  

• 5d77b871
  by Tristan Van Berkom at 2018-12-05T07:37:54Z

  Merge branch 'tristan/refactor-queues-update-state' into 'master'
  
  _scheduler/queues/queue.py: Don't call update state outside of error handling harness
  
  See merge request BuildStream/buildstream!988

• 82513109
  by Valentin David at 2018-12-05T09:47:52Z

  git source plugin: Track git tags and save them to reproduce a minimum shallow repository
  
  Instead of tag information being fetched which can change with time,
  they are tracked and saved in the projects.refs/.bst. Then we re-tag
  automatically the closest tag so that `git describe` works and is
  reproducible.
  
  This new feature is opt-in with the new `track-tags` configuration,
  and must be used to fix modules which are broken by our new policy
  of omitting the `.git/` repository when staging git sources.
  
  This fixes issue #487
  

• ec7b62c3
  by Valentin David at 2018-12-05T09:50:15Z

  NEWS: Add news about broken 'git describe' by default.
  

• 761e7570
  by Tristan Van Berkom at 2018-12-05T10:23:49Z

  Merge branch 'valentindavid/git_describe_tracking' into 'master'
  
  Track of git tags and save them to reproduce minimum shallow repository
  
  Closes #487
  
  See merge request BuildStream/buildstream!906

• 2f5966bd
  by Raoul Hidalgo Charman at 2018-12-05T10:37:10Z

  platform.py: Add mapping of uname to OS-independent values
  

• 1ad35fcd
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  _config.py: Use os and architecture settings
  
  In element, platform asks for host os and architecture to default to when
  SandboxConfig is initialised.
  This changes element cache keys so those have been updated in the tests.
  

• 9f75c274
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  optionarch.py: update to use same arch names as SandboxConfig
  
  Also update tests to be consistent with this
  

• a825096c
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  _sandboxremote.py: Add os and arch to command uploaded
  
  These are contained within the platform field.
  

• 8e09a81c
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  optionos.py: Add new option for OS configuration
  

• defec112
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  optionos.py: Add tests for OptionOS
  
  Same format as OptionArch tests, but with OS's.
  

• e5d8c7d8
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  _platform: Add checks for os and architecture
  

• b0603fb0
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  Sandbox: use linux32 for x86-32 builds on x86-64 machines
  
    o _platform/linux.py: Add linux32 flag to send to sandbox bwrap when the
      build arch is x86-32 and the machines arch is x86-64 or similarly with
      aarch32 and aarch64.
    o sandbox/_sandboxbwrap.py: Use flag passed to start bwrap command with
      linux32 if set.
  

• f4d76996
  by Raoul Hidalgo Charman at 2018-12-05T14:58:49Z

  format_project.rst: Update documentation for changes to OptionArch
  

• 81163a79
  by Raoul Hidalgo Charman at 2018-12-05T14:58:53Z

  format_declaring.rst: Update documentation for sandbox options
  

• 23c27580
  by Raoul Hidalgo Charman at 2018-12-05T14:58:53Z

  format_project.rst: Add docs for OptionOS
  

• eb1ed410
  by Jürg Billeter at 2018-12-05T15:28:52Z

  Merge branch 'raoul/775-execution-environment-reqs' into 'master'
  
  Execution environment reqs
  
  Closes #775
  
  See merge request BuildStream/buildstream!969

• 9ef1a8a2
  by Raoul Hidalgo Charman at 2018-12-05T15:32:22Z

  utils.py: Add message digest function
  
  Get cascache.py to use this
  

• 4a56bf2a
  by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

  _sandboxremote.py: Add checks to action cache before attempting to push.
  
  Stops unneccesary pushing of builds that have already been built, just checks
  the action cache to begin with.
  
  Fixes #628
  

• 3641b6f6
  by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

  _sandboxremote.py: Add action-cache-service endpoint
  
  The endpoint may not necessarily be the same as execution-service endpoint.
  
  Part of #628
  

• 9d77351f
  by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

  format_project.rst: Add docs for action-cache-service
  
  Part of #628
  

• a53d6d1f
  by Raoul Hidalgo Charman at 2018-12-05T16:36:00Z

  Merge branch 'raoul/628-RE-flow-optimisation' into 'master'
  
  Remote-execution client flow optimisation
  
  Closes #628
  
  See merge request BuildStream/buildstream!982

• 4688b2c9
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  tests/sources/git.py: Refactor ref-not-in-track test to use parameterization
  
  Instead of duplicating the whole test body, run it twice while
  checking for a warning or an error depending on the parameter.
  

• 03028c7f
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  types.py: Moving CoreWarnings to the types.py module
  
  This also ensures it is exposed via the main buildstream __init__.py
  file, fixes the imports of CoreWarnings, and adjusts the documentation
  links to point to the right place.
  

• 3b7f8df8
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  Source: Fix typo in documentation, and make it a link to `bst track`
  
  Now the commands have documentation linkability so we can link to
  them when referring to them anywhere in the docs. This reference
  was previously referring to a bogus non-existent `build-stream track`,
  maybe this dates way back to the days before we named the frontend `bst`.
  

• e5c0f067
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  git source plugin: Fixing documentation linking and typo
  
  Fixes the configurable warning part of this to properly link
  to the project.conf documentation describing what configurable
  warnings actually are.
  
  Also fix the plugin raise the actual configurable warning which
  it advertizes that it raises; which is 'inconsistent-submodule',
  not 'inconsistent-submodules'.
  

• 205c27d8
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  git source plugin: Rename symbol for consistency and readability
  
  Change 'INCONSISTENT_SUBMODULE' for 'WARN_INCONSISTENT_SUBMODULE' to
  make things a bit more readable and comprehensive.
  

• bea4d4f5
  by Tristan Van Berkom at 2018-12-06T09:37:50Z

  Merge branch 'tristan/misc-cleanup' into 'master'
  
  Misc cleanups
  
  See merge request BuildStream/buildstream!993

• b1d2f001
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  source.py: Add new delegate method validate_cache()
  
  This is guaranteed to be called only once for a given session once the
  sources are known to be Consistency.CACHED, if source tracking is enabled
  in the session for this source, then this will only be called if the
  sources become cached after tracking completes.
  

• ee7fc47f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  git source plugin: Implementing submodule warnings
  
   o Unlisted submodule warning
  
     Now the git plugin will issue a configurable warning if a submodule
     exists and is used (checking out the submodule is not disabled),
     but is not specified in the source configuration.
  
   o Invalid submodule warning
  
     Now the git source plugin will issue a warning if the configuration
     specified a submodule which does not exist in the underlying git repository.
  
  As a side effect, this patch also changes the flow control of the git
  plugin such that submodules which are explicitly set to not be
  checked out, are also not fetched but instead ignored completely.
  

• 5fc9a1da
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  git source plugin: Emmit the ref-not-in-track warning from validate_cache()
  
  Now that we have Source.validate_cache(), this is a better place to emmit
  the ref-not-in-track warning, since it will be emmitted at the earliest
  opportunity and not only at Source.stage() or Source.init_workspace().
  
  This also allows us to remove the `self.tracked` local state, and cleanup
  some convoluted calling paths, removing some unnecessary parameters from
  the usual codepaths and making the plugin overall more readable.
  

• 50cb2706
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/testutils/repo/git.py: Adding remove_path() helper
  
  A function for removing paths from a git repo, can also
  be used to remove submodules.
  

• f139ca8f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Testing the git:unlisted-submodule warning
  
   o Test that it is not triggered in show before fetch, because we
     don't know about the unlisted submodules yet
  
   o Test that it is triggered by a fetch command
  
   o Test that it is triggered by `show` after having completed a
     fetch command, since now we have the repository and know about
     the unlisted submodule
  
   o Test all of this under warning or error conditions (parameterized
     for fatal-warnings)
  

• afe1aa9f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Testing the git:invalid-submodule warning
  
     o Test that it is not triggered in show before fetch, because we
       don't know the submodules yet so we cannot know if they are
       valid or not.
  
     o Test that it is triggered by a fetch command
  
     o Test that it is triggered by `show` after having completed a
       fetch command, since now we have the repository and know which
       specified submodules are invalid
  
     o Test all of this under warning or error conditions (parameterized
       for fatal-warnings)
  

• 57300201
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Test unlisted submodules warning appearing after track
  

• a0297625
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Test invalid submodules warning appearing after track
  

• 60e96781
  by Tristan Van Berkom at 2018-12-06T15:14:37Z

  Merge branch 'tristan/submodule-warnings' into 'master'
  
  Implement submodule warnings
  
  See merge request BuildStream/buildstream!996

• 7fe55143
  by Jonathan Maw at 2018-12-06T15:48:12Z

  Fix bst source-checkout not working with open workspaces
  

• 2a6879a5
  by Jonathan Maw at 2018-12-06T17:46:33Z

  Merge branch 'jonathan/source-checkout-workspace' into 'master'
  
  Fix bst source-checkout not working with open workspaces
  
  Closes #807
  
  See merge request BuildStream/buildstream!997

• e61e6e50
  by Tristan Van Berkom at 2018-12-07T07:57:13Z

  _yaml.py: Added `allow_none` parameter to _yaml.node_get()
  
  This allows specifying whether or not the code believes a None
  value is acceptable if explicitly expressed to be None in the YAML.
  
  A previous patch 3ba544b80f9f268be8ffe62fc8589b30212ec4a2 changed
  the behavior to accept None for all values across the board, along
  with allowing explicitly setting the `default_value` to `None` which
  caused the code to be much more readable (this was the main motivation
  of the patch, but it had the side effect of allowing None for everything).
  
  In the majority of cases we load YAML however, it either has a value or
  it is not specified in the YAML, and None is hardly ever acceptable
  to be explicitly specified (it may be in the case you want to override
  something with nothing in the YAML, but it is the edge case).
  
  This had the side effect of causing unexpected crashes where the
  YAML specifies an empty dictionary for instance.
  
  Instead of forcing the caller to handle a possible None value, give
  the choice to the caller if they really want to accept a None value,
  and raise the LoadError otherwise.
  
  This fixes issue #803, which is a crash due to receival of an
  unexpected None value through _yaml.node_get()
  

• 504ecb3e
  by Tristan Van Berkom at 2018-12-07T07:57:56Z

  plugin.py: Added new `allow_none` parameter to Plugin.node_get_member(), defaulting to False.
  
  Allow plugins to conveniently leverage the new `allow_none` parameter,
  and have Plugin.node_get_member() raise a LoadError in the usual
  case where None is not an acceptable value to be explicitly set in
  the user provided YAML.
  

• 642ae4e8
  by Tristan Van Berkom at 2018-12-07T07:57:56Z

  tests/format/project.py: Added new regression test for empty dependency dictionaries
  
  This adds a regression test for issue #803
  

• 2a0676c3
  by Tristan Van Berkom at 2018-12-07T08:33:14Z

  Merge branch 'tristan/yaml-optionally-allow-none' into 'master'
  
  Only optionally allow None values in user provided YAML
  
  Closes #803
  
  See merge request BuildStream/buildstream!999

• 56c2c2cc
  by Javier Jardón at 2018-12-09T16:59:26Z

  buildstream/data/projectconfig.yaml: Remove default strip-binaries
  
  They are too specific to be included by default
  
  Recommendation is if you are building in Linux is to use the
  ones begin used in the freedesktop-sdk project, for example
  
  See #645
  

• dcb6b5e8
  by Javier Jardón at 2018-12-09T16:59:26Z

  NEWS: Update for removal of default strip-commands
  

• ae0afce5
  by Javier Jardón at 2018-12-09T17:08:16Z

  buildstream/buildelement.py: Document where to find strip commands for Linux
  

• 873b618c
  by Javier Jardón at 2018-12-10T09:49:52Z

  Merge branch 'jjardon/strip-binaries-removal' into 'master'
  
  Remove default strip-commands
  
  Closes #645
  
  See merge request BuildStream/buildstream!987

• f2fcc2f6
  by Valentin David at 2018-12-11T12:12:08Z

  Force updating tags when fetching git repository
  
  When using aliases there are multiple remotes used in the cache
  repository. When fetching, tags are not updated if the were previously
  fetched from a different remote. Commits that not in a branch and only
  tagged do not get fetched if the tag is not fetched.
  
  Fixes #812
  

• 717c10d1
  by Valentin David at 2018-12-11T12:44:56Z

  Merge branch 'valentindavid/git_force_fetch_tags' into 'master'
  
  Force updating tags when fetching git repository
  
  Closes #812
  
  See merge request BuildStream/buildstream!1000

• 7892287a
  by Jonathan Maw at 2018-12-11T12:56:32Z

  utils.py: Add a helper for searching upwards for files
  
  i.e. with a given directory and filename, check parent directories until
  either a directory with the filename is found, or you reach the root of
  the filesystem.
  
  This is a part of #222
  

• 67c7a58d
  by Jonathan Maw at 2018-12-11T12:56:32Z

  Create and store data inside projects when opening workspaces
  
  Changes to _context.py:
  * Context has been extended to contain a WorkspaceProjectCache, as there
    are times when we want to use it before a Workspaces can be initialised
    (looking up a WorkspaceProject to find the directory that the project is
    in)
  
  Changes to _stream.py:
  * Removed staging the elements from workspace_open() and workspace_reset()
  
  Changes in _workspaces.py:
  * A new WorkspaceProject contains all the information needed to refer back
    to a project from its workspace (currently this is the project path and
    the element used to create this workspace)
  * This is stored within a new WorkspaceProjectCache object, which keeps
    WorkspaceProjects around so they don't need to be loaded from disk
    repeatedly.
  * Workspaces has been extended to contain the WorkspaceProjectCache, and
    will use it when opening and closing workspaces.
  * Workspaces.create_workspace has been extended to handle the staging of
    the element into the workspace, in addition to creating the equivalent
    WorkspaceProject file.
  
  This is a part of #222
  

• 64836b18
  by Jonathan Maw at 2018-12-11T12:56:32Z

  _project.py: Rename _ensure_project_dir to _find_project_dir
  
  This is a part of #222
  

• 7deeb2c3
  by Jonathan Maw at 2018-12-11T12:56:32Z

  cli.py: Use utils' search upwards helper when searching for project.conf
  
  This is a part of #222
  

• 496f0ab7
  by Jonathan Maw at 2018-12-11T12:56:32Z

  _project.py: Find project from workspace if outside of a project
  
  This is a part of #222
  

• 921f2bcb
  by Jonathan Maw at 2018-12-11T12:56:32Z

  _workspaces.py: Do not include .bstproject.yaml in the cache key
  
  This is a part of #222
  

• f145a3e4
  by Jonathan Maw at 2018-12-11T12:56:32Z

  cli: Interactively warn if the user is trying to close the workspace they're using to load the project
  
  This involves changes in:
  * _stream.py:
    * Add the helper Stream.workspace_is_required()
  * userconfig.yaml:
    * Add a default value for prompt.really-workspace-close-project-inaccessible
  * _context.py:
    * Load the prompt 'really-workspace-close-project-inaccessible' from
      user config.
  * cli.py:
    * If buildstream is invoked interactively, prompt the user to confirm
      that they want to close the workspace they're using to load this
      project.
  
  This is a part of #222
  

• 494cb7c6
  by Jonathan Maw at 2018-12-11T14:12:54Z

  tests: Test bst commands from an external workspace
  
  This is a part of #222
  

• fd1c5c5a
  by Jonathan Maw at 2018-12-11T14:12:54Z

  NEWS: Add an entry for being able to run commands from a workspace
  
  This is a part of #222
  

• 4219a6b4
  by Jonathan Maw at 2018-12-11T16:00:55Z

  Merge branch 'jonathan/workspace-fragment-create' into 'master'
  
  Support invoking buildstream from a workspace outside a project
  
  See merge request BuildStream/buildstream!924

• e564251b
  by Jonathan Maw at 2018-12-11T16:15:19Z

  _stream.py: Fix ugly error when opening a workspace using a relative path
  

• 7cf83343
  by Jonathan Maw at 2018-12-11T16:15:19Z

  _context.py: Store the directory buildstream was invoked on
  

• a1dee91e
  by Jonathan Maw at 2018-12-11T16:15:19Z

  Make specifying elements optional in bst commands
  
  Known issues:
  * `bst shell` works, but `bst shell COMMANDS...` doesn't, because click
    has no way of separating optional args from variable-length args.
  * `bst checkout` and `bst source-checkout`'s usage strings mark LOCATION
    as an optional argument. Because click gets confused if there's an
    optional argument before a mandatory argument, I had to mark LOCATION
    as optional internally.
  * `bst workspace open` makes no sense with element being optional, so
    I skipped it.
  * `bst workspace close` will probably need to be revisited when multiple
    projects can own one workspace.
  * `bst workspace reset` will happily delete the directory you're
    currently in, requiring you to `cd $PWD` to see the contents of your
    directory.
    I could exclude the top-level directory of the workspace being
    deleted, but it is entirely valid to run workspace commands from deeper
    in the workspace.
  
  This is a part of #222
  

• 9b192015
  by Jonathan Maw at 2018-12-11T16:15:19Z

  tests: Add tests for guessing element names
  
  This is a part of #222
  

• e340371f
  by Jonathan Maw at 2018-12-11T16:15:19Z

  Add NEWS for guessing the element in commands
  
  This is a part of #222
  

• bbf81c3d
  by Jonathan Maw at 2018-12-11T16:15:19Z

  doc: Point out that element is optional when commands are run from the workspace
  
  This is a part of #222
  

• a5a53ddd
  by Jonathan Maw at 2018-12-12T13:44:09Z

  Merge branch 'jonathan/workspace-fragment-guess-element' into 'master'
  
  Guess the element when running commands in a workspace
  
  See merge request BuildStream/buildstream!954

• d55b9e39
  by Phil Dawson at 2018-12-12T13:55:19Z

  Add --tar option to source-checkout command
  
  This commit is part of the work towards #672
  

• 733aab53
  by Phil Dawson at 2018-12-12T13:55:19Z

  Add --force / -f option to source-checkout command
  

• c2efeba0
  by Phil Dawson at 2018-12-12T14:43:40Z

  Add option to source-checkout command to generate build scripts
  

• 85c61894
  by Phil Dawson at 2018-12-12T14:43:40Z

  Remove source bundle command
  
  This is part of the work towards #672
  

• a322d5c0
  by Phil Dawson at 2018-12-12T14:43:40Z

  NEWS: Add entry about the removal of source-bundle command
  

• ec909605
  by Phil Dawson at 2018-12-12T15:45:42Z

  Merge branch 'phil/source-checkout-options' into 'master'
  
  Retire bst source bundle command
  
  Closes #672
  
  See merge request BuildStream/buildstream!959

• 3697a611
  by Richard Maw at 2018-12-12T16:31:38Z

  cli: Add support for auto-completing artifact ref names
  

• b3dceb16
  by Richard Maw at 2018-12-12T16:32:41Z

  cli: Add artifact command group
  

• ba08a0cd
  by Richard Maw at 2018-12-12T16:32:41Z

  cli: Add artifact log command
  

• 70fb9554
  by Richard Maw at 2018-12-12T16:32:41Z

  tests: Add tests for artifact group commands
  

• f773e746
  by Richard Maw at 2018-12-12T16:33:02Z

  NEWS: Mention `bst artifact log`
  

• b6528441
  by richardmaw-codethink at 2018-12-12T18:00:59Z

  Merge branch 'richardmaw/artifact-log' into 'master'
  
  Add artifact log command
  
  See merge request BuildStream/buildstream!920

• d03bf316
  by Benjamin Schubert at 2018-12-13T10:24:11Z

  Mock storage space checks for tests.
  
  Fix #530
  
  - Extract free space computation in a function for easier
  mocking
  - Mock space computation during cache quota tests
  - Mock cache size during cachque quota tests
  - Add two more tests when the configuration would require
  to much storage space
  

• a116f576
  by Tristan Van Berkom at 2018-12-13T10:58:46Z

  Merge branch 'BenjaminSchubert/fix-quota-tests' into 'master'
  
  Mock storage space checks for tests.
  
  Closes #530
  
  See merge request BuildStream/buildstream!702

• 180fa774
  by Benjamin Schubert at 2018-12-13T12:05:15Z

  element.py: fix unbounded variable in nested python exception
  
  As per https://docs.python.org/3.7/reference/compound_stmts.html#except
  variables used in the "except" clause are erased at the end of the
  block.
  
  We had a conflict with a nested exception using the same variable as
  before.
  
  This renames the nested variable to avoid the clash
  

• 224aa4c2
  by Benjamin Schubert at 2018-12-13T12:34:41Z

  Merge branch 'bschubert/fix-unbound-variable-exception' into 'master'
  
  element.py: fix unbounded variable in nested python exception
  
  See merge request BuildStream/buildstream!1006

• 053beb66
  by Tristan Van Berkom at 2018-12-13T14:23:19Z

  manual plugin: Support virtual directories
  

• 29ab271c
  by Tristan Van Berkom at 2018-12-13T14:23:19Z

  makemaker plugin: Support virtual directories
  

• ba955cf0
  by Tristan Van Berkom at 2018-12-13T14:23:19Z

  pip element plugin: Support virtual directories
  

• 6010b5a4
  by Tristan Van Berkom at 2018-12-13T14:23:19Z

  modulebuild plugin: Support virtual directories
  

• 3a6d27a4
  by Tristan Van Berkom at 2018-12-13T14:23:19Z

  distutils plugin: Support virtual directories
  

• 4c0e602c
  by Tristan Van Berkom at 2018-12-13T14:23:19Z

  BuildElement: Don't enable batching of prepare and assemble by default
  
  Some external plugins depend derive from BuildElement and are broken
  by BuildElement enabling this batching by default.
  
  Instead, enable it in all of the individual build element plugin
  implementations.
  
  This fixes issue #800
  

• 60ddeeb9
  by Tristan Van Berkom at 2018-12-13T14:58:28Z

  Merge branch 'tristan/dont-batch-prepare-assemble-by-default' into 'master'
  
  Dont batch prepare assemble by default
  
  Closes #800
  
  See merge request BuildStream/buildstream!1009

• bf72cc42
  by Angelos Evripiotis at 2018-12-13T17:31:17Z

  contributing: mandate end-to-end tests, allow others
  
  Update the section on adding tests, to fulfill these broad points:
  
  - Mandate end-to-end testing, with rationale.
  - Mention internal APIs as endpoints for testing.
  - Warn against pitfalls of unit-testing.
  
  It's more text than I would have liked, perhaps in later work we'll be
  able to say it with less.
  
  This change integrates feedback from the mailing list
  'Guidance on Unit Tests' thread:
  https://mail.gnome.org/archives/buildstream-list/2018-November/msg00045.html
  

• 13eb7ed2
  by Angelos Evripiotis at 2018-12-13T18:01:56Z

  Merge branch 'aevri/contributing_e2e_tests' into 'master'
  
  contributing: more clarity on testing
  
  See merge request BuildStream/buildstream!973

• 629a6e52
  by Chandan Singh at 2018-12-14T19:34:20Z

  Introduce new "source" command group
  
  Following the message thread
  https://mail.gnome.org/archives/buildstream-list/2018-November/msg00106.html,
  implement a new command group called `source`. Move existing `track`,
  `fetch`, and the recently added `source-checkout` commands under this
  group.
  
  For `track` and `fetch`, this is a BREAKING change, as the old commands
  have been marked as obsolete. Using them will result in an error message
  that refers people to use the new versions, like `bst source fetch`
  instead of old `bst fetch`. `source-checkout` will now become
  `source checkout` (the dash has turned into a space), and is not a
  breaking change as it was added in the current development cycle.
  
  Note that the functionality to hide commands from help output was added
  only recently in Click, so the minimum version of Click that we now
  require is 7.0.
  
  Summary of changes:
  
  * _frontend/cli.py: Add `source` command group, mark previous versions
    as obsolete and hide them from the help output.
  * _frontend/complete.py: Fix completion for hidden commands.
  * setup.py: Bump Click minimum version to 7.0.
  * tests: Update to cope with the new command names.
  
  Fixes #814.
  

• f894c0a8
  by Chandan Singh at 2018-12-14T19:34:20Z

  NEWS: Add entry for new source command group
  
  Also, change the existing entry about `source-checkout` to be
  `source checkout` instead. And, while we are there, move it near the
  other announcements about the source command group.
  

• b23bec55
  by Chandan Singh at 2018-12-14T20:07:13Z

  Merge branch 'chandan/source-subgroup' into 'master'
  
  Introduce new "source" command group
  
  Closes #814
  
  See merge request BuildStream/buildstream!1003

• d2105909
  by Raoul Hidalgo Charman at 2018-12-18T11:13:56Z

  Add remote execution instance option
  
  This is used when sending execution requests, to specify which instance of the
  execution server to use.
  
  Partial fix for #627.
  

• a3bbec23
  by Jim MacArthur at 2018-12-18T11:13:56Z

  sandboxremote: Add server/storage config with defaults
  
  Adds instance name support for the remote execution storage service.
  

• 89219f61
  by Jim MacArthur at 2018-12-18T11:13:56Z

  _cascache.py: Add instance names to GRPC calls
  
  Adds the 'instance_name' parameter, which may be None, to most GRPC
  calls in the CASCache object. ByteStream requests already have
  instance_name supplied in the resource name, so do not need the
  parameter.
  
  Closes #627.
  

• 3dc20963
  by Jim MacArthur at 2018-12-18T11:13:56Z

  Documentation: Include instance-name in remote execution documentation
  

• 644d8b28
  by Jim MacArthur at 2018-12-18T11:45:30Z

  Merge branch 'raoul/627-RE-instance-config' into 'master'
  
  Remote-execution instance configuration support
  
  Closes #627
  
  See merge request BuildStream/buildstream!952

• e29aea36
  by William Salmon at 2018-12-19T13:23:19Z

  Basic options for shell --build to use buildtrees
  
  Fixes issue #740
  

• 898a23a5
  by Will Salmon at 2018-12-19T14:35:41Z

  Merge branch 'willsalmon/shellBuildTrees' into 'master'
  
  Shell --build has optional buildtrees
  
  Closes #740
  
  See merge request BuildStream/buildstream!986

• a2f1d879
  by Javier Jardón at 2018-12-19T15:36:11Z

  README.rst: Add license badge
  

• aae5e4b3
  by Javier Jardón at 2018-12-19T16:09:48Z

  Merge branch 'jjardon/license_badge' into 'master'
  
  README.rst: Add license badge
  
  See merge request BuildStream/buildstream!1014

• 2b767fe8
  by Jürg Billeter at 2018-12-20T10:06:11Z

  Move fetch logic from FetchQueue to Element
  
  The queue shouldn't need to know about individual sources. This is in
  line with _track() and _get_consistency().
  

• 7a102144
  by Jürg Billeter at 2018-12-20T10:06:11Z

  element.py: Do not call fetch() for cached sources
  

• b325989e
  by Jürg Billeter at 2018-12-20T10:07:20Z

  tests/sources: Test that fetch() is not called for cached sources
  

• 77d8ad45
  by Jürg Billeter at 2018-12-20T10:42:39Z

  Merge branch 'juerg/fetch' into 'master'
  
  Do not call fetch() for cached sources
  
  See merge request BuildStream/buildstream!992

• 9835b7f1
  by Chandan Singh at 2018-12-20T12:50:02Z

  Expose base class for Git source plugins
  
  Add a `_GitSourceBase` class to act as the base class for deriving
  source plugins that work with Git. The first user of this base class is
  the `git` source plugin that's in core at the moment.
  
  The contents of this base class is almost identical to the existing `GitSource`
  class. The two notable differences are:
  
  - the private methods now have leading underscores, as some were missing
    it originally
  - the name of the class
  
  Note that we are exposing a private member here as we expect it to move to a
  separate package soon. See the following discussion for more details:
  https://gitlab.com/BuildStream/buildstream/issues/739#note_124819869
  
  Fixes #739.
  

• 14da6955
  by Chandan Singh at 2018-12-20T13:34:10Z

  Merge branch 'chandan/abstract-git-source' into 'master'
  
  Expose base class for Git source plugins
  
  Closes #739
  
  See merge request BuildStream/buildstream!1019

• 7368f569
  by Angelos Evripiotis at 2018-12-20T13:59:23Z

  _project::_find_project_dir: fix error message
  
  Now that we are also looking for WORKSPACE_PROJECT_FILE, we should add
  it to the error message. While we're there, also mention the directory
  we are looking in - this could help scripters be clear about what went
  wrong.
  
  The new error message looks like this:
  
      Error loading project: None of ['project.conf', '.bstproject.yaml']
      found in '/src/temp/blah' or any of its parent directories
  

• 46efc91d
  by Angelos Evripiotis at 2018-12-20T13:59:23Z

  BREAK: remove auto-init behaviour
  
  In the event that the project could not be found, stop BuildStream from
  asking if the user would like to create a new project. Exit with error
  instead, and give a hint to the user in case they're new.
  
  As proposed on the mailing list here:
  https://mail.gnome.org/archives/buildstream-list/2018-December/msg00082.html
  
  The new interaction looks like this:
  
      $ bst show nonsuch.bst
      No project found. You can create a new project like so:
  
          bst init
  
      Error loading project: None of ['project.conf', '.bstproject.yaml']
      found in '/src/temp/blah' or any of its parent directories
  
  Fixes #826
  

• e0c575c4
  by Angelos Evripiotis at 2018-12-20T14:37:38Z

  Merge branch 'aevri/rm-autoinit' into 'master'
  
  BREAK: remove auto-init behaviour
  
  Closes #826
  
  See merge request BuildStream/buildstream!1015

• c05d8b4f
  by Chandan Singh at 2018-12-20T15:53:12Z

  tests/frontend/logging.py: Fix regex Deprecation Warning
  
  Use raw strings for regex searches, which is the preferred way to do
  regular expressions in Python.
  
  Without this patch, currently we get the following warnings:
  
  ```
  tests/frontend/logging.py:44
    /builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:44: DeprecationWarning: invalid escape sequence \[
      m = re.search("\[\d\d:\d\d:\d\d\]\[\]\[\] SUCCESS Checking sources", result.stderr)
  
  tests/frontend/logging.py:80
    /builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:80: DeprecationWarning: invalid escape sequence \d
      m = re.search("\d\d:\d\d:\d\d,\d\d:\d\d:\d\d.\d{6},\d\d:\d\d:\d\d,,,SUCCESS,Checking sources", result.stderr)
  ```
  

• e8055a56
  by Chandan Singh at 2018-12-20T15:53:12Z

  buildstream/utils.py: Fix regex Deprecation Warning
  
  Specify flags at the start of the _expression_ as per the recommendation
  of the standard library.
  
  Without this patch, we currently get the following warning:
  
  ```
  tests/examples/junctions.py::test_open_cross_junction_workspace
    /builds/BuildStream/buildstream/dist/buildstream/buildstream/utils.py:213: DeprecationWarning: Flags not at the start of the _expression_ '\\/[^/]*\\Z(?ms)'
      regexer = re.compile(_expression_)
  ```
  

• cd4889af
  by Chandan Singh at 2018-12-20T16:34:08Z

  Merge branch 'chandan/fix-warning' into 'master'
  
  Fix Deprecation warnings from regex module
  
  See merge request BuildStream/buildstream!1010

• ac995236
  by Tom Pollard at 2018-12-20T17:18:02Z

  _stream.py: fix _buildtree_pull_required logic
  

• c3153dea
  by Tom Pollard at 2018-12-21T10:10:58Z

  Merge branch 'tpollard/fixbuildtreereq' into 'master'
  
  _stream.py: fix _buildtree_pull_required logic
  
  See merge request BuildStream/buildstream!1021

• caf49669
  by Tristan Van Berkom at 2018-12-26T19:54:05Z

  dev-requirements.txt: Require at least pytest 3.9
  
  Recently we have been adding some tests which use the new tmp_path
  fixture from pytest, this is new in 3.9.
  

• 425781db
  by Tristan Van Berkom at 2018-12-26T19:54:05Z

  _signals.py: Fixing new (simplifiable-if-_expression_) linter errors
  

• 1f8cc996
  by Tristan Van Berkom at 2018-12-26T19:54:05Z

  element.py: Fixing new (unnecessary-pass) and (simplifiable-if-_expression_) linter errors
  

• 89632150
  by Tristan Van Berkom at 2018-12-26T19:54:05Z

  source.py: Fixing new (unnecessary-pass) linter error
  

• 26d135ba
  by Tristan Van Berkom at 2018-12-26T19:54:05Z

  _artifactcache/casserver.py: Fixed new (wrong-import-order) linter error
  

• d0988f80
  by Tristan Van Berkom at 2018-12-26T19:54:05Z

  _frontend/cli.py: Fixing new (unnecessary-pass) linter errors
  

• 4d54ab78
  by Tristan Van Berkom at 2018-12-26T19:54:05Z

  _frontend/complete.py: Fixing new (simplifiable-if-_expression_) linter error
  

• ee0d5bf6
  by Tristan Van Berkom at 2018-12-26T19:54:05Z

  _frontend/widget.py: Fixing new (wrong-import-order) linter error
  

• 36295282
  by Tristan Van Berkom at 2018-12-26T19:54:05Z

  plugins/elements/import.py: Fixing new (duplicate-string-formatting-argument) linter error
  

• 4ffe03c6
  by Tristan Van Berkom at 2018-12-26T19:54:05Z

  buildstream/storage/_casbaseddirectory.py: Fixing new (unnecessary-pass) linter errors
  

• 80c137d5
  by Tristan Van Berkom at 2018-12-26T19:54:05Z

  dev-requirements.txt: Require new version of pytest-datafiles
  
  Since Thomas Nixon fixed the datafiles plugin to be compatible
  with pytest >= 3.8.0, we can now depend on it and avoid the
  warnings and errors which the older datafiles plugin causes.
  
  This fixes issue #636
  

• 61906a14
  by Tristan Van Berkom at 2018-12-26T19:54:05Z

  dev-requirements.txt: Remove the restriction on pylint.
  
  After having fixed the more recent linting errors, the
  tests should pass with recent versions of pylint. Tested
  with pylint 2.2.2.
  

• 94b9948c
  by Tristan Van Berkom at 2018-12-26T22:35:29Z

  Merge branch 'tristan/test-fixes' into 'master'
  
  Misc fixes to the tests and newer linter errors
  
  Closes #636
  
  See merge request BuildStream/buildstream!1011

• fe4fc0b5
  by Tristan Van Berkom at 2018-12-26T23:19:07Z

  _downloadablefilesource.py: Avoid crashes when HOME is unset
  
  The python netrc module will raise OSError in the case
  that HOME is not set, this was discovered while running
  tests under tox.
  

• c61e1793
  by Tristan Van Berkom at 2018-12-26T23:19:07Z

  tests/testutils/runcli.py: Fixed broken environment handling
  
  Treat None values in the passed dictionary as keys to be removed
  from the environment, this was already happening at restoration time.
  

• 50c1bb7f
  by Tristan Van Berkom at 2018-12-26T23:19:07Z

  tests/sources/tar.py: Test that we don't crash when HOME is unset
  
  This happens when the netrc module is searching for a ~/.netrc
  file and it doesnt find any HOME set.
  

• 32a101f6
  by Tristan Van Berkom at 2018-12-26T23:53:29Z

  Merge branch 'tristan/fix-netrc-crasher' into 'master'
  
  Fix netrc crasher
  
  See merge request BuildStream/buildstream!1025

• a080b6c7
  by Tristan Van Berkom at 2018-12-31T19:03:25Z

  tests/loader/junctions.py: Use Result checking APIs instead of manually
  
  Better to use the same API everywhere for thecking CLI results.
  

• d7c8b54d
  by Tristan Van Berkom at 2018-12-31T19:14:16Z

  tests/loader/junctions.py: Test the error we expect when a cross junction element is not found
  

• 560a6342
  by Tristan Van Berkom at 2018-12-31T19:49:24Z

  Merge branch 'tristan/junction-tests' into 'master'
  
  Junction test addition
  
  See merge request BuildStream/buildstream!1029

• 8a020a5a
  by Chandan Singh at 2018-12-31T19:54:25Z

  _loader/loader.py: Refactor warnings about element names in one method
  
  Currently we some duplication in the way we check for invalid filenames.
  To make it more robust and allow room for adding more warnings, refactor
  it into a separate method `_warn_invalid_elements()` that handles just
  this.
  

• 87568074
  by Chandan Singh at 2018-12-31T19:54:25Z

  _loader/loader.py: Add warnings about invalid characters in filename
  
  See https://mail.gnome.org/archives/buildstream-list/2018-December/msg00061.html
  for some related discussion.
  

• 4b2992c1
  by Chandan Singh at 2018-12-31T19:54:25Z

  tests/buildcheckout.py: Add tests for invalid chars in filename
  

• ce9e9c2d
  by Chandan Singh at 2018-12-31T19:54:25Z

  doc: Add note about valid element names
  

• b848172c
  by Tristan Van Berkom at 2018-12-31T20:29:51Z

  Merge branch 'chandan/element-filename-requirements' into 'master'
  
  Add warnings about invalid characters in filename
  
  See merge request BuildStream/buildstream!1028

• bb9df2a1
  by Chandan Singh at 2019-01-02T11:59:35Z

  doc/Makefile: Update comment about sphinx entrypoint
  
  https://github.com/sphinx-doc/sphinx/issues/4375 has been closed as
  `wontfix`. Update the comment in doc/Makefile that mentions it, based on
  the discussion on the issue.
  

• 43eaf6ee
  by Chandan Singh at 2019-01-02T13:42:10Z

  Merge branch 'chandan/update-doc-makefile-note' into 'master'
  
  doc/Makefile: Update comment about sphinx entrypoint
  
  See merge request BuildStream/buildstream!1026

• 3b5c8a28
  by Tristan Maat at 2019-01-02T15:57:13Z

  .gitlab-ci.yml: Use new testsuite images with pycodestyle 1.4.0
  
  We need to bump the testsuite image versions to use the new
  pycodestyle update that allows excluding file paths, so that we can
  exclude generated python code.
  

• bd60e8a9
  by Javier Jardón at 2019-01-02T16:59:41Z

  Use pycodestyle instead of pep8
  
  pep8 package was renamed to pycodestyle to reduce confusion.
  
  Disabled warnings:
  W504 - Line break after binary operator (seems people like this)
  W605 - Invalid escape sequence (some of our regexes use these)
  

• 32c47d1c
  by Tristan Maat at 2019-01-02T17:32:09Z

  Merge branch 'jjardon/pycodestyle' into 'master'
  
  Use pycodestyle instead pep8 python module
  
  See merge request BuildStream/buildstream!638

• 1cbd9a73
  by Chandan Singh at 2019-01-02T19:39:58Z

  Add requirements files for install, test and plugin dependencies
  
  Add `.in` and `.txt` requirements files for BuildStream's pure python
  dependencies. For each pair, the `.in` file is supposed to capture the
  loose version requirements, and the corresponding `.txt` file is
  supposed to have frozen requirements. We have 3 such sets:
  
  * `requirements`: BuildStream's runtime dependencies
  * `dev-requirements`: Dependencies for running tests
  * `plugin-requirements`: Dependencies for core plugins
  
  Note that the frozen requirements files will only be used for testing
  purposes, and `setup.py` will continue to read loose requirements.
  

• 6b0cb5f3
  by Chandan Singh at 2019-01-02T19:41:21Z

  Add tox.ini to enable running tests using tox
  
  Add `tox.ini` file that will enable us to use
  [tox](https://tox.readthedocs.io/) as a frontend for running tests.
  Since we share the config via `setup.cfg` and requirements via
  requirements files, commands like `python3 setup.py test` will continue
  to work.
  

• 9d2d1d4f
  by Chandan Singh at 2019-01-03T03:31:52Z

  .gitlab-ci.yml: Run tests using tox
  
  Instead of invoking tests throung `setup.py`, use `tox` as a frontend in
  the CI pipelines.
  

• 8ae04283
  by Tristan Van Berkom at 2019-01-03T03:31:52Z

  CONTRIBUTING.rst: Updated to reflect running tests using tox.
  

• e7e2a5c1
  by Chandan Singh at 2019-01-03T03:31:52Z

  CONTRIBUTING.rst: Add steps for installing non-python build dependencies
  
  Since we don't allow use of site packages by default in our `tox`
  configuration, people will need to install non-python build dependencies
  of some of our dependencies that do not provide pre-built wheels. We
  have two such packages at the moment:
  
  * psuitl: requires python C headers, compiler
  * pygobject: requires python C headers, pkg-config, compiler, cairo
    headers, and gobject libraries
  

• afa0a369
  by Chandan Singh at 2019-01-03T03:31:52Z

  Move all requirements files into "tools" directory
  
  These new `.in` and `.txt` are making the repository look very
  cluttered. Move them to a separate `tools` directory to make it look a
  bit cleaner.
  

• 3fae3446
  by Chandan Singh at 2019-01-03T03:44:17Z

  Move sphinx build functionality to tox
  
  Currently the CI and the docs both have to duplicate the same inforation
  about how to gather dependencies etc, and have to use hacky ways to run
  them.
  
  Add a new `docs` environment to our tox setup so that building docs is
  as simple as running `tox -e docs`.
  

• 1f7bc655
  by Chandan Singh at 2019-01-03T03:44:17Z

  CONTRIBUTING.rst: Add instructions to recreate tox environment
  
  When we update our dependencies, developers will eventually need to
  recreate their `tox` environments to get correct results. This happens
  because `tox` isn't particularly good at recognizing changes in the
  requirements files.
  

• 669b55b0
  by Tristan Van Berkom at 2019-01-03T15:47:27Z

  Merge branch 'chandan/toxify' into 'master'
  
  Use tox as a frontend for running tests
  
  See merge request BuildStream/buildstream!1027

• ca547f19
  by Chandan Singh at 2019-01-03T19:34:49Z

  tests/frontend/buildcheckout.py: Fix bad filename issue for Windows
  
  In BuildStream/buildstream!1028, we added a test specifically to test
  that BuildStream correctly raises an warning when the name of an element
  contains characters that are invalid on Windows. Unfortunately, we
  didn't see it coming that it would make it impossible to checkout this
  branch on Windows.
  
  Fix it by generating this file, only if we are not running on Windows.
  
  * tests/testutils/site.py: Add `IS_WINDOWS` check
  * tests/frontend/buildcheckout.py: Generate file with invalid filename
    on the fly
  * Remove tests/frontend/project/elements/invalid-chars.
  
  Fixes #842.
  
  Note that this may still cause issues on WSL when running tests on a
  shared filesystem, but that seems to be a generic issue on WSL with
  `os.rename`.
  

• 01c3761c
  by Chandan Singh at 2019-01-03T20:03:32Z

  Merge branch 'chandan/fix-bad-filename-mini-disaster' into 'master'
  
  tests/frontend/buildcheckout.py: Fix bad filename issue for Windows
  
  Closes #842
  
  See merge request BuildStream/buildstream!1032

• 7850e4e2
  by Tristan Van Berkom at 2019-01-03T20:55:00Z

  setup.cfg: Specify the pycodestyle configurations separately from pytest configuration
  

• 3c9d592d
  by Tristan Van Berkom at 2019-01-03T20:55:00Z

  setup.cfg: Don't lint by default through setup.py test invocations
  

• feba48e7
  by Tristan Van Berkom at 2019-01-03T20:55:00Z

  tox.ini: Added new 'lint' environment to lint separately
  

• 9caa0ea0
  by Tristan Van Berkom at 2019-01-03T20:55:00Z

  .gitlab-ci.yml: Update the CI to run the linter separately through tox
  
  This should save us some cycles in CI, and also allow the
  developer to more conveniently lint separately from testing.
  

• 91c8b634
  by Tristan Van Berkom at 2019-01-03T21:04:49Z

  CONTRIBUTING.rst: Adding instructions to run the linter separately
  

• 892bf99e
  by Tristan Van Berkom at 2019-01-03T21:31:04Z

  tools/requirements.{in,txt}: Fix warning about duplicate jinja
  
  Apparently you need to spell Jinja2 with a capital J, otherwise
  we find it listed twice in the resulting requirements.txt files.
  

• 7c681c42
  by Tristan Van Berkom at 2019-01-03T21:32:40Z

  tools/dev-requirements.{in,txt}: Removed requirements on pylint/codestyle pytest plugins
  
  We no longer run the linters through pytest, so we don't need these
  plugins anymore.
  

• d6689ed2
  by Tristan Van Berkom at 2019-01-03T22:09:02Z

  Merge branch 'tristan/toxic-linting' into 'master'
  
  Run the linter separately from the tests
  
  See merge request BuildStream/buildstream!1033

• e15278d7
  by Thomas Coldrick at 2019-01-03T22:14:01Z

  Use collections.abc for Mapping, Iterable
  
  In _yaml.py and _frontend/complete.py we were getting pylint warnings
  for using collections.Mapping and collections.Iterable, which are
  abstract classes now provided from collections.abc. This patch just
  uses the classes from the right place.
  

• a85da591
  by Tristan Van Berkom at 2019-01-03T22:36:16Z

  Merge branch 'coldtom/collections' into 'master'
  
  Use collections.abc for Mapping, Iterable
  
  Closes #831
  
  See merge request BuildStream/buildstream!1020

• cf0e7d26
  by Jürg Billeter at 2019-01-04T07:07:59Z

  _frontend/cli.py: Reinstate support for guessing targets
  
  Moving fetch and track to the source command group accidentally dropped
  the support for guessing targets for these commands when invoked from a
  workspace directory. This brings it back.
  

• 0751fc8f
  by Jürg Billeter at 2019-01-04T11:27:58Z

  Merge branch 'juerg/source-guess-target' into 'master'
  
  _frontend/cli.py: Reinstate support for guessing targets
  
  See merge request BuildStream/buildstream!1036

• ebd965fe
  by Chandan Singh at 2019-01-04T15:21:35Z

  Rename "tools" directory to "requirements"
  
  This seems like a better name for the directory, as it more closely
  describes the purpose of its contents.
  

• bb712886
  by Chandan Singh at 2019-01-04T15:21:35Z

  Add Makefile to update requirements files
  

• b9792837
  by Chandan Singh at 2019-01-04T15:21:35Z

  CONTRIBUTING.rst: Add instructions to update requirements files
  
  Split the "The MANIFEST.in and setup.py" section in two:
  "Managing data files" and "Updating BuildStream's Python dependencies".
  
  Briefly explain the layout of `requirements` directory and add
  instructions to use the Makefile added in the last commit.
  

• 45692335
  by Tristan Van Berkom at 2019-01-04T15:50:14Z

  Merge branch 'chandan/update-requirements-one-liner' into 'master'
  
  Add Makefile to update requirements files
  
  See merge request BuildStream/buildstream!1035

• f42dcb54
  by Chandan Singh at 2019-01-04T17:16:43Z

  .gitlab-ci.yml: Remove prepare stage
  
  As we now run tests using `tox`, we don't need to worry about manually
  packing and unpacking BuildStream. So, we can remove the preapre stage
  entirely.
  
  Update `coverage` and nightly jobs to appropriately cope with this
  change. Both these jobs now install all runtime dependencies from
  requirements files.
  

• ee29f207
  by Chandan Singh at 2019-01-04T17:41:02Z

  Merge branch 'chandan/no-sdist' into 'master'
  
  .gitlab-ci.yml: Remove prepare stage
  
  See merge request BuildStream/buildstream!1037

• f122cfe8
  by Jürg Billeter at 2019-01-06T17:57:33Z

  _frontend/app.py: Use buildtree for interactive shell on build failure
  
  Fixes: e29aea36 ("Basic options for shell --build to use buildtrees")
  

• 7dfb85b3
  by Tristan Van Berkom at 2019-01-06T18:31:25Z

  Merge branch 'juerg/shell-buildtree' into 'master'
  
  _frontend/app.py: Use buildtree for interactive shell on build failure
  
  See merge request BuildStream/buildstream!1039

• 891fcb0e
  by Tristan Van Berkom at 2019-01-07T16:47:01Z

  Fix stack traces discovered with ^C forceful termination.
  
    * utils.py:_kill_process_tree(): Ignore NoSuchProcess errors
  
      These are caused because we issue SIGTERM, and if the process
      has not exited after a timeout, we kill it.
  
    * _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here
      redundantly, they are already ignored.
  
  It seems that we were ignoring it after sleeping when terminating
  tasks from the scheduler... but we were not ignoring it when performing
  the same pattern in the `Plugin.call()` -> `utils._call()` path, so
  we were still getting these exceptions at termination time from host
  tool processes launched by source plugins.
  

• 5de42d43
  by Tristan Van Berkom at 2019-01-07T18:00:37Z

  Merge branch 'tristan/keyboard-interrupt-stack-trace' into 'master'
  
  Fix stack traces discovered with ^C forceful termination.
  
  See merge request BuildStream/buildstream!1043

• 059035b9
  by Tristan Van Berkom at 2019-01-07T18:02:00Z

  _scheduler/scheduler.py: Make _schedule_jobs() private
  
  This is not used anywhere outside of the Scheduler, currently
  only the Scheduler itself is allowed to queue a job at this level.
  
  If the highlevel business logic for automatic queueing of auxiliary
  jobs moves to another location, we can make this public again.
  

• b83d1b1f
  by Tristan Van Berkom at 2019-01-07T18:02:00Z

  _scheduler/scheduler.py: Only run one cache size job at a time
  
  When queuing the special cache management related cleanup and
  cache size jobs, we now treat these jobs as special and do the
  following:
  
    * Avoid queueing a cleanup/cache_size job if one is already queued
  
      We just drop redundantly queued jobs here.
  
    * Ensure that jobs of this type only run one at a time
  
      This could have been done with the Resources mechanics,
      however as these special jobs have the same properties and
      are basically owned by the Scheduler, it seemed more straight
      forward to handle the behaviors of these special jobs together.
  
  This fixes issue #753
  

• 16a8816f
  by Tristan Van Berkom at 2019-01-07T18:02:00Z

  Scheduler: Introduced JobStatus instead of simple success boolean
  
  This changes the deepest callback from when a Job completes to
  propagate a JobStatus value instead of a simple boolean, and updates
  all of the effected code paths which used to receive a boolean
  to now handle the JobStatus values.
  
  This further improves the situation for issue #753, as now we avoid
  queueing cache size jobs for pull jobs which are skipped.
  

• c2fc2a5e
  by Tristan Van Berkom at 2019-01-07T18:02:00Z

  _scheduler/jobs/job.py: Removed 'skipped' property
  
  This is redundant now that we report it through the JobStatus.
  

• 3e3984ad
  by Tristan Van Berkom at 2019-01-07T18:50:23Z

  Merge branch 'tristan/one-cache-size-job' into 'master'
  
  Only queue one cache size job
  
  Closes #753
  
  See merge request BuildStream/buildstream!1040

• 512c726e
  by Tristan Van Berkom at 2019-01-08T03:38:11Z

  sandbox/sandbox.py: Fix regression of command logging
  
  Since we added batch commands, the batch commands print the
  text of the commands directly in the message text, but this is wrong.
  
  The detail string is the appropriate place for text of unknown lengths
  (the user can actually configure how many max lines of commands they
  want to see in their log), the message text itself should be controlled
  and brief enough to avoid text wrapping.
  

• 01171988
  by Tristan Van Berkom at 2019-01-08T04:20:14Z

  Merge branch 'tristan/fix-command-status-messages' into 'master'
  
  sandbox/sandbox.py: Fix regression of command logging
  
  See merge request BuildStream/buildstream!1044

• 6c1d06d6
  by Phil Dawson at 2019-01-08T10:24:32Z

  element.py: remove reference to source bundle command
  
  This command has been replacved by the bst source checkout command
  

• 914ecb72
  by Jürg Billeter at 2019-01-08T10:54:02Z

  Merge branch 'phil/remove-source-bundle-reference' into 'master'
  
  element.py: remove documentation reference to source bundle command
  
  See merge request BuildStream/buildstream!1041

• 81f7a964
  by Tristan Van Berkom at 2019-01-08T16:55:58Z

  sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.
  
  When code is faulty, in a plugin or in the core, we should
  get a stack trace and a BUG message.
  

• edcc43ed
  by Tristan Van Berkom at 2019-01-08T19:19:10Z

  Merge branch 'tristan/sandbox-programming-error' into 'master'
  
  sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.
  
  See merge request BuildStream/buildstream!1046

• 17c5ca2d
  by Valentin David at 2019-01-09T10:45:23Z

  Use relative path to project directory for remote execution certificates/keys
  

• 605f8d11
  by Valentin David at 2019-01-09T10:45:23Z

  Add support for user remote execution configuration
  
  Fixes #631.
  

• 65ea03ab
  by Valentin David at 2019-01-09T10:45:23Z

  Add support for https channel to remote execution and actions servers
  
  Fixes #780.
  

• ba05c4fa
  by Valentin David at 2019-01-09T12:58:59Z

  Merge branch 'valentindavid/remote_execution_configuration' into 'master'
  
  Remote execution configuration: HTTPS and user configuration
  
  Closes #780 and #631
  
  See merge request BuildStream/buildstream!1030

• eb0dbcfc
  by Valentin David at 2019-01-09T13:56:58Z

  buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY
  
  ScriptElement does not use Sandbox.get_directory. It works using it
  with remote execution.
  
  Fixes #850
  

• f7681925
  by Valentin David at 2019-01-09T14:44:22Z

  Merge branch 'valentindavid/script_virtual_directory' into 'master'
  
  buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY
  
  Closes #850
  
  See merge request BuildStream/buildstream!1047

• f87d1c93
  by Jim MacArthur at 2019-01-09T15:01:18Z

  sandboxremote.py: Rename 'instance_name' option to 'instance-name'
  
  It was 'instance-name' in the documentation.
  

• ed8bc5bc
  by Jim MacArthur at 2019-01-09T15:30:28Z

  Merge branch 'jmac/rename_instance_name' into 'master'
  
  Rename 'instance_name' option to 'instance-name'
  
  See merge request BuildStream/buildstream!1048

• f29a0995
  by Tristan Van Berkom at 2019-01-09T19:51:14Z

  Support running test environments in parallel with `detox`
  
  This patch namespaces the test temp directory and the output
  coverage report file with the name of the environment under test,
  such that separately run tests do not access the same files.
  
  When running tests without tox, directly through setup.py,
  then the tmp directory will still be `./tmp`.
  
    * .gitignore: Added new .coverage-reports/ directory
  
    * .gitlab-ci.yml: Rely on tox to combine and report coverage, only
      tell tox about the COVERAGE_PREFIX so that results can be namespaced
      by CI job name.
  
      This change also publishes the sources and final combined `.coverage`
      file in an output gitlab artifact for inspection, and lists some missing
      dependencies to the `coverage` job.
  
    * tox.ini: Add comments and refactor main [testenv] section so that
      other environments dont inherit too much unrelated cruft.
  
      Generate the coverate reports in the respective {envtmpdir} so that
      all per-process coverage files are prefixed with a full path, ensuring
      that concurrent runs don't mix reports and addressing concerns
      raised in #844.
  
      Also implemented new `tox -e coverage` environment to combine
      any found coverage and print a report.
  
    * .coveragerc: Omit .tox/ directory from coverage stats
  
  Fixes issue #844
  

• d364ad02
  by Tristan Van Berkom at 2019-01-09T19:52:22Z

  CONTRIBUTING.rst: Updated to mention new `coverage` tox environment.
  
  Also point out that it is possible to run test environments in
  parallel using the `detox` tool.
  

• 1e352434
  by Tristan Van Berkom at 2019-01-09T20:36:06Z

  Merge branch 'tristan/detox-tests' into 'master'
  
  Allow using detox, and fix/refactor collection of coverage reports
  
  Closes #844
  
  See merge request BuildStream/buildstream!1051

• 116c1070
  by Tristan Van Berkom at 2019-01-09T20:39:15Z

  .coveragerc: Omit some things which are irrelevant to cover.
  
    * Omit versioneer's _version.py
  
    * Omit our __main__.py which is used only internally for generating
      documentation
  

• c8f3616d
  by Tristan Van Berkom at 2019-01-09T21:12:29Z

  Merge branch 'tristan/refine-coverage' into 'master'
  
  .coveragerc: Omit some things which are irrelevant to cover.
  
  See merge request BuildStream/buildstream!1052

• c87bb592
  by Jürg Billeter at 2019-01-10T09:43:37Z

  _platform/platform.py: Add canonicalize_arch() method
  

• 26e33346
  by Jürg Billeter at 2019-01-10T12:50:15Z

  _options/optionarch.py: Accept architecture aliases
  
  Accept common architecture aliases for arch options instead of only
  accepting the canonicalized, OS-independent architecture name. This
  restores compatibility with existing projects and may simplify option
  handling for projects that only target a single OS (and thus, do not
  need OS-independent architecture names).
  

• 06deb4c4
  by Jürg Billeter at 2019-01-10T12:50:15Z

  element.py: Accept architecture aliases for sandbox config
  
  Accept common architecture aliases for the sandbox config for
  consistency with arch options.
  

• 0c2a66b3
  by Jürg Billeter at 2019-01-10T13:06:02Z

  tests/format/optionarch.py: Add tests for architecture aliases
  

• f86bc760
  by Jürg Billeter at 2019-01-10T13:34:56Z

  Merge branch 'juerg/arch' into 'master'
  
  Accept architecture aliases
  
  See merge request BuildStream/buildstream!1034

• 347eb34e
  by Angelos Evripiotis at 2019-01-10T14:38:23Z

  contributing: fix 'oprtation' and some other typos
  

• 6bc27bc1
  by Angelos Evripiotis at 2019-01-10T15:07:46Z

  Merge branch 'aevri/contrib-typos' into 'master'
  
  contributing: fix 'oprtation' and some other typos
  
  See merge request BuildStream/buildstream!1055

• 0b83d024
  by Tristan Van Berkom at 2019-01-10T20:02:50Z

  _pipeline.py: Fix the planner to retain the original order when depth sorting
  
  The algorithm adds elements to a dictionary and then sorts the dictionary
  by the discovered depths while recursing - using an OrderedDict is enough
  to ensure a stable order.
  
  This fixes `bst show --deps plan ...` reporting different results on
  each invocation.
  
  This fixes an aspect of #824
  

• 630e26f1
  by Tristan Van Berkom at 2019-01-10T20:03:52Z

  testutils/runcli.py: Added Result.get_start_order()
  
  Added a function to report the list of elements which appeared in
  a given queue in the order of their first appearance.
  

• d1d7de57
  by Tristan Van Berkom at 2019-01-11T04:13:16Z

  tests/frontend/order.py: Adding tests ensuring stable element processing order
  
  For each sample project and expected result order, this test ensures that:
  
    * bst show --deps plan: Always shows the expected build order.
  
    * bst source fetch: Always executes in the expected build order.
  
    * bst build: Always builds in the expected order (disabled).
  
  The build part of the test is currently disabled as the scheduler
  seems to be messing with the order even in a `--builders 1` scenario.
  

• 276b8d48
  by Tristan Van Berkom at 2019-01-11T04:42:24Z

  Merge branch 'tristan/element-processing-order' into 'master'
  
  Make build plan element list stable
  
  See merge request BuildStream/buildstream!1058

• fc3dcec8
  by Angelos Evripiotis at 2019-01-11T16:45:00Z

  dev-requirements: pytest-cov==2.6.1 for '--no-cov'
  
  Bump the version of pytest-cov, so that we pick up the fix for
  the '--no-cov' option breaking in pytest 4.0:
  
      https://github.com/pytest-dev/pytest-cov/pull/230
  
  This was the error you would get prior to this version:
  
      _pytest.warning_types.RemovedInPytest4Warning: config.warn has been
      deprecated, use warnings.warn instead
  
  Although it says 'deprecated', it did result in
  termination with stack trace.
  

• f86b7ff3
  by Angelos Evripiotis at 2019-01-11T16:45:00Z

  contributing: '--last-failed' and '--no-cov' tips
  
  To help new folks get to grips with pytest, add some tips on other
  frequently used features.
  

• d983f231
  by Angelos Evripiotis at 2019-01-11T17:19:06Z

  Merge branch 'aevri/tox_no_cov' into 'master'
  
  pytest-cov==2.6.1, contributing: '--last-failed' and '--no-cov' tips
  
  See merge request BuildStream/buildstream!1059

• 829a2b93
  by Tristan Van Berkom at 2019-01-12T21:32:13Z

  _scheduler/queues/queue.py: Put elements in the skip list, not jobs
  
  Minor correction, looks like we're not observing this queue
  otherwise we'd be seeing crashes.
  

• 993e30ae
  by Tristan Van Berkom at 2019-01-12T22:02:55Z

  Merge branch 'tristan/fix-skipped-elements' into 'master'
  
  _scheduler/queues/queue.py: Put elements in the skip list, not jobs
  
  See merge request BuildStream/buildstream!1063

• 32732e01
  by Javier Jardón at 2019-01-14T09:04:01Z

  .gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25
  

• 4f5f1184
  by Valentin David at 2019-01-14T09:40:56Z

  Merge branch 'jjardon/freedesktop-sdk_latest' into 'master'
  
  .gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25
  
  Closes #858
  
  See merge request BuildStream/buildstream!1064

• bb80a2b8
  by Chandan Singh at 2019-01-14T14:30:15Z

  .gitlab-ci.yml: Use newer image for overning aarch tests
  
  During recent reorganizing of `.gitlab-ci.yml` to work with `tox`, seems
  like we missed to update the image used by the overnight aarch tests,
  meaning that they currently fail due to `tox` being missing from them.
  
  While these tests will be skipped on MRs usually, here is an example of
  what the tests look like if they are actually run with this change:
  https://gitlab.com/BuildStream/buildstream/-/jobs/145449561.
  
  Fixes #859.
  

• 10b3ee62
  by Chandan Singh at 2019-01-14T14:57:08Z

  Merge branch 'chandan/fix-overnight-aarch' into 'master'
  
  Chandan/fix overnight aarch
  
  Closes #859
  
  See merge request BuildStream/buildstream!1065

• 542cdaf0
  by Jürg Billeter at 2019-01-14T18:46:57Z

  _frontend/cli.py: Also check original args for --config in bash completion
  
  The path of the config file generated by testutils for completion tests
  is passed as regular argument, not via COMP_WORDS. Use that config file
  in complete_artifact() to ensure the test uses the right artifact
  directory.
  

• ff666e76
  by James Ennis at 2019-01-14T18:46:57Z

  completions.py: Add a test for our artifact ref autocompletions
  
  Since the artifact subcommand group has been created, we have been able
  to tab complete both element names and artifact refs as arguments to
  the artifact subcommands (e.g. bst artifact log), this commit adds a
  test for this.
  

• 273b0f55
  by Tristan Van Berkom at 2019-01-14T20:24:57Z

  Merge branch 'jennis/add_artifacts_completion' into 'master'
  
  completions.py: Add a test for our artifact ref autocompletions
  
  See merge request BuildStream/buildstream!1054

• 2e3c2ea2
  by Javier Jardón at 2019-01-14T22:26:04Z

  .gitlab-ci.yml: Use latest bst-external
  
  which include flatpak_repo plugin needed to build fdsdk
  

• d60d2e31
  by Javier Jardón at 2019-01-15T00:39:22Z

  Merge branch 'jjardon/bst_external_0_9' into 'master'
  
  .gitlab-ci.yml: Use latest bst-external
  
  Closes #861
  
  See merge request BuildStream/buildstream!1068

• 605836c1
  by Chandan Singh at 2019-01-15T00:57:05Z

  .gitlab-ci.yml: Add job to attempt to update dependencies
  
  This will help us check if BuildStream is working with the latest
  version of dependencies, as per our constraints. This job is allowed to
  fail but its failure should signal that we need to add stricter
  constraints in some of our `.in` requirements files.
  

• 006370af
  by Chandan Singh at 2019-01-15T01:21:33Z

  Merge branch 'chandan/reqs-update-test' into 'master'
  
  .gitlab-ci.yml: Add job to attempt to update dependencies
  
  See merge request BuildStream/buildstream!1038

• 4b544555
  by Chandan Singh at 2019-01-15T21:28:40Z

  .gitlab-ci.yml: Add tests for python 3.7
  
  We already have tests for python 3.5 and 3.6 but not 3.7.
  
  Fixes https://gitlab.com/BuildStream/buildstream/issues/838.
  

• 80fe0d9a
  by Javier Jardón at 2019-01-15T22:48:22Z

  Merge branch 'chandan/python37-tests' into 'master'
  
  .gitlab-ci.yml: Add tests for python 3.7
  
  Closes #838
  
  See merge request BuildStream/buildstream!1074

• c91784ab
  by Chandan Singh at 2019-01-15T22:49:01Z

  conftest.py: Don't use deprecated get_marker() function
  
  Starting from `pytest` version 4.1.0, `Node.get_marker()` has been
  removed, and hence our tests break when running with newer versions of
  `pytest`. It was deprecated since a while back but it has recently been
  removed completely. Use `get_closest_marker()` as a replacement that is
  suggested in the changelog, and seems to work fine for our use case.
  
  See https://github.com/pytest-dev/pytest/pull/4564 for more context on
  the upstream issue.
  
  One way of verifying this change is that this should fix the recently
  added `tests-fedora-update-deps` job, that was failing before due to
  this issue.
  

• ecae4d73
  by Javier Jardón at 2019-01-15T23:23:41Z

  Merge branch 'chandan/fix-pytest-get-marker' into 'master'
  
  conftest.py: Don't use deprecated get_marker() function
  
  See merge request BuildStream/buildstream!1073

• 0eac4008
  by Valentin David at 2019-01-16T10:04:57Z

  buildstream/_gitsourcebase.py: Reduce git history for git describe.
  
  Found during #833.
  
  `git rev-list --boundary tag..HEAD` unfortunately gives boundaries
  that are deeper than should when there is a merge commit between `tag`
  and `HEAD`. The common ancestory of the two parents of the merge is
  a boundary instead of the parent of the branch that is not traversed.
  
  `--ancestry-path` fixes this issue by restricting `git` traversing
  those branches.
  

• a405e08f
  by Valentin David at 2019-01-16T10:04:57Z

  buildstream/_gitsourcebase.py: Fix case where HEAD is tagged
  
  `git rev-list --boundary HEAD..HEAD` does not return any boundary.
  So in this case we need to manually tag the HEAD as a boundary.
  

• c0631d48
  by Valentin David at 2019-01-16T10:33:52Z

  Merge branch 'valentindavid/git-reduced-history' into 'master'
  
  buildstream/_gitsourcebase.py: Reduce git history for git describe.
  
  See merge request BuildStream/buildstream!1069

• b608ac86
  by James Ennis at 2019-01-16T11:05:44Z

  _context.py: Doc fix: get_toplevel_project() returns object not list
  

• 97a3beb6
  by James Ennis at 2019-01-16T11:05:44Z

  _context.py: Add documentation to get_workspaces() command
  

• d6587aa0
  by James Ennis at 2019-01-16T11:33:21Z

  Merge branch 'jennis/doc_fixes_in_context' into 'master'
  
  Small documentation/comment fixes in context.py
  
  See merge request BuildStream/buildstream!1072

• 2683f98a
  by Raoul Hidalgo Charman at 2019-01-16T11:55:07Z

  _cas: Rename artifactcache folder and move that to a root module
  
  Other components will start to reply on cas modules, and not the artifact cache
  modules so it should be organized to reflect this.
  
  All relevant imports have been changed.
  
  Part #802
  

• d2cc4798
  by Raoul Hidalgo Charman at 2019-01-16T11:55:07Z

  casremote.py: Move remote CAS classes into its own file
  
  Part of #802
  

• 76f67483
  by Raoul Hidalgo Charman at 2019-01-16T11:55:07Z

  cas: move remote only functions to CASRemote
  
  List of methods moved
  * Initialization check: made it a class method that is run in a subprocess, for
    when checking in the main buildstream process.
  * fetch_blobs
  * send_blobs
  * verify_digest_on_remote
  * push_method
  
  Part of #802
  

• 6c428bc9
  by Jürg Billeter at 2019-01-16T12:56:38Z

  Merge branch 'raoul/cas-refactor' into 'master'
  
  Cas refactor
  
  See merge request BuildStream/buildstream!1071

• 2648b0bd
  by Angelos Evripiotis at 2019-01-16T14:44:25Z

  bst-docker-import: fix Chandan's name

• 780d0f3a
  by Angelos Evripiotis at 2019-01-16T15:13:06Z

  Merge branch 'chadnan' into 'master'
  
  bst-docker-import: fix Chandan's name
  
  See merge request BuildStream/buildstream!1076

• 4ca37dcd
  by Tristan Van Berkom at 2019-01-16T15:47:21Z

  source-determinism.py integration test: Dont use integration_cache fixture
  
  This was just deadcode, it's not really used here.
  

• 6286d820
  by Tristan Van Berkom at 2019-01-16T15:47:21Z

  conftest.py: Use different artifact directory for integration tests
  
  To ensure we can run integration tests in parallel, use a tempdir
  for the artifact cache of each separate integration test run.
  
  This makes it possible to run multiple full test runs including
  integration tests in parallel under detox.
  

• 471af316
  by Tristan Van Berkom at 2019-01-16T16:32:52Z

  Merge branch 'tristan/detoxing-integration-tests' into 'master'
  
  Make integration tests parallelizable
  
  See merge request BuildStream/buildstream!1077

• 081e8426
  by James Ennis at 2019-01-16T17:58:41Z

  setup.py: Import server main from _cas/casserver.py
  
  Since the CAS refactor, we have not been able to execute
  bst-artifact-server. This commit ensures that we can.
  
  This fix closes #867
  

• e385660c
  by James Ennis at 2019-01-16T18:33:20Z

  Merge branch 'jennis/fix_bst_artifact_server' into 'master'
  
  Ensure that we can execute `bst-artifact-server`
  
  Closes #867
  
  See merge request BuildStream/buildstream!1079

• 6991d6d6
  by Tristan Van Berkom at 2019-01-16T19:28:14Z

  _scheduler: Refactor of queues and resources.
  
  This branch makes the following changes:
  
    * jobs/job.py: No longer stores any interested resource list
  
      Jobs are ephemeral again, they only ever exist while they
      are running.
  
    * queues/queue.py: Revert to only handling lists of elements
  
      Elements pass through the queues, Queue.harvest_jobs()
      replaces Queue.pop_ready_jobs() and now the Queue stops
      creating jobs as soon as there are not enough resources
      for the job.
  
      Also removed unused `prepare()` abstract method.
  
    * queues/buildqueue.py: Adapt the part where we launch a job
  
      This part needs to be reworked anyway, just touch it up for
      now so that it doesnt break with the surrounding changes.
  
    * jobs/{cachesize,cleanup}job.py: Expose uniform complete callback
  
      Allows the scheduler to manage resource deallocation for these
      two job completions as a custom thing, at the same phase
      that the Queues take care of their own resource deallocation.
  
    * resources.py: No longer has knowledge of the job
  
      Since jobs are ephemeral, they are not a suitable place
      to store the resource identifiers, these must be provided
      by the callers wherever needed.
  
      Now the main Resources object is owned by the Scheduler
      but shared with Queues, each take care of managing the
      resources of the jobs they create through the same
      resource API.
  
    * scheduler.py: Reverted to only creating jobs on demand
  
      This changes the flow of the scheduler such that whenever
      jobs complete, the queues are interrogated for as many
      jobs which can run at the moment but not more; and this
      completely removes the waiting list.
  
      For the internal cache management jobs, we handle this
      with a little state instead of having a waiting list
      and only launch when the resources permit it.
  
  By abolishing the scheduler waiting list and creating jobs
  on demand, we fix the order of element processing and consequently
  fix issue #712.
  

• 04ba59bc
  by Tristan Van Berkom at 2019-01-16T19:28:14Z

  tests/frontend/order.py: Enable the test for build and fix the fetch tests
  
  With the scheduler changes, fetch jobs get automatically skipped
  so the output is changed, using a separate repo for each element
  fixes the test such that every fetch job gets a job launched.
  

• 32bdded8
  by Tristan Van Berkom at 2019-01-16T19:28:14Z

  _artifactcache/artifactcache.py: Rephrase failure message
  
  It was saying "There is not enough space to build the given element.",
  this makes me think the error is associated to a specific element, but
  this does not make sense to show up in a cleanup task.
  
  Instead say "There is not enough space to complete the build.", which
  should be more clear that even after cleaning up there is not enough
  space.
  

• 17f6e5a8
  by Tristan Van Berkom at 2019-01-16T20:10:30Z

  Merge branch 'tristan/element-processing-order' into 'master'
  
  Scheduler refactor, fix processing order
  
  Closes #712
  
  See merge request BuildStream/buildstream!1067

• 19adc9e5
  by Tristan Van Berkom at 2019-01-16T20:48:10Z

  element.py: Cleanup temporary staging directories on termination
  
  Use utils._tempdir() which in turn uses _signals.terminator() for
  this purpose.
  
  This also changes utils._tempdir() to use utils._force_rmtree()
  so that it is safe to use for cleaning up the staging directory.
  
  This patch fixes orphaned temporary directories being left behind
  in the artifact cache directory at forceful termination time.
  

• 5e9598c5
  by Tristan Van Berkom at 2019-01-16T21:17:26Z

  Merge branch 'tristan/cleanup-staging-tempdir' into 'master'
  
  element.py: Cleanup temporary staging directories on termination
  
  See merge request BuildStream/buildstream!1080

• 5ea2f59d
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate completions test to tests/frontend
  

• 1eafc25c
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate `--except` testing to tests/frontend/show.py
  
  The tests/pipeline directory will be removed, and this test
  is the better of the two but also redundant with the one in
  tests/frontend/show.
  
  Renamed existing test in show.py to `test_show_except_simple`
  and added the better test as `test_show_except` below it.
  

• b6183123
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate virtual directory storage test to internals directory
  
  Grouping bits of internal testing together here
  

• 9b6cc972
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate storage test to the internals directory
  

• 7e2300ce
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate yaml test to the internals directory
  

• 353a682f
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate utils tests into internals directory
  

• 3324490e
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate context test into the internals directory
  

• 99ea157a
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate plugin factory test to internals directory
  

• e825bdd3
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate plugin loading test to internals directory
  

• c7f69bcc
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Removing tests/plugins/third_party.py
  
  This tests exactly the same thing that is tested in
  tests/internals/pluginfactory.py (the new location
  of tests/plugins/basics.py).
  

• 42150422
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Rename `plugins` directory to `elements` directory
  
  Now that the remaining test "filter.py" in the plugins directory
  tests a specific element, it makes sense to create a place for
  testing elements, just like we do for sources.
  

• f712aaa5
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate internal Loader basic test to internals directory
  

• 8d676a84
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate dependencies test to tests/format
  
  This used to be an internal test, converted this to use the `cli` fixture.
  

• bbde1c21
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate dependency order/iteration testing to the format tests
  
  Created new `tests/format/iteration.py` which tests the order in
  which elements are iterated over in various scopes in a loaded
  data model.
  

• 8f2b4e9a
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate test for load_ref() unsupporting plugins into format/project.py
  
  The tests/format/project.py test already has some tests about how
  we error gracefully for bad plugins and bad plugin configurations,
  lets put it there rather than tests/pipeline/load.py which we
  will remove.
  

• de881c2a
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate preflight error handling check to tests/format/project.py
  
  This is where other load time related plugin error handling is
  checked, and is the last thing to remove in the `tests/pipeline`
  directory.
  

• a7fbd900
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Renaming some tests in tests/format/project.py
  
  Remove some redundancy from the test lines.
  

• aa997450
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Removing tests/pipeline/load.py
  
  The remaining test simply loads a project with one element
  and asserts a value on it. This is already sufficiently tested
  in tests/format/project.py.
  

• 561eddf1
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate protected variable handling tests to tests/format/variables.py
  

• 3bf40cf3
  by Tristan Van Berkom at 2019-01-16T23:35:21Z

  tests: Migrate junctions test to tests/format/junctions.py
  
  This is the directory for all things related to loading.
  

• d34a4fd1
  by Tristan Van Berkom at 2019-01-17T00:26:43Z

  Merge branch 'tristan/organizing-tests' into 'master'
  
  General refactor in tests directory
  
  See merge request BuildStream/buildstream!1062

• d212cdfa
  by Tristan Van Berkom at 2019-01-17T00:28:03Z

  sandbox/sandbox.py: Display failed commands in the detail string
  
  We should only display commands in detail strings, not in the
  message texts.
  
  This also updates tests/integration/sandbox-bwrap.py to expect
  the new message string which only contains the command exit status
  and not the whole command itself, this does not alter the validity
  of the text case which is checking that we can obtain the expected
  return value.
  

• ce91ce5d
  by Tristan Van Berkom at 2019-01-17T05:05:16Z

  Merge branch 'tristan/error-message-regression' into 'master'
  
  sandbox/sandbox.py: Display failed commands in the detail string
  
  See merge request BuildStream/buildstream!1081

• 63b02f7c
  by James Ennis at 2019-01-17T15:13:27Z

  _profile.py: Add timestamp to the logs
  

• f64ee46f
  by James Ennis at 2019-01-17T17:02:28Z

  _profile.py: Write binaries as well as logs
  
  Private class methods which write the logs and write the binaries
  have been added to Profile. The binaries are able to be used
  by various visualisation tools.
  

• 2cb37a7e
  by James Ennis at 2019-01-18T09:31:24Z

  Merge branch 'jennis/profiling_outputs_binaries' into 'master'
  
  Make the profiler output binaries (which can be used to visualise the data)
  
  See merge request BuildStream/buildstream!1082

• 56ec33cc
  by Valentin David at 2019-01-18T11:14:41Z

  .gitlab-ci.yml: Add overnight tests logs to artifacts.
  

• ad2df651
  by Javier Jardón at 2019-01-18T12:01:56Z

  Merge branch 'valentindavid/overnight-tests-logs-artifacts' into 'master'
  
  .gitlab-ci.yml: Add overnight tests logs to artifacts.
  
  See merge request BuildStream/buildstream!1084

• f874295f
  by Tristan Van Berkom at 2019-01-18T15:59:28Z

  bzr source plugin: Use lock files to avoid corrupting the source cache
  
  This patch by itself fixes #868
  

• 42a2fe3c
  by Tristan Van Berkom at 2019-01-18T15:59:28Z

  bzr source plugins: Remove all of the atomic backup directory business logic
  
  Follow up of last commit which uses exclusive locking to
  protect bzr operations instead.
  

• a8713ed2
  by Tristan Van Berkom at 2019-01-18T15:59:28Z

  testutils/runcli.py: Added cli.get_element_states()
  
  With get_element_state(), you need to invoke BuildStream once
  for every element state you want to observe in a pipeline.
  
  The new get_element_states() reports a dictionary with
  the element state hashed by element name and is better to use
  if you have more than one element to observe the state of.
  

• a895cb2a
  by Tristan Van Berkom at 2019-01-18T15:59:28Z

  tests/frontend/track.py: Use test_track_recurse() to stress test Sources
  
  This causes multiple source instances to interact with the same
  backing data store at the same time, increasing the likelyhood
  of triggering issues around concurrent access.
  
  This more reliably triggers issue #868
  

• 4236bcc7
  by Tristan Van Berkom at 2019-01-18T16:35:23Z

  Merge branch 'tristan/fix-bzr-race' into 'master'
  
  Fix bzr race conditions
  
  Closes #868
  
  See merge request BuildStream/buildstream!1083

• 951a8df1
  by Angelos Evripiotis at 2019-01-18T18:02:22Z

  cli.py: add a hint about '--' to 'bst shell' help

• 9911023f
  by Angelos Evripiotis at 2019-01-18T18:56:21Z

  Merge branch 'aevri/shell_separator_hint' into 'master'
  
  cli.py: add a hint about '--' to 'bst shell' help
  
  See merge request BuildStream/buildstream!1078

• c9ce89d2
  by Tristan Van Berkom at 2019-01-18T19:36:26Z

  _cas/cascache.py: Cleanup directories when removing refs
  
  With out this, empty directories in the refs/heads directory just
  grow unconditionally.
  

• c536ab6a
  by Tristan Van Berkom at 2019-01-18T19:36:26Z

  tests/artifactcache/expiry.py: Test refs directory cleanup
  
  Enhance the test which checks removal of the extract directories
  with an additional check that the ref directories are cleaned up
  when removing artifacts.
  

• 99699ffc
  by Tristan Van Berkom at 2019-01-18T19:36:26Z

  utils.py: Added _tempnamedfile()
  
  When used in a child process in BuildStream, this should be
  used instead of tempfile.NamedTemporaryFile() directly, otherwise
  we fail to cleanup the file on SIGTERM.
  

• 8ce483d4
  by Tristan Van Berkom at 2019-01-18T19:36:26Z

  _cas/cascache.py: Use utils._tempdir() and utils._tempnamedfile()
  
  The direct usage of tempfile.TemporaryDirectory() and
  tempfile.NamedTemporaryFile() here causes leakage of any temporary
  data when the process operating on temporary data is terminated
  with SIGTERM.
  
  Using the utilities ensures that trash is not left behind in
  ~/.cache/buildstream/artifacts/tmp when the user terminates
  BuildStream with ^C.
  

• 73c7252d
  by Tristan Van Berkom at 2019-01-18T20:57:42Z

  Merge branch 'tristan/cas-cleanup-improve' into 'master'
  
  CASCache cleanup improvements
  
  See merge request BuildStream/buildstream!1087

• adfb9291
  by Tristan Van Berkom at 2019-01-18T21:23:05Z

  tests/testutils/python_repo.py: Use subprocess to run sdist
  
  The current approach using setuptools.sandbox.run_setup() was
  causing a spurious (but highly frequent) failure where setuptools
  gets mixed up with it's manipulation of sys.modules and hits a
  RuntimeError as a dictionary changes size while being iterated over.
  
  For instance: https://gitlab.com/BuildStream/buildstream/-/jobs/147967307
  
  Since this already happens in an isolated virtual environment created
  by tox, we should not need additional sandboxing here from setuptools,
  and launching this as a subprocess will be safer.
  

• d114a6bd
  by Tristan Van Berkom at 2019-01-18T22:07:22Z

  Merge branch 'tristan/fix-pip-source-test' into 'master'
  
  tests/testutils/python_repo.py: Use subprocess to run sdist
  
  See merge request BuildStream/buildstream!1090

• bf591ade
  by Chandan Singh at 2019-01-18T22:09:04Z

  Derive import plugin from Element instead of BuildElement
  
  The `import` element is not really a build element. The main purpose of
  the `BuildElement` class is to allow users to run `build-commands`,
  `install-commands` etc. But, `import` does not run such commands.
  Moreover, we already override all the methods provided by
  `BuildElement`. So it only makes it confusing to have it derived from
  `BuildElement` class when it is not a build element.
  
  So, derive it from the base `Element` class instead.
  

• 2233a532
  by Chandan Singh at 2019-01-18T22:46:41Z

  Merge branch 'chandan/import-is-not-buildelement' into 'master'
  
  Derive import plugin from Element instead of BuildElement
  
  See merge request BuildStream/buildstream!1089

• aa3411f9
  by Tristan Maat at 2019-01-18T22:47:46Z

  testutils/runcli.py: Allow removing artifacts from arbitrary dirs
  
  `remove_artifact_from_cache` used a hard-coded path to remove
  artifacts, which wasn't sufficient for integration tests.
  

• fff882fe
  by Tristan Maat at 2019-01-18T22:47:46Z

  widget.py: Avoid "showing 0 lines" messages when there are no lines
  
  This happened when bst is invoked with --message-lines 0 or
  --error-lines 0, and was arguably a little too verbose (the user
  explicitly asked us not to show them any lines, after all).
  
  Fixes #779
  

• e230dedb
  by Tristan Van Berkom at 2019-01-18T23:23:35Z

  Merge branch 'tlater/message-lines' into 'master'
  
  Avoid "showing 0 lines" messages when we're asked to show no lines
  
  Closes #779
  
  See merge request BuildStream/buildstream!1031

• 43797617
  by Richard Maw at 2019-01-21T10:41:37Z

  _gitsourcebase.py: Fetch with explicit refspecs
  
  Old versions of git lack --force and --tags,
  but the same effect can be had by specifying refspecs.
  

• ce8dab0f
  by Richard Maw at 2019-01-21T10:41:37Z

  tests/frontend/workspace.py: Skip test_open_multi_unwritable when root
  
  The test assumes that a directory with write permission removed isn't writable,
  this isn't the case if the process running the tests has CAP_DAC_OVERRIDE
  which is common when running as root.
  

• 50165081
  by Richard Maw at 2019-01-21T10:41:37Z

  tests/sources/git.py: Skip tests that assume too new a git
  
  test_track_invalid_submodule depends on being able to remove a submodule
  by `git rm $submoduledir`, but old versions of git don't update .gitmodules
  so BuildStream still thinks there's a submodule present.
  
  For expediency the test is skipped rather than changed to manually remove
  the entry from .gitmodules if git hasn't done it,
  since in the common case git is new enough to do that itself.
  
  test_git_describe expects --first-parent to find another tag,
  but `bst track` will gracefully degrade if the option doesn't work
  so a different history will be retained with old versions of git.
  
  It's of marginal benefit to add additional cruft
  to test for different output on old versions of git that won't persist forever.
  

• 8677a256
  by Benjamin Schubert at 2019-01-21T10:41:37Z

  Add a Centos runner
  
  Centos is apparently different enough from fedora when running tests.
  

• 33782865
  by Valentin David at 2019-01-21T11:47:18Z

  Merge branch 'richardmaw/centos-oldgit-test-fixes' into 'master'
  
  Fix CentOS
  
  Closes #833
  
  See merge request BuildStream/buildstream!1085

• c00b3782
  by Chandan Singh at 2019-01-21T19:37:53Z

  CONTRIBUTING.rst: Fix formatting of link to pip docs
  
  The link explaining python's requirements files was formatted
  incorrectly (missing underscore at the end). This resulted in the text
  being rendered literally, as opposed to being converted into a
  hyperlink.
  

• 638ceb8a
  by Chandan Singh at 2019-01-21T20:16:38Z

  Merge branch 'chandan/fix-contrib-link' into 'master'
  
  CONTRIBUTING.rst: Fix formatting of link to pip docs
  
  See merge request BuildStream/buildstream!1094

• 383142e3
  by James Ennis at 2019-01-22T12:32:43Z

  using_commands.rst: Split out top level commands and subcommand groups
  

• ac135333
  by James Ennis at 2019-01-22T12:32:43Z

  using_commands.rst: Add the source checkout command
  

• a3fc350f
  by James Ennis at 2019-01-22T12:32:43Z

  Move push and pull to the new artifact subcommand group
  
  This commit also ensures that if we try to use the 'old' commands,
  BuildStream will fail and instruct the user to use the new command.
  

• 9eefe863
  by James Ennis at 2019-01-22T12:32:43Z

  cli: Add artifact checkout subcommand
  
  'artifact checkout' has slightly different behaviour from 'checkout',
  that is, either '--directory' or '--tar' are now required options.
  This is a step towards allowing checkout to take multiple args.
  

• fbd15939
  by James Ennis at 2019-01-22T12:32:43Z

  Mark 'old' checkout command as obsolete
  
  This commit marks 'bst checkout' as a 'hidden' command. If used,
  the user will be prompted to use the new 'bst artifact checkout'
  command.
  
  All tests which used 'bst checkout' have been modified to use
  the new artifact sub-command.
  
  This partially solves #822.
  

• 9e8034e9
  by James Ennis at 2019-01-22T12:32:43Z

  Add NEWS entry describing the deprecation of checkout, pull and push
  

• 1bccf1d2
  by James Ennis at 2019-01-22T12:38:40Z

  using_commands.rst: Split out the artifact subcommands
  

• d9072371
  by James Ennis at 2019-01-22T12:38:40Z

  using_commands.rst: Add artifact log subcommand
  

• 077d5a96
  by James Ennis at 2019-01-22T12:41:29Z

  Change bst checkout/pull/push references to bst artifact checkout/pull/push
  

• ea2fbe4d
  by James Ennis at 2019-01-22T13:02:14Z

  man: Regenerate all of our man pages
  
  Due to an upstream click_man issue:
  https://github.com/click-contrib/click-man/issues/10
  generating the man pages with our current setup.py only generates
  a man page for bst-artifact-server, our first entry point.
  I then had to remove this entry point from setup.py and regenerate to
  obtain man pages for the commands in cli.py
  

• 9c2a6b93
  by James Ennis at 2019-01-22T15:08:17Z

  Merge branch 'jennis/migrate_pull_push_commands' into 'master'
  
  Move push/pull/checkout to the artifact subcommand group
  
  See merge request BuildStream/buildstream!1045

• 35377522
  by James Ennis at 2019-01-22T15:28:13Z

  cli.py: Add an obsoletion note to our deprecated commands.
  
  Click 7.0 allows us to declare commands as 'hidden'.
  However, sphinx-click and click-man still generate these hidden
  commands in the documentation and man pages, respectively.
  
  This is a stop gap solution until the upstream issues (tracked in
  issues #879 and #881) have been addressed.
  

• 76148785
  by James Ennis at 2019-01-22T16:15:20Z

  Merge branch 'jennis/add_obsoletion_note' into 'master'
  
  Add an obsoletion note to our deprecated commands.
  
  See merge request BuildStream/buildstream!1097

• 1443c542
  by Tristan Van Berkom at 2019-01-22T16:26:56Z

  _scheduler: Fix dont display a failure for terminated jobs
  
  This fixes a recent regression introduced in c2fc2a5ea
  

• 6e5c9987
  by Tristan Van Berkom at 2019-01-22T17:14:50Z

  Merge branch 'tristan/fix-terminated-status' into 'master'
  
  _scheduler: Fix dont display a failure for terminated jobs
  
  See merge request BuildStream/buildstream!1096

• 3315b9a1
  by Tristan Van Berkom at 2019-01-22T18:46:24Z

  tests/integration/pullbuildtrees.py: Fix the non-integration case.
  
  This test has one test case which is marked as an integration test,
  and the other is not an integration test, but was using the integration
  cli. The integration cli does not work correctly if not run in
  integration mode.
  
  This was causing an error locally in conftest.py when trying to
  create a tmpdir inside a nonexisting integration cache directory.
  

• 56e857f4
  by Tristan Van Berkom at 2019-01-22T19:59:57Z

  Merge branch 'tristan/fix-pullbuildtrees-test' into 'master'
  
  tests/integration/pullbuildtrees.py: Fix the non-integration case.
  
  See merge request BuildStream/buildstream!1098

• 3895571a
  by Chandan Singh at 2019-01-23T00:45:47Z

  tox.ini: Unpin sphinx dependency
  
  https://github.com/rtfd/sphinx_rtd_theme/pull/672 has been fixed
  upstream, and the newer versions of `sphinx_rtd_theme` do not break
  search functionality with Sphinx >= 1.8.
  

• 528e8ed7
  by Chandan Singh at 2019-01-23T01:31:49Z

  Merge branch 'chandan/unpin-sphinx' into 'master'
  
  tox.ini: Unpin sphinx dependency
  
  See merge request BuildStream/buildstream!1093

• 0098a900
  by Tristan Van Berkom at 2019-01-23T13:57:53Z

  tests: Migrated cache quota test into artifactcache/cache_size.py
  
  Instead of sitting mysteriously alone in internals/utils.py
  

• 9be31641
  by Tristan Van Berkom at 2019-01-23T13:57:53Z

  _artifactcache.py: Raise ArtifactError() when quota size exceeds disk space.
  
  This is not an error related to loading data, like a parse error
  in the quota specification is, but a problem raised by the artifact
  cache - this allows us to assert more specific machine readable
  errors in test cases (instead of checking the string in stderr, which
  this patch also fixes).
  
  This also removes a typo from the error message in the said error.
  
    * tests/artifactcache/cache_size.py
  
      Updated test case to expect the artifact error, which consequently
      changes the test case to properly assert a machine readable error
      instead of asserting text in the stderr (which is the real, secret
      motivation behind this patch).
  
    * tests/artifactcache/expiry.py: Reworked test_invalid_cache_quota()
  
      Now expect the artifact error for the tests which check configurations
      which create caches too large to fit on the disk.
  

• 3a6e953f
  by Tristan Van Berkom at 2019-01-24T01:59:13Z

  Merge branch 'tristan/insufficient-storage-error' into 'master'
  
  Tristan/insufficient storage error
  
  See merge request BuildStream/buildstream!1102

• 68339b19
  by Tristan Van Berkom at 2019-01-24T05:01:55Z

  tests/testutils/runcli.py: Make get_element_states() take a list of targets
  
  Instead of a single target, we can always provide a single target
  in a list.
  

• 6ecc2b0a
  by Tristan Van Berkom at 2019-01-24T05:01:55Z

  tests/artifactcache/expiry.py: Refactored to use get_element_states()
  

• 1140aed5
  by Tristan Van Berkom at 2019-01-24T05:01:55Z

  tests/elements/filter.py: Refactored to use get_element_states()
  

• b7ea8b74
  by Tristan Van Berkom at 2019-01-24T05:01:55Z

  tests/sources/remote.py: Refactored to use get_element_states()
  

• c2c004f9
  by Tristan Van Berkom at 2019-01-24T05:01:55Z

  tests/frontend/pull.py: Refactored to use get_element_states()
  

• 21b2958b
  by Tristan Van Berkom at 2019-01-24T05:01:55Z

  tests/frontend/push.py: Refactored to use get_element_states()
  

• 83fcaa9f
  by Tristan Van Berkom at 2019-01-24T05:01:56Z

  tests/frontend/track.py: Refactored to use get_element_states()
  

• 46eb3018
  by Tristan Van Berkom at 2019-01-24T05:01:56Z

  tests/frontend/workspace.py: Refactored to use get_element_states()
  

• 341b131b
  by Tristan Van Berkom at 2019-01-24T06:13:57Z

  Merge branch 'tristan/test-element-states' into 'master'
  
  Reduce number of calls to `bst show` in tests
  
  See merge request BuildStream/buildstream!1103

• ef2b4648
  by Tristan Van Berkom at 2019-01-24T06:14:20Z

  tests/frontend/track.py: test_track_error_cannot_write_file() fixup
  
  This tests how BuildStream reacts when it fails to write the tracking
  results to the element files or project.refs file, which is an operation
  that plugins do not play a part in.
  
  As such, removing the per repo kind parameterization from this test
  as multiple runs are redundant here.
  

• 40c18174
  by Tristan Van Berkom at 2019-01-24T07:00:50Z

  Merge branch 'tristan/track-test-reduce' into 'master'
  
  test_track_error_cannot_write_file() fixup
  
  See merge request BuildStream/buildstream!1104

• 5b0bba85
  by Jonathan Maw at 2019-01-24T10:15:06Z

  Test that tracking in workspaces actually works
  
  Previously, it merely tested that buildstream did not fall other,
  rather than whether it did anything useful.
  

• 58d7d722
  by Jonathan Maw at 2019-01-24T10:18:01Z

  tests: Test that fetching an open workspace will fetch its dependencies
  
  Previously, there was no way of detecting whether fetching happened, as
  an element with an open workspace will not be fetched.
  

• 24bd8994
  by Jürg Billeter at 2019-01-24T13:36:33Z

  Merge branch 'jonathan/test-missing-workspace-guessing' into 'master'
  
  Add tests to cover reinstated support for guessing targets
  
  See merge request BuildStream/buildstream!1042

• e03dd5fc
  by Jürg Billeter at 2019-01-24T13:37:28Z

  _frontend/cli.py: Guess element also for bst build --all
  
  There is no reason to disallow guess_element() for bst build --all.
  

• 3850274b
  by Phillip Smyth at 2019-01-24T13:37:28Z

  projectconfig.yaml: Add key for default targets
  
  _versions.py: Bump format version
  

• ab72d384
  by Jürg Billeter at 2019-01-24T13:37:28Z

  _project.py: Add get_default_target() and get_default_targets() methods
  
  _frontend/cli.py: Use new methods.
  
  Based on patches by Phillip Smyth.
  

• ee2d8434
  by Jürg Billeter at 2019-01-24T13:37:28Z

  _stream.py: Add ignore_junction_targets parameter
  
  This filters out junctions from the list of targets.
  

• 2f175f0a
  by Jürg Billeter at 2019-01-24T13:37:28Z

  _frontend/cli.py: Ignore junctions in default targets where appropriate
  
  Junctions cannot be built, pulled, or pushed. Specifying a junction on
  the command line for these commands will result in an error. However,
  junctions may be in the list of default targets, so they need to be
  ignored for build, pull, and push commands.
  

• 5a351dee
  by Jürg Billeter at 2019-01-24T13:38:50Z

  format_project.rst: Add documentation for default targets
  

• 77414518
  by Phillip Smyth at 2019-01-24T13:38:50Z

  tests/frontend: Add default target tests for bst show and build
  

• ce1c10ec
  by Jürg Billeter at 2019-01-24T13:38:50Z

  tests/frontend/fetch.py: Add default target test for bst source fetch
  

• 99b5c0af
  by Jürg Billeter at 2019-01-24T13:38:50Z

  tests/frontend/pull.py: Add default target test for bst push/pull
  

• 3642c8e7
  by Jürg Billeter at 2019-01-24T13:38:50Z

  tests/frontend/buildcheckout.py: Add default target test with junction
  
  Test that `bst build` does not fail in a project where the list of
  default targets includes a junction (junctions cannot be built).
  

• 50c5159f
  by Phillip Smyth at 2019-01-24T13:38:50Z

  NEWS: Add entry for default target feature
  

• 05587f22
  by Jürg Billeter at 2019-01-24T15:10:08Z

  Merge branch 'issue-638-validate-all-files' into 'master'
  
  Add support for default targets
  
  See merge request BuildStream/buildstream!925

• 3e36e363
  by Tristan Van Berkom at 2019-01-24T16:55:24Z

  _scheduler/resources.py: Dont error out in unregister_exclusive_interest()
  
  Don't require the interest to be registered, just discard any interest,
  this function just sets a bit in a mask, and is not intended to maintain
  a balance like the reserve() function is.
  

• ce01f87e
  by Tristan Van Berkom at 2019-01-24T16:55:24Z

  _scheduler/scheduler.py: Run cache size exclusively at startup
  
  When running any session that has Queues which require Resource.CACHE,
  check if our loaded estimated size exceeds the quota, and if so;
  lock the Resource.CACHE resource exclusively right away and run
  an exclusive initial cache size job.
  
  This ensures we cleanup first before doing anything which might
  add to the cache at startup time, if deemed needed.
  
  This is a partial fix for issue #737
  

• 2479e8df
  by Tristan Van Berkom at 2019-01-24T16:55:24Z

  _frontend/widget.py: Render core messages more like other messages
  
  In order to test when core activities occur by parsing the stderr
  in tests, we should make the messages conform more.
  
  At the same time, this restores alignment of columns in core
  messages with the element processing related messages.
  
  Also, _scheduler/scheduler.py is updated to make it's activity names
  conform to the (current) 5 character limit for the sake of alignment.
  
  The tests/frontend/logging.py test gets it's regexes updated for
  the log lines it checks for in stderr.
  

• fdb8ff65
  by Tristan Van Berkom at 2019-01-24T16:55:24Z

  tests/artifactcache/expiry.py: Test that expiry happens first
  

• acd0bf22
  by Tristan Van Berkom at 2019-01-24T18:01:41Z

  Merge branch 'tristan/cache-management' into 'master'
  
  Cache management fixes
  
  See merge request BuildStream/buildstream!1091

• 41f03296
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _frontend/app.py: Initialize logging before preflighting the artifact cache
  
  The artifact cache emits messages, and we want to allow that in preflight.
  

• 24ca2f46
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  utils.py: Add _get_volume_size()
  
  We can streamline this call to os.statvfs() in a few places.
  

• 9fd9fbb9
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Refactored to use utils._get_volume_size()
  
  This will benefit from a better UtilError being raised, and
  and turns the artifact cache's local function into a one liner.
  
  The loop which finds the first existing directory in the
  given path has been removed, being meaningless due to the
  call to os.makedirs() in ArtifactCache.__init__().
  
  The local function was renamed to _get_cache_volume_size() and
  no longer takes any arguments, which is more suitable for the
  function as it serves as a testing override surface for
  unittest.mock().
  
  The following test cases which use the function to override
  the ArtifactCache behavior have been updated to use the new
  overridable function name:
  
     tests/artifactcache/cache_size.py
     tests/artifactcache/expiry.py
  

• 7ee0c579
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Added ArtifactCacheUsage()
  
  A simple object which creates a snapshot of current
  usage statistics for easy reporting in the frontend.
  

• 353293b6
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _context.py: Added get_artifact_cache_usage()
  
  A frontend facing API for obtaining usage statistics.
  
  I would have put this on Stream instead, but the Context
  seems to be the de facto place for looking up the artifact cache
  in general so let's put it here.
  

• 51ed36de
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _frontend/widget.py: Added cache usage entry in the startup heading
  

• 5797238b
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _frontend/status.py: Added Cache size usage indicator to status bar
  
  This also adds some comments around the main status bar heading
  rendering function.
  

• 8074ebf4
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Add status messages in cache management operations
  
  Added some useful status messages when:
  
    * Calculating a new artifact cache usage size
  
    * Starting a cleanup
  
    * Finishing a cleanup
  
  Also enhanced messaging about what was cleaned up so far when
  aborting a cleanup.
  

• 49c11bc8
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Added client progress callback to ArtifactCache.clean()
  

• 3616e939
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _scheduler/jobs/job.py: Allow subclasses to message the frontend
  

• bcd19266
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _scheduler/jobs/cleanupjob.py: Update cache size while processing
  
  Updates the known cache size in the main process while the cleanup
  process is ongoing, so that the status indicators update live
  while the cleanup happens.
  

• 9c33107f
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Correcting API documenting comment for remove()
  
  This seems to have been copy/pasted from cascache, and
  documents the function to possibly return None if defer_prune
  was specified, but this function does not expose defer_prune.
  

• a2140d74
  by Tristan Van Berkom at 2019-01-24T19:44:52Z

  Merge branch 'tristan/cache-management-logging' into 'master'
  
  Cache management logging enhancements
  
  See merge request BuildStream/buildstream!1105

• d5847462
  by James Ennis at 2019-01-25T16:35:21Z

  _profile.py: Added a new profiling topic, scheduler
  
  profile_start() and profile_end() calls have been incorporated into
  Scheduler.run()
  

• 56a3954c
  by James Ennis at 2019-01-25T16:35:21Z

  _profile.py: Added a new profiling topic, load-selection
  
  profile_start() and profile_end() calls have been added to
  Stream.load_selection()
  

• 116da6d7
  by James Ennis at 2019-01-25T16:35:21Z

  _profile.py: Update copyright statement
  

• 137d31cd
  by James Ennis at 2019-01-25T17:59:22Z

  Merge branch 'jennis/add_new_profile_topic' into 'master'
  
  Add new 'scheduler' and 'load-selection' profiling topics
  
  See merge request BuildStream/buildstream!1088

• 22b3a0c1
  by Tristan Van Berkom at 2019-01-25T18:35:36Z

  _artifactcache.py: Don't require the quota to be available on disk.
  
  Instead only rely on the headroom to be enough to protect against
  out of space conditions. The headroom can become configurable as
  a separate step is required.
  
  The changes to achieve this are:
  
    * Rename ArtifactCache.has_quota_exceeded() to ArtifactCache.full().
  
    * ArtifactCache.full() now also reports True if the available
      space on the artifact cache volume is smaller than the headroom.
  
      This ensures jobs get triggered to cleanup the cache when
      reaching the end of the disk.
  
    * When loading the artifact quota, it is now only an error if
      the quota exceeds the overall disk space, not if it does not
      fit in the available space.
  
      It is still a warning if the quota does not fit in the
      available space on the artifact cache volume.
  
    * Updated scheduler.py and buildqueue.py for the API rename
  
    * tests: Updated the artifactcache/expiry.py test for its
      expectations in this regard.
  
      Added a new test to test an error when quota was specified to
      exceed total disk space, and adjusted the existing tests to
      expect a warning when the quota does not fit in the available
      space.
  
  This fixes issue #733 and #869.
  

• 27ca6cc7
  by Tristan Van Berkom at 2019-01-25T21:18:15Z

  Merge branch 'tristan/cache-quota-max-only' into 'master'
  
  _artifactcache.py: Don't require the quota to be available on disk.
  
  Closes #869 and #733
  
  See merge request BuildStream/buildstream!1106

• 89ba2abe
  by Chandan Singh at 2019-01-25T23:50:49Z

  tox.ini: Add environment to update man pages
  
  Previously, one had to manually install `click-man` package, and
  remember to run the correct command. Now, we can simply run `tox -e man`
  to update the man pages.
  

• d7438688
  by Chandan Singh at 2019-01-25T23:50:49Z

  man: Refresh man pages
  
  Notable changes:
  
  * New `source` and `artifact` command groups
  * Man pages corresponding to obsolete commands, that were marked as
    hidden in Click, have now been removed.
  
  Fixes #881.
  

• 4edbbd27
  by Chandan Singh at 2019-01-25T23:50:49Z

  setup.py, CONTRIBUTING.rst: Recommend using tox to generate man pages
  
  Simplify our docs, by requesting users to run `tox -e man` to update man
  pages, instead of manually installing `click-man` and running the
  command manually.
  

• a3e7aee8
  by Chandan Singh at 2019-01-25T23:50:49Z

  setup.py: Do not error out when man directory is empty/missing
  
  If the `man` directory is empty, then it won't be copied in the source
  distribution, and `list_man_pages()` will throw an exception when trying
  to list files in a non-existent directory. This prevents us from
  installing the BuildStream package when the man pages are not there.
  
  The most common use-case for this is when we want to re-generate the man
  pages but want to install the package before re-generating them.
  

• 222753ac
  by Tristan Van Berkom at 2019-01-26T05:10:36Z

  Merge branch 'chandan/toxic-man' into 'master'
  
  Generate man pages using tox & update them
  
  Closes #880 and #881
  
  See merge request BuildStream/buildstream!1107

• 5df4105a
  by Angelos Evripiotis at 2019-01-28T10:13:40Z

  news: fix 'osbolete' spelling
  

• 9c981eff
  by Angelos Evripiotis at 2019-01-28T10:17:57Z

  Fixup refs to 'bst track'
  
  Now that 'bst track' is obsolete, change guidance to refer to the
  replacement 'bst source track' instead.
  

• bef80291
  by Angelos Evripiotis at 2019-01-28T10:19:17Z

  Fixup refs to 'bst fetch'
  
  Now that 'bst fetch' is obsolete, change guidance to refer to the
  replacement 'bst source fetch' instead.
  

• 564cb245
  by Angelos Evripiotis at 2019-01-28T11:36:16Z

  Merge branch 'aevri/bst_track_guidance' into 'master'
  
  Fixup refs to 'bst track' and 'bst fetch'
  
  See merge request BuildStream/buildstream!1086

• a3e2cdd2
  by Tom Pollard at 2019-01-28T12:14:40Z

  _stream.py: Add use_artifact_config opt arg for load_selection()
  
  use_artifact_config added as an optional default arg, allowing
  for loading of given elements artifact remote config.
  

• 805baf7d
  by Tom Pollard at 2019-01-28T12:14:40Z

  Download buildtrees on demand for bst shell --use-buildtree
  
  Provide bst shell --use-buildtree the ability to attempt to
  acquire missing buildtrees, given respective option, user
  pull-buildtree context and remote availability.
  
  _frontend/cli.py: Refactor logic for determining --use-buildtree
  option with given opportunity to attempt pulling a non-local
  buildtree. Element loaded with artifact_config to allow remote
  querying.
  
  _stream.py: With given user option and element state, construct
  PullQueue to fetch remote buildtree. Continue or Error without
  buildtree if cannot be attained.
  
  tests/integration/build-tree.py: Update to support new usecases
  

• 38356932
  by Tom Pollard at 2019-01-28T13:44:37Z

  Merge branch 'tpollard/829' into 'master'
  
  Download buildtrees on demand for bst shell --use-buildtree
  
  Closes #829
  
  See merge request BuildStream/buildstream!1050

• 3590ca8c
  by Abderrahim Kitouni at 2019-01-28T14:59:19Z

  requirements/requirements.in: require protobuf >= 3.6
  
  This is needed since 0f2bc3754
  

• 39b952dc
  by Abderrahim Kitouni at 2019-01-28T15:19:04Z

  requirements/requirements.in: require Click >= 7.0
  
  This is needed since 629a6e524, and was lost in the conversion to requirements.in
  

• 80b36d0c
  by Javier Jardón at 2019-01-28T17:54:52Z

  Merge branch 'abderrahim/protobuf-version' into 'master'
  
  requirements/requirements.in: update minimum versions
  
  Closes #884
  
  See merge request BuildStream/buildstream!1114

• a1ab48da
  by Valentin David at 2019-01-28T21:30:26Z

  Fix crash when spawned job completes very fast
  
  Job can complete before we return from `Job.span()` to
  `Scheduler._spawn_job()`, so that `_active_jobs` would not yet contain
  the job.
  
  This would print a stack on the console and try to run a second time
  the job which can have unexpected effects.
  
  In order to reproduce the issue, in
  `buildstream/_scheduler/jobs/job.py`, in `Job.spawn`,
  add a call to `time.sleep()` right before call to
  `asyncio.get_child_watcher()`.
  
  This fixes issue #857.
  

• 2fcb4491
  by Jürg Billeter at 2019-01-28T22:36:22Z

  Merge branch 'valentindavid/crash_in_scheduler_857' into 'master'
  
  Fix crash when spawned job completes very fast
  
  Closes #857
  
  See merge request BuildStream/buildstream!1095

• 1c05a092
  by Valentin David at 2019-01-29T05:58:17Z

  Fix type of error codes in CAS server
  
  Fixes #882.
  

• 785da59c
  by Jürg Billeter at 2019-01-29T06:49:10Z

  Merge branch 'valentindavid/wrong_type_in_status_code' into 'master'
  
  Fix type of error codes in CAS server
  
  Closes #882
  
  See merge request BuildStream/buildstream!1099

• ddef91ea
  by Valentin David at 2019-01-29T07:23:35Z

  Make sure testing cache directory exists
  
  Fixes #873
  

• 6a4c8611
  by Jürg Billeter at 2019-01-29T08:18:45Z

  Merge branch 'valentindavid/make_cache_dir' into 'master'
  
  Make sure testing cache directory exists
  
  Closes #873
  
  See merge request BuildStream/buildstream!1092

• 86c8e414
  by Angelos Evripiotis at 2019-01-29T10:39:29Z

  BREAK:remove unconditional 'are you sure?' prompts
  
  This is a breaking change, as it affects behaviour that people might be
  relying on. An entry has been added to NEWS.
  
  As proposed on the mailing list, this change removes the unconditional
  prompts on:
  
      o: bst workspace reset
      o: bst workspace close --remove-dir
  
  If interactive, these commands would always interrupt you with a prompt
  like this:
  
      This will remove all your changes, are you sure?
  
  This seems like it may just save someone's work some time. It may also
  condition folks to hit 'y' quickly without thinking.
  
  This change also makes the non-interactive behaviour consistent with the
  interactive behaviour in the default case. There is also the case of the
  prompt configured by 'really-workspace-close-project-inaccessible',
  which may be tackled in later work.
  
  This change also removes the new config options to suppress those
  prompts, and their associated news entry.
  
  The relevant bit of the mailing list conversation is here:
  https://mail.gnome.org/archives/buildstream-list/2018-December/msg00106.html
  
  The issue to make interactive and non-interactive behaviour consistent
  is here:
  https://gitlab.com/BuildStream/buildstream/issues/744
  

• aae35e13
  by Angelos Evripiotis at 2019-01-29T12:08:07Z

  Merge branch 'aevri/are_you_sure' into 'master'
  
  BREAK:remove unconditional 'are you sure?' prompts
  
  See merge request BuildStream/buildstream!1061

• 03111d39
  by Dor Askayo at 2019-01-30T10:35:06Z

  filter.py: don't recurse when staging dependencies
  
  Also bump the element's version so cached artifacts would be
  invalidated.
  
  Fixes #883
  

• 7256bb0c
  by James Ennis at 2019-01-30T11:34:04Z

  Merge branch 'doraskayo/filter-indirect-deps-fix' into 'master'
  
  filter.py: don't recurse when staging dependencies
  
  Closes #883
  
  See merge request BuildStream/buildstream!1110

• 36746730
  by Chandan Singh at 2019-01-31T10:50:05Z

  tox.ini: Specify minimum version of click-man
  
  `click-man` versions < 0.3.0 do not properly support multiple
  entrypoints. Since this was added to `tox` after `0.3.0` was released,
  `tox` should never be pulling older versions. But, let's add it here for
  documentation purposes.
  
  See
  https://gitlab.com/BuildStream/buildstream/merge_requests/1107#note_135187046
  for some background on this.
  

• fa4a21ce
  by Chandan Singh at 2019-01-31T12:15:43Z

  Merge branch 'chandan/min-version-click-man' into 'master'
  
  tox.ini: Specify minimum version of click-man
  
  See merge request BuildStream/buildstream!1120

• dd791373
  by Chandan Singh at 2019-01-31T14:32:44Z

  testutils/site.py: Support parsing more exotic git versions
  
  We use output of `git --version` to determine if we can run some tests
  that rely on features from newer git versions. Usually, we expect the
  output to be like:
  
      git version 2.17.2
  
  On some platforms, like MacOS, there could be a suffix after the version
  string, so that it looks something like:
  
      git version 2.17.2 (Apple Git-113)
  
  This causes things to fail like so:
  
      ValueError: invalid literal for int() with base 10: '2 (Apple Git-113)\n'
  
  Fix logic around `HAVE_OLD_GIT` such that we split the output of
  `git --version` without limit on how many times we split. Previously we
  used to split only twice so the suffixes like `(Apple Git-113)` are not
  part of the parsed version.
  

• 96c0fbd6
  by Chandan Singh at 2019-01-31T15:39:19Z

  Merge branch 'chandan/fix-git-version-mac' into 'master'
  
  testutils/site.py: Support parsing more exotic git versions
  
  See merge request BuildStream/buildstream!1118

• d25e2795
  by Benjamin Schubert at 2019-01-31T17:06:23Z

  Add LoadElement in dependency list for LoadElement idrectly
  
  This removes the need for the 'Dependency' list to then be matched
  with the corresponding LoadElement and will allow iterating the
  graph more easily
  

• 2d0eebbf
  by Benjamin Schubert at 2019-01-31T17:06:23Z

  Pass element directly to collect_element
  
  This simplifies the loading
  

• 583bd97d
  by Benjamin Schubert at 2019-02-01T10:26:37Z

  Merge branch 'bschubert/loader' into 'master'
  
  Cleanup loader by linking LoadElements sooner
  
  See merge request BuildStream/buildstream!1122

• 51cec3da
  by Phil Dawson at 2019-02-01T14:25:44Z

  tests/cachekey: Test cache keys are independent of target elements
  

• 2b38aabe
  by Phil Dawson at 2019-02-01T15:33:00Z

  Merge branch 'phil/cache-key-stability-test' into 'master'
  
  tests/cachekey: Test cache keys are independent of target elements
  
  See merge request BuildStream/buildstream!1123

• dbb3d232
  by James Ennis at 2019-02-01T15:51:32Z

  filter.py/filter.yaml: Documentation improvements
  

• 7e4205cb
  by James Ennis at 2019-02-01T15:51:32Z

  filter.py: Add an example to the documentation
  

• 4109a34a
  by James Ennis at 2019-02-01T17:11:29Z

  Merge branch 'jennis/filter-docs' into 'master'
  
  Improve our filter documentation
  
  Closes #278
  
  See merge request BuildStream/buildstream!1112

• c9345014
  by James Ennis at 2019-02-04T13:53:42Z

  filter.py: Fail if declared domains do not exist in the parent element
  
  This patch also uncovered the fact that our test_filter_deps_ok() test
  has been inaccurate. Thus the element built in this test
  (deps-permitted.bst) has been modified so that it build depends on the
  input.bst element, as it should.
  
  tests/filter.py: Ensure deps_ok test passes
  

• 3ab09651
  by James Ennis at 2019-02-04T14:47:43Z

  Merge branch 'jennis/warn_for_nonexistent_domains' into 'master'
  
  Fail when we explictly try to include/exclude non-existent domains in a filter element
  
  See merge request BuildStream/buildstream!1117

• cd8e5e27
  by Tom Pollard at 2019-02-05T11:22:01Z

  Add --remote, -r option to bst build, inline with pull & push
  
  Providing a remote will limit build's pull/push remote actions to
  the given remote specifically, ignoring those defined via user or
  project configuration.
  

• a46650d0
  by Tom Pollard at 2019-02-05T12:41:30Z

  Merge branch 'tpollard/buildremote' into 'master'
  
  Add --remote, -r option to bst build, inline with pull & push
  
  See merge request BuildStream/buildstream!1119

• 9c94e8e5
  by Daniel Silverstone at 2019-02-05T13:15:24Z

  requirements: Add roaringbitmap to the requirements
  
  In order to support use of roaring bitmaps in the loader, we need to
  depend on it here.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 7682ef49
  by Daniel Silverstone at 2019-02-05T13:15:24Z

  loadelement.py: Use roaring bitmaps in dep cache
  
  The dependency cache in LoadElement causes the peak RAM consumption of
  the loader to be exceedingly large, upwards of 25GB for a test of a
  Debian stack.  By switching from the old dict cache to a roaring bitmap
  based cache, we reduce the cost of the loader cache in RAM terms to
  around 5.5GB for the same stack, which makes it plausible for the short
  term.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 0e471144
  by Benjamin Schubert at 2019-02-05T15:45:37Z

  Merge branch 'danielsilverstone-ct/roaring-bitmaps' into 'master'
  
  Switch to roaring bitmaps for the loader dependency caches
  
  See merge request BuildStream/buildstream!1128

• f17ea6a6
  by Jonathan Maw at 2019-02-06T10:32:05Z

  gitlab-ci: Add commands to be run by WSL runners
  

• d4d2f897
  by Jonathan Maw at 2019-02-06T10:45:34Z

  tests: Skip tests that use sandboxes on WSL
  

• f4c8a0ec
  by Jonathan Maw at 2019-02-06T10:45:34Z

  tests: Fix test failures caused by unix sockets' path length limit
  
  The unix specification for unix sockets only allocates ~100 characters
  for the length of the path. This may be longer than the path for a CI
  runner, or the path to a user's buildstream directory.
  

• 256bcde5
  by Jonathan Maw at 2019-02-06T15:29:18Z

  Merge branch 'jonathan/wsl-tests' into 'master'
  
  Add pre-merge tests that use a WSL runner
  
  Closes #852
  
  See merge request BuildStream/buildstream!1108

• 335ef2b8
  by Daniel Silverstone at 2019-02-06T17:30:13Z

  loadelement.py: Switch from roaringbitmap to pyroaring
  
  The licence for roaringbitmap was not compatible.  pyroaring is
  a similar project which is MIT licensed and is slightly better on
  memory too.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 8f90be91
  by Benjamin Schubert at 2019-02-06T18:36:17Z

  Update testing images to include c++ compiler
  

• 1a75b252
  by Benjamin Schubert at 2019-02-06T19:34:09Z

  Merge branch 'danielsilverstone-ct/other-roaring' into 'master'
  
  loadelement.py: Switch from roaringbitmap to pyroaring
  
  Closes #899
  
  See merge request BuildStream/buildstream!1133

• 7285e87e
  by William Salmon at 2019-02-07T09:42:36Z

  Add more log formatting options
  
  The 'wallclock-us' is a option that we thought we already had.
  

• e61f4713
  by Will Salmon at 2019-02-07T10:58:54Z

  Merge branch 'willsalmon/log_formating' into 'master'
  
  Add more log formatting options
  
  See merge request BuildStream/buildstream!1125

• b4d4c4f5
  by Phil Dawson at 2019-02-08T14:27:54Z

  Expose basic api for testing external plugins.
  
  We want external plugins to be able to make use of the core testing utils.
  This commit exposes the basic utilities which are currently in use in
  bst-external plugins. If necessary, more utilities could be exposed in the
  future.
  
  Moves the following files from tests/testutils/ to
  buildstream/plugintestingutils/:
  o runcli.py
  o integration.py
  
  As part of this, this commit makes the following changes to runcli.py
  and integration.py:
  o runcli.py: Fix linting errors
  o runcli.py: Add user facing documentation
  o Integration.py: Add user facing documentation
  

• 77ce0c16
  by Phil Dawson at 2019-02-08T16:28:05Z

  Merge branch 'phil/plugin-testing-api' into 'master'
  
  Expose basic api for testing external plugins.
  
  Closes #847
  
  See merge request BuildStream/buildstream!1075

• 1ee4a4ba
  by Abderrahim Kitouni at 2019-02-08T16:42:23Z

  _artifactcache.py: don't leak the project specific remote caches
  
  the code for initializing remotes added the project specific remote
  caches to the global list instead of making a copy.
  
  Fixes #618
  

• 24c0de16
  by Abderrahim Kitouni at 2019-02-08T16:42:23Z

  _project.py: use artifact caches from the parent project for junctions
  
  This makes a junction use the artifact cache of the parent project
  before the ones defined for the junction
  
  Fixes #401
  

• a937f99a
  by Javier Jardón at 2019-02-08T20:29:05Z

  Merge branch 'abderrahim/artifact-cache-junction' into 'master'
  
  Use artifact cache specs from the parent project before those defined in junctions
  
  Closes #618 and #401
  
  See merge request BuildStream/buildstream!1113

• da1560e4
  by Javier Jardón at 2019-02-09T09:10:01Z

  .gitlab-ci.yml: Remove testing on Fedora 27
  
  Fedora 27 is EOL since 2018-11-30
  
  See https://fedoraproject.org/wiki/End_of_life
  

• 79fbab61
  by Jürg Billeter at 2019-02-09T10:30:20Z

  Merge branch 'jjardon/fedora_27' into 'master'
  
  .gitlab-ci.yml: Remove testing on Fedora 27
  
  See merge request BuildStream/buildstream!1136

• 354c563d
  by Jürg Billeter at 2019-02-10T20:03:46Z

  import.py: Validate config node
  
  Fixes #662.
  

• 5e1be71f
  by Jürg Billeter at 2019-02-11T05:10:56Z

  Merge branch 'juerg/import' into 'master'
  
  import.py: Validate config node
  
  Closes #662
  
  See merge request BuildStream/buildstream!1141

• 99e1be45
  by Jürg Billeter at 2019-02-11T05:12:25Z

  local.py: Do not follow symlinks in local directories
  
  isdir() follows symlinks on the host, resulting in potential host
  contamination. This change reorders the file checks to avoid this issue.
  

• f95e222e
  by Jürg Billeter at 2019-02-11T05:12:25Z

  sandbox/sandbox.py: Do not follow symlinks in _has_command()
  
  This is required to ensure symlinks are not resolved on the host.
  

• 89973fb3
  by Jürg Billeter at 2019-02-11T05:12:25Z

  utils.py: Remove list_dirs parameter from list_relative_paths()
  
  list_dirs was always True in the BuildStream code base. There was also a
  bug in the list_dirs=False code path as it did not return symlinks in
  `dirnames`.
  
  This is an API break, however, there are no known external callers.
  

• d1da3fb0
  by Jürg Billeter at 2019-02-11T05:12:25Z

  utils.py: Fix sorting of symlinks to directories
  
  os.walk() resolves symlinks to check whether they point to a directory
  even when followlinks is set to False. We already work around that
  broken behavior by extracting symlinks from `dirnames`. However, the
  sort order was still incorrect as we returned symlinks in dirnames
  before files and other symlinks. This change fixes this, sorting all
  files and symlinks in a single list.
  

• 7cf67ed3
  by Jürg Billeter at 2019-02-11T05:12:25Z

  _casbaseddirectory.py: Do not mimic os.walk() in list_relative_paths()
  
  This matches the change in utils.list_relative_paths() that now sorts
  all symlinks as files, instead of following the broken behavior of
  os.walk().
  

• 7ec0bb5e
  by Jürg Billeter at 2019-02-11T05:44:20Z

  tests/sources/local.py: Add directory symlink test
  

• c07cc967
  by Jürg Billeter at 2019-02-11T07:13:28Z

  Merge branch 'juerg/symlinks' into 'master'
  
  Symlink fixes
  
  See merge request BuildStream/buildstream!1138

• 9e3d3e05
  by Angelos Evripiotis at 2019-02-11T07:14:50Z

  contributing: snakeviz replaces pyflame+flamegraph
  
  Replace the instructions for pyflame+flamegraph with simpler ones for
  snakeviz. For our general use-case this seems to be easier and better.
  Usage of this tool was demonstrated at the 2019 BuildStream Gathering in
  January by Daniel Silverstone, when presenting the aggregate results of
  profiling on many target environments.
  
  Here is the relevant mailing list thread:
  
  "Profiling before the gathering"
  https://mail.gnome.org/archives/buildstream-list/2019-January/msg00057.html
  

• bb6a692d
  by Jürg Billeter at 2019-02-11T09:20:34Z

  Merge branch 'snakeviz' into 'master'
  
  contributing: snakeviz replaces pyflame+flamegraph
  
  See merge request BuildStream/buildstream!1129

• 1ed63e54
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: better provenance on recursive include
  
  Use the provenance of the include block, instead of the whole node.
  

• 02e48209
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: better error on missing include
  
  Previously, a missing include would result in an error like this:
  
      Could not find file at not-a-file.include
  
  Note that the file containing the include was not mentioned.
  
  Now we get an error like this instead:
  
      element.bst [line 7 column 5]: Include block references a file that
      could not be found: 'not-a-file.include'.
  

• 4336e3bf
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: better error on including directory
  
  Previously, include a directory result in an error like this:
  
      mydir is a directory. bst command expects a .bst file.
  
  Note that the file containing the include was not mentioned.
  
  Now we get an error like this instead:
  
      element.bst [line 12 column 0]: Include block references a
      directory instead of a file: 'mydir'.
  

• 3f6c5000
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: re-use file_path variable
  
  Avoid an unnecessary call to os.path.join().
  

• adde0c94
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  tests/format/include: remove unused tmpdir's
  
  Don't create and remove temp dirs unnecessarily when they are not used,
  looks like these were just copy-pastes without intended side-effects.
  

• a66f8379
  by Jürg Billeter at 2019-02-11T13:52:54Z

  Merge branch 'aevri/include-error' into 'master'
  
  More user-friendly reporting on include errors
  
  See merge request BuildStream/buildstream!891

• a482b008
  by Javier Jardón at 2019-02-11T19:23:02Z

  .gitlab-ci.yml: Do not automatically run "test-wsl" until runner problems are sorted out
  

• 0816b8b1
  by Jürg Billeter at 2019-02-11T21:29:29Z

  Merge branch 'jjardon/allow_fail_WSL' into 'master'
  
  .gitlab-ci.yml: allow "test-wsl" to fail until runner problems are sorted out
  
  See merge request BuildStream/buildstream!1145

• a7aed65a
  by Jürg Billeter at 2019-02-12T05:52:07Z

  utils.py: Increase buffer size in sha256sum()
  
  Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
  factor 4, according to a very simple benchmark.
  

• 8b9e1d24
  by Jürg Billeter at 2019-02-12T05:52:07Z

  _cas/cascache.py: Increase buffer size in add_object()
  
  Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
  factor 4, according to a very simple benchmark.
  

• 86a9048a
  by Jürg Billeter at 2019-02-12T07:08:18Z

  Merge branch 'juerg/buffer-size' into 'master'
  
  Increase read buffer size to improve performance
  
  See merge request BuildStream/buildstream!1142

• 9b6c18e4
  by Chandan Singh at 2019-02-12T10:20:52Z

  _frontend: Allow printing dependencies using `bst show`
  
  At present, there isn't an easy way to print anything from `bst show`
  that would give the users an idea of what the dependency graph looks
  like. One could use things like `--deps build`, but that will just print
  a list, without any information about the dependency edges.
  
  Add `%{deps}`, `%{build-deps}` and `%{runtime-deps}` format strings to
  `bst show` that would simply print the list of all dependencies, build
  dependencies and runtime dependencies respectively.
  
  Summary of changes:
  
    * buildstream/_frontend/cli.py: Add help for new format symbols.
    * buildstream/_frontend/widget.py: Add support for new format symbols
                                       for dependencies.
    * tests/frontend/show.py: Add tests for new format symbols.
  

• 126bc006
  by Chandan Singh at 2019-02-12T10:23:44Z

  Re-generate man pages
  

• dbce3434
  by Chandan Singh at 2019-02-12T12:19:57Z

  Merge branch 'chandan/deps' into 'master'
  
  _frontend: Allow printing dependencies using `bst show`
  
  Closes #890
  
  See merge request BuildStream/buildstream!1121

• 24d29a17
  by Javier Jardón at 2019-02-12T12:27:42Z

  .gitlab-ci.yml: Add job to run test in current Fedora:29
  

• 80dcf65d
  by Javier Jardón at 2019-02-12T12:27:42Z

  .gitlab-ci.yml: run all generic fedora test in current stable version:29
  

• 58b84905
  by Javier Jardón at 2019-02-12T12:27:42Z

  .gitlab-ci.yml: Remove "tests-python-3.7-stretch" job
  
  Fedora 29 includes python 3.7 so this is not needed anymore
  

• 95d9b9ae
  by Javier Jardón at 2019-02-12T13:31:56Z

  Merge branch 'jjardon/fedora_29' into 'master'
  
  .gitlab-ci.yml: Test with current fedora release: 29
  
  See merge request BuildStream/buildstream!1137

• a39c4767
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Only retrieve provenance in node_get() when needed
  
  We were indiscriminately retrieving the node's provenance data in the
  `node_get()` function which was accounting for approximately a third of
  the total runtime of `node_get()` which dominates pre-scheduler time in
  `bst build`.  This change ameliorates that situation by only retrieving
  the provenance data when it's actually needed.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• cda03313
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Reduce use of `isinstance()` in `node_sanitize()`
  
  We know that nodes are typically one of:
  str, list, dict, bool, tuple, NoneType or our ChainMap
  Of these, dict and ChainMap are Mapping, only list is list
  and the rest are returned unchanged.  We can reduce/defer our use
  of isinstance here, dramatically, improving performance.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• b6f08e1b
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Remove use of isinstance() in `{node,list}_{chain_,}copy`
  
  A non-trivial proportion of time pre-scheduler in `bst build` is spent
  copying (or chain-copying) nodes.  Approximately a quarter of the time
  spent in that effort is in `isinstance()`.  This removes that CPU load.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 69ee11c6
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Reduce cost of node_final_assertions
  
  By re-using the isinstance replacements from earlier commits and
  using a tuple of the string constants for checking for composition
  markers, we reduce the cost of node_final_assertions by two thirds
  in basic testing.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 0928e570
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _context.py: Cache result of get_strict()
  
  While get_strict() doesn't look expensive per-se, it is called so many
  times that it is valuable to cache the result once computed.  Since I
  don't think it can change once it is computable, cache it immediately
  that becomes possible and we save 20s in my test case.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 022a59f0
  by Benjamin Schubert at 2019-02-12T16:14:05Z

  Merge branch 'danielsilverstone-ct/further-optimisations' into 'master'
  
  Further optimisations
  
  See merge request BuildStream/buildstream!1131

• 352f4ad9
  by Jonathan Maw at 2019-02-12T16:44:35Z

  loader.py: Stage junctions into .bst instead of a tmpdir
  

• 62396af9
  by Jonathan Maw at 2019-02-12T16:44:35Z

  loader.py: Load junction from workspace if one's open
  

• f1e9cb66
  by Jonathan Maw at 2019-02-12T16:44:35Z

  Remove cleanup functionality from loader
  
  Since junction check-outs are now persistent, there is no need to clean
  up temporary directories once finished.
  Now, junction checkouts are stored within the top-level project dir and
  will be cleaned-up when that directory is removed.
  
  This involves changes in:
  * loader.py: Remove cleanup logic and passing-around of tmpdir.
  * _project.py: Remove passing around tmpdir and calling loader's
                 cleanup.
  

• e51116d5
  by Jonathan Maw at 2019-02-12T17:59:25Z

  Merge branch 'jonathan/junction-no-tmpdir' into 'master'
  
  Stage junctions into .bst instead of a tmpdir
  
  Closes #895
  
  See merge request BuildStream/buildstream!1134

• 478e5c47
  by Valentin David at 2019-02-12T18:06:50Z

  buildstream/_cas/cascache.py: Set 0644 rights to pulled files
  
  This was broken by 5ef19a0b31df84caed1e41719ef7ea5c6bd8a8bc.
  

• 6de65306
  by Valentin David at 2019-02-12T19:13:43Z

  Merge branch 'valentindavid/pull-chmod-bug' into 'master'
  
  buildstream/_cas/cascache.py: Set 0644 rights to pulled files
  
  See merge request BuildStream/buildstream!1144

• 9b140fa0
  by James Ennis at 2019-02-13T09:35:45Z

  element.py: Lift ArtifactCache.get_artifact_fullname() to here
  
  This commit removes the method ArtifactCache.get_artifact_fullname()
  and replaces it with Element.get_artifact_name()
  
  Given a key, we are now able to construct the full name of any of an
  element's artifacts.
  

• 039d7c1d
  by James Ennis at 2019-02-13T09:35:45Z

  element.py: Make calculate_cache_key() API private
  

• 7cb6be74
  by James Ennis at 2019-02-13T09:35:45Z

  element.py: Return early in __init_defaults if no plugin config
  

• ddf642e1
  by James Ennis at 2019-02-13T09:35:45Z

  metaelement.py: Provide constructor with default keyword arguments
  

• 55c15f8f
  by James Ennis at 2019-02-13T09:35:45Z

  _artifactelement.py: New ArtifactElement object (derived from Element)
  
  This object should be used when we want to handle artifact refs directly
  from the command line.
  An ArtifactElementError has also been added to _exceptions.py
  

• 6951cfc8
  by James Ennis at 2019-02-13T09:35:45Z

  _project.py: Add create_artifact_element() method
  

• bf1933b6
  by James Ennis at 2019-02-13T09:35:45Z

  _stream.py: Add the _classify_artifacts() helper
  

• ea6ff6da
  by James Ennis at 2019-02-13T09:35:45Z

  _stream.py: Modify behaviour of _classify_artifacts()
  
  * There is no need for this method to use a cas object.
  * Search for artifact globs in the project's element path
  * An artifact key is always 64 chars long
  

• fcc79917
  by James Ennis at 2019-02-13T09:35:45Z

  _stream.py: Allow loading to handle artifact refs
  

• 77345317
  by James Ennis at 2019-02-13T09:35:45Z

  _artifactcache.py: Add get_artifacts_log() method
  
  A CasBasedDirectory object of an artifacts logs can be obtained
  with ArtifactCache.get_artifacts_log(). This ultimately calls
  CASCache.get_top_level_dir() to obtain a CasBasedDirectory
  of an artifact's subdirectory (or subdirectories).
  

• 775ac472
  by James Ennis at 2019-02-13T09:35:45Z

  cli.py: Move artifact ref handling logic to stream
  
  The loading of elements and the handling of artifacts does not belong
  in this module. Such logic should be invoked using the Stream API
  

• f95d6ee8
  by James Ennis at 2019-02-13T09:35:45Z

  cli.py: Remove _classify_artifacts() method
  
  This method has been moved to Stream, where is it used there exclusively.
  

• 491937d7
  by James Ennis at 2019-02-13T09:35:45Z

  cascache.py: Allow CASCache.list_refs() to handle globs
  
  This commit ensures that CASCache.list_refs(), and
  ArtifactCache.list_artifacts(), can both handle glob expressions.
  

• 6ae41474
  by James Ennis at 2019-02-13T09:35:45Z

  _stream.py: _classify_artifacts() should be able to handle globs
  
  _classify_artifacts() no longer filters glob expressions by first
  obtaining a list of ALL refs locally cached. We now only obtain refs
  specified by the glob _expression_.
  
  Furthermore, the Project.element_path is used to start searching for
  globbed elements as opposed to the Project.directory.
  

• 8c6baf23
  by James Ennis at 2019-02-13T10:48:21Z

  Merge branch 'jennis/refactor_artifact_log' into 'master'
  
  Refactor artifact log command
  
  See merge request BuildStream/buildstream!1101

• 14db3287
  by Jürg Billeter at 2019-02-13T10:48:45Z

  projectconfig.yaml: Consistently include directories in split rules
  
  Most split rules already included the relevant directories themselves in
  addition to the directory contents.  Add the missing bin, sbin, and
  libexec directories.
  
  This is required to fix tests with the following commit that changes
  list_relative_paths() to return all directories.
  

• c7625cdf
  by Jürg Billeter at 2019-02-13T10:48:45Z

  tests/integration/project: Add tests directory to split rule
  
  This is required to fix tests with the following commit that changes
  list_relative_paths() to return all directories.
  

• 925429ea
  by Jürg Billeter at 2019-02-13T10:48:45Z

  utils.py: Return all directories in list_relative_paths()
  
  Returning only empty directories leads to inconsistencies when computing
  a manifest by combining results from multiple list_relative_paths()
  calls as done by the compose plugin.
  
  I.e., the same directory may be empty in one dependency and non-empty in
  another dependency.  The merged file list will still contain that
  directory even though it's no longer empty.
  
  This inconsistency causes problems when calculating differences between
  manifests. Returning all directories fixes these inconsistencies.
  
  This is a change in API behavior.
  

• 353745a4
  by Jürg Billeter at 2019-02-13T10:48:45Z

  _casbaseddirectory.py: Return all directories in list_relative_paths()
  
  This matches the change in utils.list_relative_paths().
  

• ae0ee361
  by Jürg Billeter at 2019-02-13T12:42:42Z

  Merge branch 'juerg/list-all-directories' into 'master'
  
  Return all directories in list_relative_paths()
  
  See merge request BuildStream/buildstream!1139

• 118644b2
  by Tom Pollard at 2019-02-13T13:37:32Z

  Add cli main & user conf option for 'cache-buildtrees' context
  
  _context.py: Add cache_buildtrees global user context, the default
  of which is set to by default to 'always' via the addition of
  cache-buildtrees to userconfig.yaml cache group. 'failure' & 'never'
  can be given as valid options.
  
  app.py & cli.py: Add --cache-buildtrees as a bst main option, which
  when passed with a valid option can override the default or user
  defined context for cache_buildtrees.
  
  tests/completions/completions.py: Update for the added flag.
  

• b546bac1
  by Tom Pollard at 2019-02-13T13:37:32Z

  _frontend/cli.py: Ensure failed buildtree warning is correct
  
  not _cached_sucess() could resolve to true if the element wasn't
  cached at all. switch to _cached_failure() to ensure condition
  reflects expected artifact state
  

• d2029444
  by Tom Pollard at 2019-02-13T13:37:32Z

  Provide configuration for the optional creation of buildtrees
  
  Artifacts can be cached explicitly with an empty `build tree` when
  built via the cli main options or user config for all or only
  successful build artifacts. Default behaviour is to still create
  and cache all expected buildtrees.
  
  element.py: _cache_artifact() Check if context for cache_buildtrees
  has been set to always or failure with a corresponding build
  result, if not skip attempting to export the build-root. Element
  types without a build-root are cached with an empty buildtree
  regardless. Update _stage_sources_at() to warn the user that the
  buildtree import is empty.
  
  tests/integration: Add test to artifact.py for the optional caching
  of buildtree content from bst build. Rename build-tree.py to
  shellbuildtrees.py to reflect included test cases, add test for
  empty buildtree warning and failure option.
  
  NEWS: Add entry for new option.
  

• 54ec032a
  by Jürg Billeter at 2019-02-13T14:50:15Z

  Merge branch 'tpollard/896' into 'master'
  
  Optional creation of buildtrees
  
  Closes #896
  
  See merge request BuildStream/buildstream!1135

• ec4bbf35
  by Benjamin Schubert at 2019-02-13T14:59:47Z

  Cleanup internal Loader cache after loading elements
  

• b41a82d3
  by Benjamin Schubert at 2019-02-13T14:59:47Z

  Don't register exceptions when not running the testsuite
  
  This fix a problem with the garbage collector not being able to
  clean the MetaElements that are loaded.
  
  On small projects this is not a problem, but in bigger projects, this
  can save a few hundred of MBs at runtime
  
  The reason behind this is, whenever we have a "stack" element, which
  has no stack.yaml configuration, since it doesn't need it, we would
  get an exception thrown when initiating the first one, as loading the
  yaml file would fail.
  
  This would capture the frame in which this command was executed, which
  references meta_elements. Therefore, as long as another exception is not
  thrown, the garbage collector would not be able to clean all the
  MetaElements.
  

• 9db7f489
  by Benjamin Schubert at 2019-02-13T16:13:16Z

  Merge branch 'bschubert/cleanup-local-state' into 'master'
  
  Cleanup MetaElement local state
  
  See merge request BuildStream/buildstream!1147

• c2c10d68
  by Daniel Silverstone at 2019-02-14T13:02:07Z

  Cache Keys: Update to use JSON rather than pickle
  
  This affects the cache key version (updated to 7) and introduces
  a dependency on `ujson` which is BSD licenced as of the version
  locked in `requirements.txt`
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 868c057e
  by Tristan Van Berkom at 2019-02-14T13:58:39Z

  Merge branch 'danielsilverstone-ct/json-cache-key' into 'master'
  
  Update cache keys to use JSON
  
  See merge request BuildStream/buildstream!1151

• ace15fa6
  by Benjamin Schubert at 2019-02-14T14:04:45Z

  Don't keep MetaSource around in Source
  
  We can easily rebiuld MetaSource from any source so there is no
  reason to keep them around.
  
  This will slightly improve memory usage.
  

• 174f5394
  by Tristan Van Berkom at 2019-02-14T15:04:27Z

  Merge branch 'bschubert/dont-keep-metasource' into 'master'
  
  Don't keep MetaSource around in Source
  
  See merge request BuildStream/buildstream!1150

• e7e4086e
  by Jürg Billeter at 2019-02-14T15:05:38Z

  utils.py: Change _ensure_real_directory() to not resolve symlinks
  
  Resolving symlinks during staging causes various issues:
  
  * Split rules may not work properly as the resolved paths will differ
    depending on whether another artifact with a directory symlink has
    been staged in the same root directory or not, e.g., as part of
    compose.
  
  * The order of symlinks in file lists is difficult to get right to
    guarantee consistent and predictable behavior as paths in a file list
    might rely on symlinks in the same file list. See #647 and #817.
  
  * Staging order differences can lead to surprising results. See #390.
  
  * Difficult to properly support absolute symlinks. Absolute symlinks are
    currently converted to relative symlinks, however, this doesn't always
    work. See #606 and #830.
  
  This will require changes in projects that rely on the current behavior.
  However, the changes are expected to be small and are often a sign of
  buggy element files. E.g., elements that don't fully obey `bindir` or
  `sbindir` variables.
  

• c0570871
  by Jürg Billeter at 2019-02-14T15:05:38Z

  _casbaseddirectory.py: Do not resolve symlinks
  
  This matches the change in utils._process_list().
  
  This also removes the _Resolver class as it is now unused. We may want
  to support controlled symlink resolution in the future, in which case
  the _Resolver class can be resurrected from this commit.
  

• 55bccf63
  by Jürg Billeter at 2019-02-14T15:05:38Z

  utils.py: Do not mangle absolute symlinks
  
  Copy symlinks as they are, absolute or relative. We no longer resolve
  symlinks when copying files, which makes this safe.
  

• 94c6d59b
  by Jürg Billeter at 2019-02-14T15:05:38Z

  Bump artifact version for changes in symlink handling
  

• 949962f8
  by Jürg Billeter at 2019-02-14T15:05:38Z

  NEWS: Add entry for change in symlink handling
  

• 56c07baa
  by Jürg Billeter at 2019-02-14T16:08:42Z

  Merge branch 'juerg/symlinks2' into 'master'
  
  Do not resolve or mangle symlinks during staging
  
  See merge request BuildStream/buildstream!1140

• 69520011
  by Chandan Singh at 2019-02-15T06:58:27Z

  contrib/bst-graph: Add script to print graph in DOT format
  
  This script leverages the recently added format strings
  (`%{build-deps}`, `%{runtime-deps}`) to `bst show` to print a graph in
  DOT format. This requires users to have the `graphviz` python package
  installed.
  
  Additionally, users can also render the graph using the `--format`
  option if they have the `graphviz` command line tool installed.
  

• 51dae747
  by Chandan Singh at 2019-02-15T08:50:21Z

  Merge branch 'chandan/dot-graph' into 'master'
  
  contrib/bst-graph: Add script to print graph in DOT format
  
  Closes #705
  
  See merge request BuildStream/buildstream!1148

• 0f78a47d
  by Daniel Silverstone at 2019-02-15T10:03:46Z

  Variables: Rework how expansion strings work
  
  Rather than constantly using regular expressions and retrieval from YAML
  nodes, pre-parse expansion strings into a list representation cached
  for reuse, and then expand them as simple string concatenation.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 14176e51
  by Benjamin Schubert at 2019-02-15T11:04:10Z

  Merge branch 'danielsilverstone-ct/variables-rework' into 'master'
  
  Variables: Rework how expansion strings work
  
  See merge request BuildStream/buildstream!1152

• 39febfda
  by Angelos Evripiotis at 2019-02-15T11:10:51Z

  storage.Directory.export_to_tar: mtime=_magic...
  
  Change the default value of mtime when doing export_to_tar() from `0` to
  `_utils._magic_timestamp`. This avoids problems in other software, which
  assume that an mtime of `0` means the file does not exist.
  
  There is more than one example where files with an mtime of zero are
  treated as non-existant by other software. e.g. [ninja][1] and [Template
  Toolkit][2].
  
  The OSTree project also [express an intention][3] to move from an mtime
  of 0 to an mtime of 1:
  
  > For this reason, OSTree acts as though all timestamps are set to
  > time_t 0, so that comparisons will be considered up-to-date. Note that
  > for a few releases, OSTree used 1 to fix warnings such as GNU Tar
  > emitting "implausibly old time stamp" with 0; however, until we have a
  > mechanism to transition cleanly to 1, for compatibilty OSTree is
  > reverted to use zero again.
  
  >From the comments on export_to_tar(), the motivation for having an mtime
  of 0 was to have reproducible results, rather than it specifically being
  the value 0.
  
  Additionally, the reproducible builds project has a [page on
  archives][4]; it mentions the benefits of setting all the files to have
  the same mtime, or clamping the mtime. It makes no mention of a
  motivation for the mtime to be specifically 0.
  
  Fixes #914
  
  [1]: https://github.com/ninja-build/ninja/issues/1120
  [2]: https://github.com/abw/Template2/blob/8d7d37200af436f1ad43628278d3caad257c8e27/lib/Template/Provider.pm#L635
  [3]: https://ostree.readthedocs.io/en/latest/manual/repo/
  [4]: https://reproducible-builds.org/docs/archives/
  

• 8b34e356
  by Angelos Evripiotis at 2019-02-15T12:25:21Z

  Merge branch 'aevri/mtime1' into 'master'
  
  storage.Directory.export_to_tar: default mtime=utils._magic_timestamp
  
  Closes #914
  
  See merge request BuildStream/buildstream!1149

• 3bbb90e7
  by Benjamin Schubert at 2019-02-15T12:34:36Z

  Use sets when checking for existence of an element
  

• 0921ccf4
  by Benjamin Schubert at 2019-02-15T13:39:52Z

  Merge branch 'bschubert/set-as-set' into 'master'
  
  Use sets when checking for existence of an element
  
  See merge request BuildStream/buildstream!1154

• 8efd0a29
  by Adam Jones at 2019-02-15T14:16:22Z

  Add a link to the label descriptions in the issue templates
  

16 changed files:

• .coveragerc
• .gitignore
• .gitlab-ci.yml
• .gitlab/issue_templates/bst_bug.md
• .gitlab/issue_templates/bst_task.md
• CONTRIBUTING.rst
• MANIFEST.in
• NEWS
• README.rst
• buildstream/__init__.py
• buildstream/_artifactcache/artifactcache.py → buildstream/_artifactcache.py
• + buildstream/_artifactelement.py
• buildstream/_cachekey.py
• buildstream/_artifactcache/__init__.py → buildstream/_cas/__init__.py
• buildstream/_artifactcache/cascache.py → buildstream/_cas/cascache.py
• + buildstream/_cas/casremote.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.