[Notes] [Git][BuildStream/buildstream][bschubert/docker-sandbox] 136 commits: gitlab-ci.yml: Use CI_PROJECT_DIR instead of $(pwd) for cache

[Benjamin Schubert <gitlab mg gitlab com>]

Title: GitLab

Benjamin Schubert pushed to branch bschubert/docker-sandbox at BuildStream / buildstream

Commits:

• 87751350
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  gitlab-ci.yml: Use CI_PROJECT_DIR instead of $(pwd) for cache
  

• 8668df9f
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  Export variables globally for less repetition.
  
  This doesn't change the test behavior and enable for better
  composability of tests
  

• cc6a7f0d
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  gitlab-cy.yml: make linux-tests-template more composable.
  
  This will allow for further de-duplication of work.
  

• eb37dd89
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  gitlab-ci.yml: remove unused artifacts for unix tests
  
  The unix-logs is always empty, there is therefore no reasons of keeping
  it there
  

• e030b48e
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  gitlab-ci.yml: use template for unix tests
  
  This makes all tests use the same template, which makes reasoning
  about them simpler
  

• 57e6358f
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  Ensure tests requiring bubblewrap are correctly marked
  

• e70dd1f8
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  gitlab-ci.yml: Add linux without brwap/ostree to ensure tests behave
  
  This allows us to make sure that tests behave nicely while missing brwap
  and ostree.
  

• f7cb4386
  by Benjamin Schubert at 2018-11-20T18:10:48Z

  gitlab-ci.yml: Extract test command as a variable
  
  This removes the need of having it synchronized in multiple places
  

• 7f316a76
  by Benjamin Schubert at 2018-11-20T18:41:03Z

  Merge branch 'bschubert/add-fedora-nobwrap-tests' into 'master'
  
  Add tests for fedora with no bubblewrap available
  
  See merge request BuildStream/buildstream!949

• a50c598e
  by Benjamin Schubert at 2018-11-21T09:24:15Z

  utils.py: Ensure move_atomic booleans are keyword only arguments
  
  This is to be coherent with the rest of the codebase
  

• b2f604b8
  by Benjamin Schubert at 2018-11-21T09:24:15Z

  utils.py: improve documentation of move_atomic
  

• daca0c01
  by Benjamin Schubert at 2018-11-21T09:52:42Z

  Merge branch 'bschubert/mr938-comments' into 'master'
  
  Followup on MR 938, addressing additional comments
  
  See merge request BuildStream/buildstream!958

• 050cd1bd
  by Tom Pollard at 2018-11-21T11:37:48Z

  tests/plugin/pipeline.py: Avoid using host user conf
  
  create_pipeline was passing no config parameter to context.load,
  leading to _context.py loading in the host's buildstream user conf
  when running tests locally, potentially leading to unwanted errors.
  

• 58353aad
  by Jürg Billeter at 2018-11-21T14:29:49Z

  Merge branch 'tpollard/pipelinehostconfig' into 'master'
  
  tests/plugin/pipeline.py: Avoid using host user conf
  
  See merge request BuildStream/buildstream!961

• 2022c3e8
  by William Salmon at 2018-11-21T15:53:03Z

  tests/frontend/workspace.py: Refactor support function
  
  The Refactor now makes it possible to test for the forth coming multi
  element workspace open commands and to test for failure.
  

• 7bdd44b2
  by William Salmon at 2018-11-21T15:53:03Z

  Workspace CLI update
  
  This is to update the workspace CLI to as agreed on the mailing list
  https://mail.gnome.org/archives/buildstream-list/2018-September/msg00046.html
  
  This patch also introduces the default workspace directory.
  

• 6612e708
  by William Salmon at 2018-11-21T15:53:03Z

  Added a news item for workspace open changes
  

• 4d6e2cd3
  by Will Salmon at 2018-11-21T16:20:26Z

  Merge branch 'willsalmon/defaultWorkspaces' into 'master'
  
  Updated Workspace CLI
  
  See merge request BuildStream/buildstream!897

• dddf0eaa
  by Valentin David at 2018-11-22T11:12:53Z

  .gitlab-ci.yml: Bump version of fd.o SDK and bst_external in overnight tests
  

• abef70fe
  by Javier Jardón at 2018-11-22T11:41:31Z

  Merge branch 'valentindavid/overnight-tests-bump-fdo-sdk-version' into 'master'
  
  Bump version of fd.o SDK and bst_external in overnight tests
  
  See merge request BuildStream/buildstream!962

• 5c208698
  by Jim MacArthur at 2018-11-26T10:23:31Z

  Docs: Add remote execution architecture documentation.
  

• 5af455a4
  by Jim MacArthur at 2018-11-26T10:55:22Z

  Merge branch 'jmac/execution-architecture-docs' into 'master'
  
  Docs: Add remote execution architecture documentation.
  
  See merge request BuildStream/buildstream!923

• 4ca892e3
  by Jim MacArthur at 2018-11-27T09:30:13Z

  tests: Avoid hangs due to exceptions in subprocesses
  

• d09a18b8
  by Jim MacArthur at 2018-11-27T10:31:07Z

  Merge branch 'jmac/fix-test-hangs-2' into 'master'
  
  Avoid hanging artifact cache tests
  
  See merge request BuildStream/buildstream!964

• df60258a
  by Jürg Billeter at 2018-11-27T12:11:24Z

  source.py: Add optional _get_local_path() method
  

• 42809933
  by Jürg Billeter at 2018-11-27T12:11:24Z

  local.py: Implement _get_local_path()
  

• eb5b10f7
  by Jürg Billeter at 2018-11-27T12:11:24Z

  _loader.py: Use Source._get_local_path() for subprojects
  
  This eliminates unneeded copies of the project directory for junctions
  with a single local source.
  

• b2ea208c
  by Jürg Billeter at 2018-11-27T12:50:46Z

  Merge branch 'juerg/local-junctions' into 'master'
  
  Optimization for local junctions
  
  See merge request BuildStream/buildstream!290

• 223c0073
  by Jürg Billeter at 2018-11-27T13:41:09Z

  tests/integration/manual.py: Add test for command logging
  

• 38d16c62
  by Jürg Billeter at 2018-11-27T13:41:09Z

  sandbox: Deduplicate code to process cwd, env, and command arguments
  

• 85f5b574
  by Jürg Billeter at 2018-11-27T13:41:09Z

  sandbox/sandbox.py: Add SandboxFlags.NONE
  
  This improves readability.
  

• bdd6925f
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Use SandboxFlags.NONE instead of 0
  

• 69005c76
  by Jürg Billeter at 2018-11-27T13:41:09Z

  scriptelement.py: Use SandboxFlags.NONE instead of 0
  

• f1767de2
  by Jürg Billeter at 2018-11-27T13:41:09Z

  sandbox/sandbox.py: Add command batching API
  
  This adds the batch() context manager.
  

• 87f7c99a
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Pass element object to sandbox
  
  This allows the sandbox to output messages using the plugin id.
  

• 024d584d
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Cache artifact also on SandboxCommandError
  

• bb7c3213
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Defer workspace.prepared setting in case of pending commands
  

• 04f05b1a
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Support command batching for integration commands
  

• 7827ac50
  by Jürg Billeter at 2018-11-27T13:44:49Z

  element.py: Add batch_prepare_assemble() method
  
  This allows batching of all sandbox commands aross prepare() and
  assemble().
  

• 63db90b2
  by Jürg Billeter at 2018-11-27T13:44:49Z

  buildelement.py: Support batching for integration and build commands
  

• 73dec86e
  by Jürg Billeter at 2018-11-27T13:44:49Z

  scriptelement.py: Support batching for integration and build commands
  

• 79442e0a
  by Jürg Billeter at 2018-11-27T13:44:49Z

  compose.py: Support batching for integration commands
  

• 6018b3bd
  by Jürg Billeter at 2018-11-27T13:44:49Z

  sandbox/_sandboxremote.py: Implement command batching
  
  Execute batched commands in a single shell script.
  

• ec3d4486
  by Jürg Billeter at 2018-11-27T13:44:49Z

  NEWS: Add entry for command batching
  

• 127d332f
  by Jürg Billeter at 2018-11-27T14:16:50Z

  Merge branch 'juerg/command-batching' into 'master'
  
  Command batching
  
  Closes #675
  
  See merge request BuildStream/buildstream!915

• d5b5538d
  by Jim MacArthur at 2018-11-27T15:32:19Z

  artifactcache: Move ArtifactCacheSpec code into CASRemoteSpec.
  
  There is nothing in ArtifactCacheSpec that's actually specific to
  artifacts, so I've made it a CAS class so we can use the same
  spec for remote execution.
  

• f224d4cb
  by Jim MacArthur at 2018-11-27T15:32:19Z

  artifactcache.py, _context.py: Move CASCache object into context
  
  Since the artifact cache and remote execution share the same
  local CAS store, they should share the same CASCache object. Moving
  this into context allows us to do this.
  

• 926c1446
  by Jim MacArthur at 2018-11-27T15:33:34Z

  _sandboxremote: Remove references to artifact cache.
  
  Uses the new config options introduced earlier to specify remote
  exec storage and execution service.
  

• d7cdb03e
  by Jim MacArthur at 2018-11-27T15:33:36Z

  Documentation: Update to show new config options
  

• 95d8244f
  by Jim MacArthur at 2018-11-27T15:33:36Z

  tests/sandboxes/remote-exec-config.py: New test.
  
  Tests that we get a sensible error message when we specify incomplete
  or wrong information in the configuration for remote execution.
  

• 7f2f62eb
  by Jim MacArthur at 2018-11-27T15:33:36Z

  artifactcache.py: Remove unused verify_digest_pushed
  

• c55ca3a5
  by Jim MacArthur at 2018-11-27T15:33:36Z

  Docs: Correct URLs in artifact cache example
  
  The port number must come after the hostname and before the path.
  The path is ignored by CASCache anyway, so including it is misleading.
  

• 2a8a3b19
  by Jim MacArthur at 2018-11-27T18:17:03Z

  Merge branch 'jmac/remote_execution_split' into 'master'
  
  Split remote execution from artifact cache
  
  Closes #750
  
  See merge request BuildStream/buildstream!946

• 499c70fd
  by Valentin David at 2018-11-28T09:11:21Z

  setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52
  

• 3513580c
  by Valentin David at 2018-11-28T14:02:32Z

  Merge branch 'valentindavid/ruamel_0_15_41' into 'master'
  
  setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52
  
  See merge request BuildStream/buildstream!975

• 26cdee08
  by Valentin David at 2018-11-28T14:29:52Z

  Use f_bavail to query available space. Not f_bfree.
  
  f_bfree space might not be usable. In practice we see failures in big
  disks because f_bfree is over 2GB and f_bavail is 0. We get ENOSPC if
  writing on disk then.
  

• 58ca298f
  by Valentin David at 2018-11-28T14:29:52Z

  Make cache clients not fail when a blob is not available.
  
  We plan to make cache incomplete. That is some blobs are missing.  For
  most of cases we will delete references when requested if they are
  incomplete. But there will be corner cases where objects are removed
  after the reference is requested.
  

• 227fa26d
  by Valentin David at 2018-11-28T14:29:52Z

  Fix type error in RPC messages
  

• 5ef19a0b
  by Valentin David at 2018-11-28T14:29:52Z

  Avoid copying temporary file when adding object to CAS in server.
  
  The file is already a temporary file and does not need copy.  ENOSPC
  is thrown during that copy in issue #609.
  
  Fixes #678.
  

• 8d2946ff
  by Valentin David at 2018-11-28T14:29:52Z

  "Fallocate" object disk space to avoid getting NOSPC error when writing
  
  This locks the temporary object file so that cleanup does not need to
  be done for every write.
  

• b587953a
  by Valentin David at 2018-11-28T14:29:52Z

  Update mtimes of objects for requested references.
  
  This also remove references when some objects are missing. This is in
  preparation for the move from reference to object garbage collection.
  

• a64f667d
  by Valentin David at 2018-11-28T14:29:52Z

  Move cas server from ref-based to object-based garbage collection.
  

• 353b90dd
  by Valentin David at 2018-11-28T14:29:52Z

  Cleanup cache in cas server more agressively
  
  When there is less than 2GB left, it cleans up have 10GB available.
  These values are configurable.
  

• ba9afa98
  by Valentin David at 2018-11-28T14:29:52Z

  Lock cache cleanup in cas server
  
  Cleaning up in parallel might slow down the cleaning process
  

• 9a458402
  by Valentin David at 2018-11-29T08:47:40Z

  Merge branch 'valentindavid/cache_server_fill_up' into 'master'
  
  Fix cleanup of cache in server when disk is full
  
  Closes #678
  
  See merge request BuildStream/buildstream!830

• 47b7a9ba
  by Jim MacArthur at 2018-11-29T09:58:48Z

  _sandboxremote.py: Remove unnecessary tests.
  
  push_message and push_directory will both raise assertion failures if
  they fail to send the digest to the server. Checking the digest
  returned by each only tests that the content was hashed locally, which
  cannot reasonably fail.
  

• 4a8d0565
  by Jim MacArthur at 2018-11-29T10:42:31Z

  Merge branch 'jmac/no-verify-digests' into 'master'
  
  _sandboxremote.py: Remove unnecessary tests.
  
  Closes #786
  
  See merge request BuildStream/buildstream!976

• 2e78e0d1
  by Valentin David at 2018-11-29T13:18:17Z

  Add support for .netrc in remote/tar/zip plugins
  
  Fixes #723.
  

• a6144100
  by Valentin David at 2018-11-29T13:18:17Z

  .gitlab-ci.yml: Update image versions where forgotten
  

• f6c184f5
  by Valentin David at 2018-11-29T16:15:56Z

  Merge branch 'valentindavid/netrc' into 'master'
  
  Add support for .netrc in remote/tar/zip plugins
  
  Closes #723
  
  See merge request BuildStream/buildstream!908

• 1f31aefc
  by Angelos Evripiotis at 2018-11-30T09:34:38Z

  man/: update with changes since Apr 2018
  
  It seems that the regenerated man pages haven't been committed for some
  time. Add them all in one lump now.
  
  Had to manually strip the version number from the generated files.
  
  Had to manually fix the NAME field of bst.1 to not be 'bst - None'.
  
  Oddly had to invoke click-man directly in order to get the changes to
  be applied, the usual setup.py route mentioned in contributing.rst
  didn't work.
  
  Eyeballed the result of invoking man on each of the commands, seems
  good.
  

• 033a5ad9
  by Jürg Billeter at 2018-11-30T10:09:11Z

  Merge branch 'aevri/update_man' into 'master'
  
  man/: update with changes since Apr 2018
  
  See merge request BuildStream/buildstream!981

• 1123b9a1
  by Tom Pollard at 2018-11-30T13:30:35Z

  _stream.py: Ability to pull missing buildtrees outside of pull/build
  
  Adds helper function _buildtree_pull_required() to determine if a
  pullqueue should be constructed, for commands outside of bst pull
  and build where it is determined that an element's buildtree
  artifact is to be required given the respective semantics and
  config. Utilised in push() to attempt to mitigate skipping the push
  of partial elements without the user having to have preceded it with
  an explicit pull.
  
  cli.py: Add new behaviour to push command description
  
  element.py: Move _cached_buildtree() to be non local private method,
  use _KeyStrength types to reduce duplication.
  
  tests/integration/pullbuildtrees.py also updated to cover this
  use-case.
  

• ad293ed3
  by Jürg Billeter at 2018-11-30T14:10:06Z

  Merge branch 'tpollard/774' into 'master'
  
  _stream.py: Ability to pull missing buildtrees outside of pull/build
  
  Closes #774
  
  See merge request BuildStream/buildstream!978

• 0d634c21
  by Phillip Smyth at 2018-11-30T14:16:36Z

  plugin.py: Add BAD_ELEMENT_SUFFIX warning
  

• d97ef822
  by Phillip Smyth at 2018-11-30T14:16:36Z

  _project.py: Check for fatal_warnings earlier
  

• 59d442e5
  by Phillip Smyth at 2018-11-30T15:41:57Z

  loader: Added file suffix check and helper function
  
  _loader/loader.py: Added Suffix check and warning for named elements and add helper function
  

• 2d6ec07d
  by Phillip Smyth at 2018-11-30T15:41:57Z

  cli.py: no autocomplete on files without .bst
  

• ac0bc4c0
  by Phillip Smyth at 2018-11-30T16:50:35Z

  Added tests for .bst suffix and completions
  
  buildcheckout.py: Add mandatory .bst suffix tests
  completions.py: Add test for fail on invalid suffix
  Added required files for testing
  integration/source-determinism.py: renamed test elements to end with .bst
  loader/__init__.py: initialised context properly with load and a message handler
  

• d4bc662a
  by Phillip Smyth at 2018-11-30T16:50:50Z

  NEWS: Added update about the Mandatory .bst change
  

• 7cea464a
  by Jürg Billeter at 2018-11-30T17:21:37Z

  Merge branch 'mandatory_suffix' into 'master'
  
  Mandatory .bst suffix
  
  See merge request BuildStream/buildstream!967

• cc5ef68c
  by Tristan Van Berkom at 2018-12-03T08:41:42Z

  MANIFEST.in: Include the sources for rebuilding the documentation session HTML
  
  This was previously missing, causing our CI to forget to rebuild the sessions
  when updating the documentation.
  
  This fixes issue #778
  

• 87ba19d7
  by Tristan Van Berkom at 2018-12-03T09:19:54Z

  Merge branch 'tristan/fix-doc-session-rebuilds' into 'master'
  
  MANIFEST.in: Include the sources for rebuilding the documentation session HTML
  
  Closes #778
  
  See merge request BuildStream/buildstream!984

• 1b24148f
  by Chandan Singh at 2018-12-03T09:46:13Z

  Ensure `--deps=none` option works for `bst checkout`
  
  Currently, `bst checkout --deps none` command always produces empty
  output. Fix this issue and add regression test for the same.
  
  - element_enums.py: Add Scope.NONE.
  
  - element.py: Ensure Scope.NONE works correctly in addition to
    Scope.RUN/Scope.ALL in Element.dependencies() and Element.search().
  
  - tests/frontend/buildcheckout.py: Fix tests for `--deps none`.
  
  Fixes #670.
  

• 8ac1e757
  by Chandan Singh at 2018-12-03T09:46:14Z

  Add `--deps build` option to `bst checkout`
  
  As discussed in
  https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
  add `--deps build` option to `bst checkout`. This will allow users to
  checkout the all build dependencies of a given element using a single
  command.
  
  - _frontend/cli.py: Add `--deps build` option for `bst checkout`.
  
  - element.py: Support `deps='build'` in Element._prepare_sandbox().
  
  - tests/frontend/buildcheckout.py: Ensure `--deps build` works as
    expected.
  

• ac60f47a
  by Benjamin Schubert at 2018-12-03T09:46:14Z

  Refactor and simplify _prepare_sandbox for elements
  
  Before we would have a intricate logics with multiple arguments
  that might get ignored.
  
  This simplifies the design and introduces a bool `shell` instead
  of having two different variables concerned about scope
  

• b1430fb5
  by Tristan Van Berkom at 2018-12-03T11:01:54Z

  Merge branch 'chandan/bst-checkout-build' into 'master'
  
  Add `--deps build` option to `bst checkout`
  
  Closes #670
  
  See merge request BuildStream/buildstream!819

• 9022a778
  by Javier Jardón at 2018-12-03T11:14:55Z

  buildstream/plugins/sources/git.py: warn about checkout by default
  
  See #783
  

• 4462892e
  by Tristan Van Berkom at 2018-12-03T11:45:07Z

  Merge branch 'jjardon/doc_git_submodules' into 'master'
  
  git: warn about checkout submodules  by default
  
  See merge request BuildStream/buildstream!983

• 6fcfa452
  by Jonathan Maw at 2018-12-03T13:21:05Z

  _yamlcache.py: Use a project's junction name if present
  
  This is required because if there are projects with the same name
  loaded, that have elements in the same path, with the same contents,
  they are considered the same, despite actually belonging to different
  Projects.
  
  This fixes buildstream issue #795
  

• d0f9c724
  by James Ennis at 2018-12-03T13:21:05Z

  tests/loader/junctions.py: Test that we can build a junction that is referred differently, twice.
  
  Test required to close #795 - We should be able to build a project which refers
  to the same junction twice, but with two different filenames.
  

• f7581026
  by Jürg Billeter at 2018-12-03T16:27:03Z

  Merge branch 'jonathan/fix-identical-element' into 'master'
  
  _yamlcache.py: Use a project's junction name if present
  
  Closes #795
  
  See merge request BuildStream/buildstream!980

• 98c15463
  by Tristan Van Berkom at 2018-12-05T07:04:56Z

  _scheduler/queues: Don't call update state outside of error handling harness
  
  Commit 3fa79d8da, part of an initiative for caching of the failed builds,
  introduced a call to Element._update_state() after a job completes and before
  entering the error handling harness intended for handling plugin raised errors.
  
  Element._update_state() can result in triggering plugin code to
  run, so this is incorrect, and causes raised errors to crash BuildStream
  if they happen here.
  
  After analyzing the code, it appears that this additional call to
  Element._update_state() is unneeded, and was only added because the
  state needs to be updated for a failure as well as a success.
  
  Instead, we now have the BuildQueue call Element._assemble_done()
  unconditionally, regardless of whether the build was successful
  or not, which has the same effect and also reads better.
  
  In addition, added a FIXME comment that we are still conditionally
  updating the artifact cache size from BuildQueue.done() only
  if the build is successful, which is incorrect because failed builds
  also increase the local artifact cache size - to fix this we need
  to communicate the added artifact size through Element._assemble()
  regardless of whether the build succeeded or failed.
  

• 5d77b871
  by Tristan Van Berkom at 2018-12-05T07:37:54Z

  Merge branch 'tristan/refactor-queues-update-state' into 'master'
  
  _scheduler/queues/queue.py: Don't call update state outside of error handling harness
  
  See merge request BuildStream/buildstream!988

• 82513109
  by Valentin David at 2018-12-05T09:47:52Z

  git source plugin: Track git tags and save them to reproduce a minimum shallow repository
  
  Instead of tag information being fetched which can change with time,
  they are tracked and saved in the projects.refs/.bst. Then we re-tag
  automatically the closest tag so that `git describe` works and is
  reproducible.
  
  This new feature is opt-in with the new `track-tags` configuration,
  and must be used to fix modules which are broken by our new policy
  of omitting the `.git/` repository when staging git sources.
  
  This fixes issue #487
  

• ec7b62c3
  by Valentin David at 2018-12-05T09:50:15Z

  NEWS: Add news about broken 'git describe' by default.
  

• 761e7570
  by Tristan Van Berkom at 2018-12-05T10:23:49Z

  Merge branch 'valentindavid/git_describe_tracking' into 'master'
  
  Track of git tags and save them to reproduce minimum shallow repository
  
  Closes #487
  
  See merge request BuildStream/buildstream!906

• 2f5966bd
  by Raoul Hidalgo Charman at 2018-12-05T10:37:10Z

  platform.py: Add mapping of uname to OS-independent values
  

• 1ad35fcd
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  _config.py: Use os and architecture settings
  
  In element, platform asks for host os and architecture to default to when
  SandboxConfig is initialised.
  This changes element cache keys so those have been updated in the tests.
  

• 9f75c274
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  optionarch.py: update to use same arch names as SandboxConfig
  
  Also update tests to be consistent with this
  

• a825096c
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  _sandboxremote.py: Add os and arch to command uploaded
  
  These are contained within the platform field.
  

• 8e09a81c
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  optionos.py: Add new option for OS configuration
  

• defec112
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  optionos.py: Add tests for OptionOS
  
  Same format as OptionArch tests, but with OS's.
  

• e5d8c7d8
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  _platform: Add checks for os and architecture
  

• b0603fb0
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  Sandbox: use linux32 for x86-32 builds on x86-64 machines
  
    o _platform/linux.py: Add linux32 flag to send to sandbox bwrap when the
      build arch is x86-32 and the machines arch is x86-64 or similarly with
      aarch32 and aarch64.
    o sandbox/_sandboxbwrap.py: Use flag passed to start bwrap command with
      linux32 if set.
  

• f4d76996
  by Raoul Hidalgo Charman at 2018-12-05T14:58:49Z

  format_project.rst: Update documentation for changes to OptionArch
  

• 81163a79
  by Raoul Hidalgo Charman at 2018-12-05T14:58:53Z

  format_declaring.rst: Update documentation for sandbox options
  

• 23c27580
  by Raoul Hidalgo Charman at 2018-12-05T14:58:53Z

  format_project.rst: Add docs for OptionOS
  

• eb1ed410
  by Jürg Billeter at 2018-12-05T15:28:52Z

  Merge branch 'raoul/775-execution-environment-reqs' into 'master'
  
  Execution environment reqs
  
  Closes #775
  
  See merge request BuildStream/buildstream!969

• 9ef1a8a2
  by Raoul Hidalgo Charman at 2018-12-05T15:32:22Z

  utils.py: Add message digest function
  
  Get cascache.py to use this
  

• 4a56bf2a
  by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

  _sandboxremote.py: Add checks to action cache before attempting to push.
  
  Stops unneccesary pushing of builds that have already been built, just checks
  the action cache to begin with.
  
  Fixes #628
  

• 3641b6f6
  by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

  _sandboxremote.py: Add action-cache-service endpoint
  
  The endpoint may not necessarily be the same as execution-service endpoint.
  
  Part of #628
  

• 9d77351f
  by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

  format_project.rst: Add docs for action-cache-service
  
  Part of #628
  

• a53d6d1f
  by Raoul Hidalgo Charman at 2018-12-05T16:36:00Z

  Merge branch 'raoul/628-RE-flow-optimisation' into 'master'
  
  Remote-execution client flow optimisation
  
  Closes #628
  
  See merge request BuildStream/buildstream!982

• 4688b2c9
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  tests/sources/git.py: Refactor ref-not-in-track test to use parameterization
  
  Instead of duplicating the whole test body, run it twice while
  checking for a warning or an error depending on the parameter.
  

• 03028c7f
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  types.py: Moving CoreWarnings to the types.py module
  
  This also ensures it is exposed via the main buildstream __init__.py
  file, fixes the imports of CoreWarnings, and adjusts the documentation
  links to point to the right place.
  

• 3b7f8df8
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  Source: Fix typo in documentation, and make it a link to `bst track`
  
  Now the commands have documentation linkability so we can link to
  them when referring to them anywhere in the docs. This reference
  was previously referring to a bogus non-existent `build-stream track`,
  maybe this dates way back to the days before we named the frontend `bst`.
  

• e5c0f067
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  git source plugin: Fixing documentation linking and typo
  
  Fixes the configurable warning part of this to properly link
  to the project.conf documentation describing what configurable
  warnings actually are.
  
  Also fix the plugin raise the actual configurable warning which
  it advertizes that it raises; which is 'inconsistent-submodule',
  not 'inconsistent-submodules'.
  

• 205c27d8
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  git source plugin: Rename symbol for consistency and readability
  
  Change 'INCONSISTENT_SUBMODULE' for 'WARN_INCONSISTENT_SUBMODULE' to
  make things a bit more readable and comprehensive.
  

• bea4d4f5
  by Tristan Van Berkom at 2018-12-06T09:37:50Z

  Merge branch 'tristan/misc-cleanup' into 'master'
  
  Misc cleanups
  
  See merge request BuildStream/buildstream!993

• b1d2f001
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  source.py: Add new delegate method validate_cache()
  
  This is guaranteed to be called only once for a given session once the
  sources are known to be Consistency.CACHED, if source tracking is enabled
  in the session for this source, then this will only be called if the
  sources become cached after tracking completes.
  

• ee7fc47f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  git source plugin: Implementing submodule warnings
  
   o Unlisted submodule warning
  
     Now the git plugin will issue a configurable warning if a submodule
     exists and is used (checking out the submodule is not disabled),
     but is not specified in the source configuration.
  
   o Invalid submodule warning
  
     Now the git source plugin will issue a warning if the configuration
     specified a submodule which does not exist in the underlying git repository.
  
  As a side effect, this patch also changes the flow control of the git
  plugin such that submodules which are explicitly set to not be
  checked out, are also not fetched but instead ignored completely.
  

• 5fc9a1da
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  git source plugin: Emmit the ref-not-in-track warning from validate_cache()
  
  Now that we have Source.validate_cache(), this is a better place to emmit
  the ref-not-in-track warning, since it will be emmitted at the earliest
  opportunity and not only at Source.stage() or Source.init_workspace().
  
  This also allows us to remove the `self.tracked` local state, and cleanup
  some convoluted calling paths, removing some unnecessary parameters from
  the usual codepaths and making the plugin overall more readable.
  

• 50cb2706
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/testutils/repo/git.py: Adding remove_path() helper
  
  A function for removing paths from a git repo, can also
  be used to remove submodules.
  

• f139ca8f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Testing the git:unlisted-submodule warning
  
   o Test that it is not triggered in show before fetch, because we
     don't know about the unlisted submodules yet
  
   o Test that it is triggered by a fetch command
  
   o Test that it is triggered by `show` after having completed a
     fetch command, since now we have the repository and know about
     the unlisted submodule
  
   o Test all of this under warning or error conditions (parameterized
     for fatal-warnings)
  

• afe1aa9f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Testing the git:invalid-submodule warning
  
     o Test that it is not triggered in show before fetch, because we
       don't know the submodules yet so we cannot know if they are
       valid or not.
  
     o Test that it is triggered by a fetch command
  
     o Test that it is triggered by `show` after having completed a
       fetch command, since now we have the repository and know which
       specified submodules are invalid
  
     o Test all of this under warning or error conditions (parameterized
       for fatal-warnings)
  

• 57300201
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Test unlisted submodules warning appearing after track
  

• a0297625
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Test invalid submodules warning appearing after track
  

• 60e96781
  by Tristan Van Berkom at 2018-12-06T15:14:37Z

  Merge branch 'tristan/submodule-warnings' into 'master'
  
  Implement submodule warnings
  
  See merge request BuildStream/buildstream!996

• 7fe55143
  by Jonathan Maw at 2018-12-06T15:48:12Z

  Fix bst source-checkout not working with open workspaces
  

• 2a6879a5
  by Jonathan Maw at 2018-12-06T17:46:33Z

  Merge branch 'jonathan/source-checkout-workspace' into 'master'
  
  Fix bst source-checkout not working with open workspaces
  
  Closes #807
  
  See merge request BuildStream/buildstream!997

• e61e6e50
  by Tristan Van Berkom at 2018-12-07T07:57:13Z

  _yaml.py: Added `allow_none` parameter to _yaml.node_get()
  
  This allows specifying whether or not the code believes a None
  value is acceptable if explicitly expressed to be None in the YAML.
  
  A previous patch 3ba544b80f9f268be8ffe62fc8589b30212ec4a2 changed
  the behavior to accept None for all values across the board, along
  with allowing explicitly setting the `default_value` to `None` which
  caused the code to be much more readable (this was the main motivation
  of the patch, but it had the side effect of allowing None for everything).
  
  In the majority of cases we load YAML however, it either has a value or
  it is not specified in the YAML, and None is hardly ever acceptable
  to be explicitly specified (it may be in the case you want to override
  something with nothing in the YAML, but it is the edge case).
  
  This had the side effect of causing unexpected crashes where the
  YAML specifies an empty dictionary for instance.
  
  Instead of forcing the caller to handle a possible None value, give
  the choice to the caller if they really want to accept a None value,
  and raise the LoadError otherwise.
  
  This fixes issue #803, which is a crash due to receival of an
  unexpected None value through _yaml.node_get()
  

• 504ecb3e
  by Tristan Van Berkom at 2018-12-07T07:57:56Z

  plugin.py: Added new `allow_none` parameter to Plugin.node_get_member(), defaulting to False.
  
  Allow plugins to conveniently leverage the new `allow_none` parameter,
  and have Plugin.node_get_member() raise a LoadError in the usual
  case where None is not an acceptable value to be explicitly set in
  the user provided YAML.
  

• 642ae4e8
  by Tristan Van Berkom at 2018-12-07T07:57:56Z

  tests/format/project.py: Added new regression test for empty dependency dictionaries
  
  This adds a regression test for issue #803
  

• 2a0676c3
  by Tristan Van Berkom at 2018-12-07T08:33:14Z

  Merge branch 'tristan/yaml-optionally-allow-none' into 'master'
  
  Only optionally allow None values in user provided YAML
  
  Closes #803
  
  See merge request BuildStream/buildstream!999

• 58b38f32
  by Benjamin Schubert at 2018-12-07T14:33:06Z

  First draft for docker sandbox
  

30 changed files:

• .gitlab-ci.yml
• MANIFEST.in
• NEWS
• buildstream/__init__.py
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_artifactcache/casserver.py
• buildstream/_context.py
• buildstream/_frontend/cli.py
• buildstream/_loader/loader.py
• buildstream/_options/optionarch.py
• + buildstream/_options/optionos.py
• buildstream/_options/optionpool.py
• buildstream/_platform/linux.py
• buildstream/_platform/platform.py
• buildstream/_platform/unix.py
• buildstream/_project.py
• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/queue.py
• buildstream/_stream.py
• buildstream/_versions.py
• buildstream/_yaml.py
• buildstream/_yamlcache.py
• buildstream/buildelement.py
• buildstream/data/projectconfig.yaml
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/plugin.py
• buildstream/plugins/elements/compose.py
• buildstream/plugins/sources/_downloadablefilesource.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.