[Notes] [Git][BuildStream/buildstream][mablanch/630-remote-execution-reconn] 70 commits: element.py: Fix cache check in non-strict mode

[Martin Blanchard <gitlab mg gitlab com>]

Title: GitLab

Martin Blanchard pushed to branch mablanch/630-remote-execution-reconn at BuildStream / buildstream

Commits:

• da7e038b
  by Jürg Billeter at 2018-09-24T13:58:55Z

  element.py: Fix cache check in non-strict mode
  
  The behavior of __assert_cached() was changed when the keystrength
  parameter was introduced. This restores the previous behavior.
  
  Fixes #607.
  
  Fixes: 3d308894 ("element.py: Cache the result of checking whether...")
  

• 86ea1173
  by Jürg Billeter at 2018-09-24T13:59:59Z

  tests/frontend/rebuild.py: Add strict/non-strict rebuild test
  
  Test rebuild that requires staging of weakly cached dependency.
  
  This is a regression test for #607.
  

• a76339de
  by Jürg Billeter at 2018-09-24T16:02:38Z

  Merge branch 'juerg/rebuild' into 'master'
  
  element.py: Fix cache check in non-strict mode
  
  Closes #607
  
  See merge request BuildStream/buildstream!822

• b199afe6
  by Jürg Billeter at 2018-09-25T09:01:51Z

  _artifactcache/cascache.py: Add _ensure_blob helper
  
  This adds directory objects to the local repository before downloading
  files in the directory. However, artifact references are still stored
  only after downloading the complete directory and thus, there won't be
  dangling references. This will anyway be required for partial download
  support.
  

• 7d199322
  by Jürg Billeter at 2018-09-25T09:01:51Z

  _artifactcache/cascache.py: Increase payload size limit for uploads
  
  gRPC can handle 1 MiB payloads. Increase size limit from 64 KiB to speed
  up uploads.`
  

• e2e24015
  by Jürg Billeter at 2018-09-25T09:01:51Z

  _artifactcache/casserver.py: Harmonize payload size limit
  
  Use 1 MiB as payload size limit on the server side for both individual
  downloads and batch uploads.
  

• 697d10f2
  by Jürg Billeter at 2018-09-25T09:01:51Z

  _artifactcache/cascache.py: Use BatchReadBlobs
  
  This uses BatchReadBlobs instead of individual blob download to speed up
  artifact pulling, if the server supports it.
  
  Fixes #554.
  

• 81c51dbf
  by Jürg Billeter at 2018-09-25T09:31:55Z

  Merge branch 'juerg/cas-batch' into 'master'
  
  _artifactcache/cascache.py: Use BatchReadBlobs
  
  Closes #554
  
  See merge request BuildStream/buildstream!813

• 2df7d140
  by Jim MacArthur at 2018-09-25T10:36:37Z

  sandbox.py: Stop caching get_virtual_directory if get_directory is used
  
  Also update the documentation for get_virtual_directory to make the
  implications of using both methods clearer.
  

• 62f59382
  by Jim MacArthur at 2018-09-25T10:36:37Z

  element.py/_assemble(): Call get_virtual_directory later.
  

• 8cea7b17
  by Jim MacArthur at 2018-09-25T10:58:40Z

  Merge branch 'jmac/stop-caching-vdirs' into 'master'
  
  Stop caching virtual directories if get_directory is used.
  
  See merge request BuildStream/buildstream!818

• 83ab183e
  by Tiago Gomes at 2018-09-25T16:37:30Z

  CI: test building freedesktop-sdk overnight
  
  Closes #514.
  

• c8594806
  by Tiago Gomes at 2018-09-25T17:03:55Z

  Merge branch 'tiagogomes/issue-514' into 'master'
  
  CI: test building freedesktop-sdk overnight
  
  Closes #514
  
  See merge request BuildStream/buildstream!826

• fdbf9be1
  by Tiago Gomes at 2018-09-26T08:17:46Z

  ci: update freedesktop-sdk ref
  
  The current one doesn't build due server being down.
  

• 9280b0cf
  by Tiago Gomes at 2018-09-26T15:54:53Z

  ci: add ACME challenge
  
  To finish the creation of the Let's Encrypt certificate, which will
  allows us to use https://docs.buildstream.build/ for accessing the
  documentation.
  

• af131503
  by Tiago Gomes at 2018-09-26T16:35:58Z

  Merge branch 'tiagogomes/acme-challenge' into 'master'
  
  ci: add ACME challenge
  
  See merge request BuildStream/buildstream!831

• f7f14f25
  by Tiago Gomes at 2018-09-26T17:42:12Z

  ci: add missing `-p` argument
  

• 0f2bc375
  by Martin Blanchard at 2018-09-27T08:46:27Z

  _protos: Import google.rpc.Code proto
  
  Imports code.proto from https://github.com/googleapis/googleapis and
  corresponding modules generated with ./setup.py build_grpc.
  
  https://gitlab.com/BuildStream/buildstream/issues/668
  

• 16cf9d5f
  by Martin Blanchard at 2018-09-27T08:46:27Z

  _sandboxremote.py: Handle remote build execution exit code
  
  https://gitlab.com/BuildStream/buildstream/issues/668
  

• f5f3cb7c
  by Jim MacArthur at 2018-09-27T09:16:38Z

  Merge branch 'mablanch/668-remote-build-failure' into 'master'
  
  Better handle remote build failures
  
  See merge request BuildStream/buildstream!825

• b99a6740
  by Jürg Billeter at 2018-09-27T09:18:03Z

  element.py: Add __use_remote_execution() helper method
  

• f8bbe008
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _platform: Add check_sandbox_config() method
  

• f9494f1f
  by Jürg Billeter at 2018-09-27T09:18:03Z

  element.py: Validate sandbox config
  

• b4ad84c3
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _platform/linux.py: Remove redundant message
  
  Unsupported sandbox config (UID/GID) is now reported by the element.
  

• 966af3d2
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _platform/linux.py: Do not disable push globally
  
  On systems without user namespace support, elements with unsupported
  sandbox config (UID/GID) are now individually tainted, which disables
  artifact push.
  

• f08e5eae
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _artifactcache/cascache.py: Remove unused enable_push parameter
  

• 10ed9158
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _artifactcache: Add preflight() method
  

• 7aec8b0f
  by Jürg Billeter at 2018-09-27T09:18:03Z

  Move artifact cache creation to Context
  
  The artifact cache is no longer platform-specific.
  

• a5025c33
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _loader: Migrate to Context.artifactcache
  

• ef1cb374
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _scheduler: Migrate to Context.artifactcache
  

• 52991be1
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _stream.py: Migrate to Context.artifactcache
  

• cb6b35dc
  by Jürg Billeter at 2018-09-27T09:18:42Z

  sandbox/_sandboxremote.py: Migrate to Context.artifactcache
  

• 28c62b00
  by Jürg Billeter at 2018-09-27T09:18:43Z

  _platform/platform.py: Remove unused artifactcache property
  

• 99e90918
  by Jürg Billeter at 2018-09-27T09:19:12Z

  _artifactcache/casserver.py: Use Context for artifact cache creation
  

• b8f920e2
  by Jürg Billeter at 2018-09-27T09:19:12Z

  storage/_casbaseddirectory.py: Use Context for artifact cache creation
  

• 3d88c5d4
  by Jürg Billeter at 2018-09-27T09:19:12Z

  tests/testutils: Use Context for artifact cache creation
  

• ed878cb6
  by Jürg Billeter at 2018-09-27T09:19:12Z

  tests/artifactcache/pull.py: Use Context for artifact cache creation
  

• fe82c9c2
  by Jürg Billeter at 2018-09-27T09:19:12Z

  tests/artifactcache/push.py: Use Context for artifact cache creation
  

• 0d0f700d
  by Jürg Billeter at 2018-09-27T09:19:12Z

  _platform: Remove unused context
  

• dd770ec3
  by Jürg Billeter at 2018-09-27T09:19:12Z

  Use lazy platform instantiation
  
  Now that the platform is independent of the context, explicit
  instantiation is no longer required. This avoids issues with platform
  instances used across test cases with mismatching context.
  

• 44da8175
  by Jürg Billeter at 2018-09-27T09:48:20Z

  Merge branch 'juerg/platform' into 'master'
  
  Make platform independent of context
  
  See merge request BuildStream/buildstream!829

• 900e8900
  by Jürg Billeter at 2018-09-27T13:12:34Z

  git.py: Make `ref` human readable
  
  Use `git describe` to prefix `ref` with closest tag, if available.
  

• 107269c1
  by Jürg Billeter at 2018-09-27T13:12:34Z

  testutils/repo/git.py: Add add_tag() method
  

• 8060ad8c
  by Jürg Billeter at 2018-09-27T13:12:34Z

  tests/sources/git.py: Add track and fetch test with and without tag
  

• ab1cb672
  by Jürg Billeter at 2018-09-27T13:46:41Z

  Merge branch 'juerg/git-describe' into 'master'
  
  git.py: Make `ref` human readable
  
  See merge request BuildStream/buildstream!291

• fcf37f9c
  by knownexus at 2018-09-27T14:22:09Z

  Added `set_resource_limits()` to platform
  
  This has been moved from app.py
  As it will have different functionality depending on platform
  Once the Darwin (MacOS) platform is added
  
  Removed `resource.setrlimit()` functionality from app.py
  Added `resource.setrlimit()` functionality to platform.py as function
  

• 80ebdd2c
  by knownexus at 2018-09-27T14:22:09Z

  Added FUSE check, as WSL does not support FUSE yet
  
  _platform/linux.py: Add FUSE check
  And Ensure exception is not raised when performing sandbox availability check
  Without this check, WSL would error upon finding FUSE missing
  This must be bypassed to allow remote execution
  

• 5e81573b
  by knownexus at 2018-09-27T14:22:09Z

  Max-jobs is now defined on a platform basis
  
  This is due to Darwin (MacOS) having a different Max-Jobs limit
  _platform/platform.py: Adding max-jobs call
  
  _project.py: redirected 'max-jobs' to point at the platform
  

• 3c93fe97
  by knownexus at 2018-09-27T14:22:09Z

  Adding Dummy sandbox
  
  This is to allow platforms that do not support sandboxing
  To error in a controlled/known way
  

• 0f3ef369
  by knownexus at 2018-09-27T14:22:24Z

  Adding darwin.py (MacOS) platform
  
  Adding functionality to recognise Darwin as a platform in plaform.py
  

• b78ae767
  by knownexus at 2018-09-27T14:29:29Z

  utils.py: Reworked safe_remove
  
  Non-Linux platforms don't return EISDIR when attempting to unlink a
  directory.
  
  Stopped safe_remove attempting to unlink dirs
  Previously safe_remove would attempt to unlink a path
  Before attempting to remove it if it was a dir
  
  Now it checks for a dir before that step
  

• ab5e78b4
  by James Ennis at 2018-09-27T14:29:32Z

  Import sandbox subclass when instantiating sandbox
  
  This patch was required due to not being able to execute `bst` on
  a Linux machine without FUSE available.
  

• 261f65ca
  by Jürg Billeter at 2018-09-27T14:54:47Z

  Merge branch 'mac_fixes' into 'master'
  
  Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412"
  
  See merge request BuildStream/buildstream!726

• d2ea3069
  by Richard Maw at 2018-09-27T15:16:57Z

  tests: Add regression test for pushing failed builds
  
  This test ensures that failed builds are pushed when set to on-error=quit.
  
  Closes https://gitlab.com/BuildStream/buildstream/issues/534
  

• c128429a
  by richardmaw-codethink at 2018-09-27T15:42:18Z

  Merge branch 'richardmaw/push-failed-build-regression' into 'master'
  
  tests: Add regression test for pushing failed builds
  
  Closes #534
  
  See merge request BuildStream/buildstream!824

• e91eb38b
  by Valentin David at 2018-09-27T16:32:57Z

  Fix outside-of-project check when project path is not canonical.
  
  The issue happens on Silverblue where /home is a symlink to /var/home.
  With this element-path is something like
  /var/home/user/project/elements, when the project path is
  /home/usr/project. Comparing canonical paths solves the issue.
  
  Fixes #673
  

• 7babd55c
  by Tiago Gomes at 2018-09-27T16:57:12Z

  Merge branch 'valentindavid/silverblue-path' into 'master'
  
  Fix outside-of-project check when project path is not canonical.
  
  Closes #673
  
  See merge request BuildStream/buildstream!832

• 5cdfac81
  by Mathieu Bridon at 2018-09-28T09:20:20Z

  data/userconfig.yaml: Document the cache.quota setting
  

• 5d3f039f
  by Tom Pollard at 2018-09-28T10:34:13Z

  Merge branch 'bochecha/cache-size' into 'master'
  
  data/userconfig.yaml: Document the cache.quota setting
  
  See merge request BuildStream/buildstream!833

• 14e1a3b3
  by Jürg Billeter at 2018-10-01T08:18:05Z

  _platform: Fix get_cpu_count() with cap=None
  
  Comparison between None and integer is not supported.
  

• 232662f1
  by Jürg Billeter at 2018-10-01T08:53:06Z

  Merge branch 'juerg/cpucount' into 'master'
  
  _platform: Fix get_cpu_count() with cap=None
  
  See merge request BuildStream/buildstream!840

• f447aedd
  by Tiago Gomes at 2018-10-01T10:33:11Z

  ci: upper case variables
  
  For consistency.
  

• 682dddce
  by Tiago Gomes at 2018-10-01T10:35:12Z

  ci: correct freedestop-sdk version comment
  

• fafa8136
  by Tiago Gomes at 2018-10-01T10:59:54Z

  Merge branch 'tiagogomes/ci-minor-tweaks' into 'master'
  
  Minor CI tweaks
  
  See merge request BuildStream/buildstream!842

• 26e1a3c7
  by Jürg Billeter at 2018-10-01T14:58:06Z

  _artifactcache/casserver.py: Implement BatchUpdateBlobs
  
  Fixes #676.
  

• f47895c0
  by Jürg Billeter at 2018-10-01T14:58:06Z

  _artifactcache/cascache.py: Use BatchUpdateBlobs
  
  This uses BatchUpdateBlobs instead of individual blob upload to speed up
  artifact pushing, if the server supports it.
  
  Fixes #677.
  

• cf00c0a1
  by Jürg Billeter at 2018-10-01T15:32:30Z

  Merge branch 'juerg/cas-batch' into 'master'
  
  CAS: Implement BatchUpdateBlobs support
  
  Closes #677 and #676
  
  See merge request BuildStream/buildstream!839

• 5f4ae90b
  by Jürg Billeter at 2018-10-02T06:34:02Z

  sandbox/_sandboxremote.py: Use helpers to set up cwd and env
  
  This matches the other sandbox implementations.
  

• 0458bc4e
  by Jürg Billeter at 2018-10-02T07:08:35Z

  Merge branch 'juerg/sandbox' into 'master'
  
  sandbox/_sandboxremote.py: Use helpers to set up cwd and env
  
  See merge request BuildStream/buildstream!841

• 4d7f83be
  by Martin Blanchard at 2018-10-02T12:17:17Z

  _sandboxremote.py: Try to reopen operation steam on failure
  
  The REAPI allows a client to reconnect to an ongoing operation stream by
  providing a WaitExecution(). If implemented on server side, BuildStream
  will try to recover from connection errors using it.
  
  https://gitlab.com/BuildStream/buildstream/issues/630
  

30 changed files:

• .gitlab-ci.yml
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_artifactcache/casserver.py
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_loader/loader.py
• + buildstream/_platform/darwin.py
• buildstream/_platform/linux.py
• buildstream/_platform/platform.py
• buildstream/_platform/unix.py
• buildstream/_project.py
• + buildstream/_protos/google/rpc/code.proto
• + buildstream/_protos/google/rpc/code_pb2.py
• + buildstream/_protos/google/rpc/code_pb2_grpc.py
• buildstream/_scheduler/jobs/cachesizejob.py
• buildstream/_scheduler/jobs/cleanupjob.py
• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/scheduler.py
• buildstream/_stream.py
• buildstream/_yaml.py
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/plugins/sources/git.py
• buildstream/sandbox/__init__.py
• + buildstream/sandbox/_sandboxdummy.py
• buildstream/sandbox/_sandboxremote.py
• buildstream/sandbox/sandbox.py
• buildstream/storage/_casbaseddirectory.py
• buildstream/utils.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.