[Notes] [Git][BuildStream/buildstream][coldtom/strip-rules] 59 commits: _artifactcache/cascache.py: Add _ensure_blob helper

[Thomas Coldrick <gitlab mg gitlab com>]

Title: GitLab

Thomas Coldrick pushed to branch coldtom/strip-rules at BuildStream / buildstream

Commits:

• b199afe6
  by Jürg Billeter at 2018-09-25T09:01:51Z

  _artifactcache/cascache.py: Add _ensure_blob helper
  
  This adds directory objects to the local repository before downloading
  files in the directory. However, artifact references are still stored
  only after downloading the complete directory and thus, there won't be
  dangling references. This will anyway be required for partial download
  support.
  

• 7d199322
  by Jürg Billeter at 2018-09-25T09:01:51Z

  _artifactcache/cascache.py: Increase payload size limit for uploads
  
  gRPC can handle 1 MiB payloads. Increase size limit from 64 KiB to speed
  up uploads.`
  

• e2e24015
  by Jürg Billeter at 2018-09-25T09:01:51Z

  _artifactcache/casserver.py: Harmonize payload size limit
  
  Use 1 MiB as payload size limit on the server side for both individual
  downloads and batch uploads.
  

• 697d10f2
  by Jürg Billeter at 2018-09-25T09:01:51Z

  _artifactcache/cascache.py: Use BatchReadBlobs
  
  This uses BatchReadBlobs instead of individual blob download to speed up
  artifact pulling, if the server supports it.
  
  Fixes #554.
  

• 81c51dbf
  by Jürg Billeter at 2018-09-25T09:31:55Z

  Merge branch 'juerg/cas-batch' into 'master'
  
  _artifactcache/cascache.py: Use BatchReadBlobs
  
  Closes #554
  
  See merge request BuildStream/buildstream!813

• 2df7d140
  by Jim MacArthur at 2018-09-25T10:36:37Z

  sandbox.py: Stop caching get_virtual_directory if get_directory is used
  
  Also update the documentation for get_virtual_directory to make the
  implications of using both methods clearer.
  

• 62f59382
  by Jim MacArthur at 2018-09-25T10:36:37Z

  element.py/_assemble(): Call get_virtual_directory later.
  

• 8cea7b17
  by Jim MacArthur at 2018-09-25T10:58:40Z

  Merge branch 'jmac/stop-caching-vdirs' into 'master'
  
  Stop caching virtual directories if get_directory is used.
  
  See merge request BuildStream/buildstream!818

• 83ab183e
  by Tiago Gomes at 2018-09-25T16:37:30Z

  CI: test building freedesktop-sdk overnight
  
  Closes #514.
  

• c8594806
  by Tiago Gomes at 2018-09-25T17:03:55Z

  Merge branch 'tiagogomes/issue-514' into 'master'
  
  CI: test building freedesktop-sdk overnight
  
  Closes #514
  
  See merge request BuildStream/buildstream!826

• fdbf9be1
  by Tiago Gomes at 2018-09-26T08:17:46Z

  ci: update freedesktop-sdk ref
  
  The current one doesn't build due server being down.
  

• 9280b0cf
  by Tiago Gomes at 2018-09-26T15:54:53Z

  ci: add ACME challenge
  
  To finish the creation of the Let's Encrypt certificate, which will
  allows us to use https://docs.buildstream.build/ for accessing the
  documentation.
  

• af131503
  by Tiago Gomes at 2018-09-26T16:35:58Z

  Merge branch 'tiagogomes/acme-challenge' into 'master'
  
  ci: add ACME challenge
  
  See merge request BuildStream/buildstream!831

• f7f14f25
  by Tiago Gomes at 2018-09-26T17:42:12Z

  ci: add missing `-p` argument
  

• 0f2bc375
  by Martin Blanchard at 2018-09-27T08:46:27Z

  _protos: Import google.rpc.Code proto
  
  Imports code.proto from https://github.com/googleapis/googleapis and
  corresponding modules generated with ./setup.py build_grpc.
  
  https://gitlab.com/BuildStream/buildstream/issues/668
  

• 16cf9d5f
  by Martin Blanchard at 2018-09-27T08:46:27Z

  _sandboxremote.py: Handle remote build execution exit code
  
  https://gitlab.com/BuildStream/buildstream/issues/668
  

• f5f3cb7c
  by Jim MacArthur at 2018-09-27T09:16:38Z

  Merge branch 'mablanch/668-remote-build-failure' into 'master'
  
  Better handle remote build failures
  
  See merge request BuildStream/buildstream!825

• b99a6740
  by Jürg Billeter at 2018-09-27T09:18:03Z

  element.py: Add __use_remote_execution() helper method
  

• f8bbe008
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _platform: Add check_sandbox_config() method
  

• f9494f1f
  by Jürg Billeter at 2018-09-27T09:18:03Z

  element.py: Validate sandbox config
  

• b4ad84c3
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _platform/linux.py: Remove redundant message
  
  Unsupported sandbox config (UID/GID) is now reported by the element.
  

• 966af3d2
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _platform/linux.py: Do not disable push globally
  
  On systems without user namespace support, elements with unsupported
  sandbox config (UID/GID) are now individually tainted, which disables
  artifact push.
  

• f08e5eae
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _artifactcache/cascache.py: Remove unused enable_push parameter
  

• 10ed9158
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _artifactcache: Add preflight() method
  

• 7aec8b0f
  by Jürg Billeter at 2018-09-27T09:18:03Z

  Move artifact cache creation to Context
  
  The artifact cache is no longer platform-specific.
  

• a5025c33
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _loader: Migrate to Context.artifactcache
  

• ef1cb374
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _scheduler: Migrate to Context.artifactcache
  

• 52991be1
  by Jürg Billeter at 2018-09-27T09:18:03Z

  _stream.py: Migrate to Context.artifactcache
  

• cb6b35dc
  by Jürg Billeter at 2018-09-27T09:18:42Z

  sandbox/_sandboxremote.py: Migrate to Context.artifactcache
  

• 28c62b00
  by Jürg Billeter at 2018-09-27T09:18:43Z

  _platform/platform.py: Remove unused artifactcache property
  

• 99e90918
  by Jürg Billeter at 2018-09-27T09:19:12Z

  _artifactcache/casserver.py: Use Context for artifact cache creation
  

• b8f920e2
  by Jürg Billeter at 2018-09-27T09:19:12Z

  storage/_casbaseddirectory.py: Use Context for artifact cache creation
  

• 3d88c5d4
  by Jürg Billeter at 2018-09-27T09:19:12Z

  tests/testutils: Use Context for artifact cache creation
  

• ed878cb6
  by Jürg Billeter at 2018-09-27T09:19:12Z

  tests/artifactcache/pull.py: Use Context for artifact cache creation
  

• fe82c9c2
  by Jürg Billeter at 2018-09-27T09:19:12Z

  tests/artifactcache/push.py: Use Context for artifact cache creation
  

• 0d0f700d
  by Jürg Billeter at 2018-09-27T09:19:12Z

  _platform: Remove unused context
  

• dd770ec3
  by Jürg Billeter at 2018-09-27T09:19:12Z

  Use lazy platform instantiation
  
  Now that the platform is independent of the context, explicit
  instantiation is no longer required. This avoids issues with platform
  instances used across test cases with mismatching context.
  

• 44da8175
  by Jürg Billeter at 2018-09-27T09:48:20Z

  Merge branch 'juerg/platform' into 'master'
  
  Make platform independent of context
  
  See merge request BuildStream/buildstream!829

• 900e8900
  by Jürg Billeter at 2018-09-27T13:12:34Z

  git.py: Make `ref` human readable
  
  Use `git describe` to prefix `ref` with closest tag, if available.
  

• 107269c1
  by Jürg Billeter at 2018-09-27T13:12:34Z

  testutils/repo/git.py: Add add_tag() method
  

• 8060ad8c
  by Jürg Billeter at 2018-09-27T13:12:34Z

  tests/sources/git.py: Add track and fetch test with and without tag
  

• ab1cb672
  by Jürg Billeter at 2018-09-27T13:46:41Z

  Merge branch 'juerg/git-describe' into 'master'
  
  git.py: Make `ref` human readable
  
  See merge request BuildStream/buildstream!291

• fcf37f9c
  by knownexus at 2018-09-27T14:22:09Z

  Added `set_resource_limits()` to platform
  
  This has been moved from app.py
  As it will have different functionality depending on platform
  Once the Darwin (MacOS) platform is added
  
  Removed `resource.setrlimit()` functionality from app.py
  Added `resource.setrlimit()` functionality to platform.py as function
  

• 80ebdd2c
  by knownexus at 2018-09-27T14:22:09Z

  Added FUSE check, as WSL does not support FUSE yet
  
  _platform/linux.py: Add FUSE check
  And Ensure exception is not raised when performing sandbox availability check
  Without this check, WSL would error upon finding FUSE missing
  This must be bypassed to allow remote execution
  

• 5e81573b
  by knownexus at 2018-09-27T14:22:09Z

  Max-jobs is now defined on a platform basis
  
  This is due to Darwin (MacOS) having a different Max-Jobs limit
  _platform/platform.py: Adding max-jobs call
  
  _project.py: redirected 'max-jobs' to point at the platform
  

• 3c93fe97
  by knownexus at 2018-09-27T14:22:09Z

  Adding Dummy sandbox
  
  This is to allow platforms that do not support sandboxing
  To error in a controlled/known way
  

• 0f3ef369
  by knownexus at 2018-09-27T14:22:24Z

  Adding darwin.py (MacOS) platform
  
  Adding functionality to recognise Darwin as a platform in plaform.py
  

• b78ae767
  by knownexus at 2018-09-27T14:29:29Z

  utils.py: Reworked safe_remove
  
  Non-Linux platforms don't return EISDIR when attempting to unlink a
  directory.
  
  Stopped safe_remove attempting to unlink dirs
  Previously safe_remove would attempt to unlink a path
  Before attempting to remove it if it was a dir
  
  Now it checks for a dir before that step
  

• ab5e78b4
  by James Ennis at 2018-09-27T14:29:32Z

  Import sandbox subclass when instantiating sandbox
  
  This patch was required due to not being able to execute `bst` on
  a Linux machine without FUSE available.
  

• 261f65ca
  by Jürg Billeter at 2018-09-27T14:54:47Z

  Merge branch 'mac_fixes' into 'master'
  
  Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412"
  
  See merge request BuildStream/buildstream!726

• d2ea3069
  by Richard Maw at 2018-09-27T15:16:57Z

  tests: Add regression test for pushing failed builds
  
  This test ensures that failed builds are pushed when set to on-error=quit.
  
  Closes https://gitlab.com/BuildStream/buildstream/issues/534
  

• c128429a
  by richardmaw-codethink at 2018-09-27T15:42:18Z

  Merge branch 'richardmaw/push-failed-build-regression' into 'master'
  
  tests: Add regression test for pushing failed builds
  
  Closes #534
  
  See merge request BuildStream/buildstream!824

• e91eb38b
  by Valentin David at 2018-09-27T16:32:57Z

  Fix outside-of-project check when project path is not canonical.
  
  The issue happens on Silverblue where /home is a symlink to /var/home.
  With this element-path is something like
  /var/home/user/project/elements, when the project path is
  /home/usr/project. Comparing canonical paths solves the issue.
  
  Fixes #673
  

• 7babd55c
  by Tiago Gomes at 2018-09-27T16:57:12Z

  Merge branch 'valentindavid/silverblue-path' into 'master'
  
  Fix outside-of-project check when project path is not canonical.
  
  Closes #673
  
  See merge request BuildStream/buildstream!832

• bcc5209e
  by Thomas Coldrick at 2018-09-28T10:16:00Z

  Upstream freedesktop-sdk strip rules
  
  Adds a modified version of the freedesktop-sdk `strip-binaries` to
  buildstream default `project.conf`. We implemented a few changes to
  the current default, but the main change is to fix a bug due to
  permissions.
  
  The modifications I made to the freedesktop-sdk commands as they are
  at present are as follows:
  * Remove debugedit rules to include source debugging
  * Add a switch to allow failing on error, rather than this being
    the default behaviour.
  These changes mean that we should now have a sensible default for
  a wide variety of users.
  

• 6e205462
  by Thomas Coldrick at 2018-09-28T10:16:00Z

  Try with less fatal commands
  

• effd0ec9
  by Thomas Coldrick at 2018-09-28T10:16:00Z

  Switch to using fatal-strip
  

• 51d7da1a
  by Thomas Coldrick at 2018-09-28T10:16:00Z

  tests/: Update expected cache keys
  

• af8bf827
  by Thomas Coldrick at 2018-09-28T10:16:00Z

  projectconfig.yaml: Make debugdir
  

30 changed files:

• .gitlab-ci.yml
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_artifactcache/casserver.py
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_loader/loader.py
• + buildstream/_platform/darwin.py
• buildstream/_platform/linux.py
• buildstream/_platform/platform.py
• buildstream/_platform/unix.py
• buildstream/_project.py
• + buildstream/_protos/google/rpc/code.proto
• + buildstream/_protos/google/rpc/code_pb2.py
• + buildstream/_protos/google/rpc/code_pb2_grpc.py
• buildstream/_scheduler/jobs/cachesizejob.py
• buildstream/_scheduler/jobs/cleanupjob.py
• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/scheduler.py
• buildstream/_stream.py
• buildstream/_yaml.py
• buildstream/data/projectconfig.yaml
• buildstream/element.py
• buildstream/plugins/sources/git.py
• buildstream/sandbox/__init__.py
• + buildstream/sandbox/_sandboxdummy.py
• buildstream/sandbox/_sandboxremote.py
• buildstream/sandbox/sandbox.py
• buildstream/storage/_casbaseddirectory.py
• buildstream/utils.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.