[Notes] [Git][BuildStream/buildstream][tpollard/494] 50 commits: _project.py: Fix option resolution in element & source overrides

[Tom Pollard <gitlab mg gitlab com>]

Title: GitLab

Tom Pollard pushed to branch tpollard/494 at BuildStream / buildstream

Commits:

• 727f2faa
  by Tristan Van Berkom at 2018-09-18T07:43:07Z

  _project.py: Fix option resolution in element & source overrides
  
  This ensures that option expressions are resolved in the project
  level overrides before attempting to composite them on the instantiated
  elements. Seems this is a regression from introducing the include
  directive.
  
  This fixes issue #658
  

• ffa0bb36
  by Tristan Van Berkom at 2018-09-18T07:47:44Z

  tests/format/optionoverrides.py: Added test for options in element overrides
  
  This is a regression test for issue #658
  

• f2ae46f8
  by Tristan Van Berkom at 2018-09-18T08:14:23Z

  Merge branch 'tristan/fix-override-options' into 'master'
  
  Fix override options
  
  Closes #658
  
  See merge request BuildStream/buildstream!802

• 7b117e40
  by Daniel Silverstone at 2018-09-18T08:41:32Z

  _artifactcache/artifactcache.py: Ensure no double-setup of remotes
  
  Since ArtifactCache.setup_remotes() can be expensive and should only
  happen once, this commit will assert() if it is called a second time
  on an artifact cache instance.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 345f5f49
  by Daniel Silverstone at 2018-09-18T08:41:32Z

  tests/artifactcache/pull.py: Do not double-initialize remotes
  
  The initialization of remotes is done by ArtifactCache.setup_remotes()
  and as such it was wrong for these tests to be calling
  CASCache.initialize_remotes() a second time.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• e32221b6
  by Daniel Silverstone at 2018-09-18T08:45:50Z

  sandbox/_sandboxremote.py: Acquire artifact cache via Platform
  
  The SandboxRemote used to construct its own CASCache which was
  considered dangerous.  This patch replaces that with acquisition of
  the cache via the Platform singleton, hopefully eliminating issues
  from having more than one artifact cache object in a single process.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• b587579f
  by Tristan Van Berkom at 2018-09-18T09:53:26Z

  Merge branch 'danielsilverstone-ct/fix-654' into 'master'
  
  sandbox/_sandboxremote.py: Acquire cache via Platform
  
  See merge request BuildStream/buildstream!797

• 30b41959
  by Tristan Van Berkom at 2018-09-18T09:56:45Z

  _artifactcache/artifactcache.py: Error out gracefully when push remote is mal-specified
  
  When configuring a push remote and specifying either the client-cert
  or the client-key, then both must be specified. This ensures we
  get an informative error instead of a stack trace and BUG.
  
  Fixes issue #625
  

• 41e8dc81
  by Tristan Van Berkom at 2018-09-18T09:56:45Z

  tests/artifactcache/config.py: Added test for invalid push remote configuration
  
  Test that we get the expected error when configuring a client-cert
  without client-key, or the inverse.
  

• 97071b6e
  by Tristan Van Berkom at 2018-09-18T10:16:43Z

  Merge branch 'tristan/fix-artifact-config-crash' into 'master'
  
  Fix artifact config crash
  
  Closes #625
  
  See merge request BuildStream/buildstream!804

• 081dcafa
  by Richard Maw at 2018-09-18T13:22:38Z

  fuse: Report the correct device number for devices
  
  This fixes all devices being mapped to the non-existant device 0,
  which prevents being able to use even safe devices like /dev/null
  through the hardlinks FUSE layer.
  

• d0425608
  by Richard Maw at 2018-09-18T13:22:38Z

  FUSE: Mount with -odev in chroot sandbox
  
  This is needed to permit access to the device nodes added to /dev
  on Linux when FUSE is used as root.
  
  The chroot sandbox only works with all privileges,
  so there's no explicit check for being root
  or having the appropriate capabilities.
  
  A check for whether it's running as root isn't needed on Linux with bubblewrap
  because /dev or its devices are mounted on top of the FUSE layer,
  so device nodes are accessed directly rather than through the FUSE layer.
  

• 8430fdc7
  by Richard Maw at 2018-09-18T13:22:38Z

  tests: test that integration commands can use /dev
  

• d8450166
  by Tristan Van Berkom at 2018-09-18T13:50:15Z

  Merge branch 'richardmaw/fix-chroot-sandbox-devices' into 'master'
  
  fix chroot sandbox devices
  
  See merge request BuildStream/buildstream!781

• ba4da3a4
  by knownexus at 2018-09-18T16:45:09Z

  Incorrect error when malformed project.conf
  
  bst build returns "missing project.conf" when a project.conf is invalid
  This results in an existing project with malformed yaml being dismissed
  and attempts to make a new project
  
  Added new exception
  

• 89a16e17
  by knownexus at 2018-09-18T16:45:09Z

  Adding test for Invalid Yaml
  

• 72b59021
  by Phillip Smyth at 2018-09-19T08:48:51Z

  Merge branch 'issue-642-Invalid_project.conf_seen_as_missing' into 'master'
  
  Incorrect error when malformed project.conf
  
  Closes #642
  
  See merge request BuildStream/buildstream!792

• ca811a4d
  by Josh Smith at 2018-09-19T09:25:21Z

  Rework Skipped usage
  
  The SKIPPED message type is now used to indicate the end of a task which
  was successful without having to perform the given task.
  
  This overhauls the use of `Queue.done()` and therefore queues do not
  need to provide a processed/skipped return value from `done()`. Instead
  this is replaced with the action of raising a `SkipJob` exception from
  within `Queue.process()`.
  

• 16fbc3e3
  by Josh Smith at 2018-09-19T09:25:21Z

  tests.py: Test skip on push
  
  Adds a test to ensure that BuildStream alerts the user of a skipped push
  when the remote already has the artifact cached.
  

• ab56f693
  by Josh Smith at 2018-09-19T09:25:21Z

  element.py: Remove redundant timed_activity
  
  This removes the timed_activity for an element _push action. This is
  unnecessary as the job is already being timed elsewhere.
  

• 198fbfe8
  by Josh Smith at 2018-09-19T09:25:21Z

  cascache.py: Move push/pull messaging to cascache
  
  Pulled/Pushed messages will no longer be produced from within
  element.py, instead they will be produced during CasCache push() and
  pull() appropriately.
  
  Message consistency has also been improved.
  

• 213f7794
  by Josh Smith at 2018-09-19T09:25:21Z

  cascache.py: Modify messaging API calls
  
  Both pulling and pushing INFO messages are now status messages.
  
  Calls to the messaging API through `self.context.message()` have now
  been switched to `element.info`.
  

• dddd6025
  by Qinusty at 2018-09-19T10:59:48Z

  Merge branch 'Qinusty/skipped-rework' into 'master'
  
  Add SkipJob for indicating a skipped activity
  
  See merge request BuildStream/buildstream!765

• 16d9c6e5
  by Tristan Van Berkom at 2018-09-19T11:08:33Z

  _frontend/status.py: Completely remove the blessings dependency from BuildStream
  
  This actually improves reliability of the status bar because we
  now disable it completely in the case that not all of the terminal
  escape sequences are supported on the given terminal.
  
  This replaces the few functions we were using, to move the cursor
  up one line, move it to the beginning of the line, and to clear a
  line, with low level functions provided by the curses module in
  the standard library.
  
  This change makes it easier for downstream distro package maintainers
  to package BuildStream, particularly on Fedora.
  
  Asides from changing _frontend/status.py, this commit includes the
  following changes:
  
    * _frontend/app.py: Use python isatty() function to determine if
                        we are connected to a tty, instead of relying
  		      on blessings.
  
    * setup.py: Remove the dependency on blessings.
  

• 7dd74b01
  by Tristan Van Berkom at 2018-09-19T11:27:55Z

  Merge branch 'tristan/remove-blessings' into 'master'
  
  _frontend/status.py: Completely remove the blessings dependency from BuildStream
  
  See merge request BuildStream/buildstream!808

• 335d5a45
  by Adam Jones at 2018-09-19T13:00:45Z

  Fix rst links for guideline links
  

• 667dd4ca
  by Javier Jardón at 2018-09-19T13:25:36Z

  Merge branch 'adamjones/contributing-links' into 'master'
  
  Fix rst link formatting for guideline links
  
  See merge request BuildStream/buildstream!811

• 68ef69e4
  by Tristan Van Berkom at 2018-09-21T05:20:46Z

  source.py: Fix re-instantiation
  
  This fixes an issue where the re-instantiated Source used with
  Source mirroring enabled is not completely initialized.
  
  Failing to load the ref from the project.refs file for instance,
  will result in a crash at `fetch` time.
  
  This fixes issue #666
  

• 662c729f
  by Tristan Van Berkom at 2018-09-21T05:59:30Z

  Merge branch 'tristan/fix-source-reinstantiation' into 'master'
  
  source.py: Fix re-instantiation
  
  Closes #666
  
  See merge request BuildStream/buildstream!817

• 461a0588
  by Jim MacArthur at 2018-09-21T10:53:11Z

  element.py: Remove early warning and check directory is not None
  
  The check for directory should have always been here. The early warning
  test doesn't work for `bst checkout` because caches haven't been configured
  at the point of the test, so I've had to remove it.
  

• aa9caaac
  by Jim MacArthur at 2018-09-21T10:53:11Z

  _sandboxremote.py: Ignore return value of push_directory
  
  The return value is always upload_vdir.ref.
  

• 2aae68c7
  by Jim MacArthur at 2018-09-21T10:53:11Z

  cascache.py: make push_remote raise exception if no push remotes exist
  
  Also add docstring to method and remove return value since it was useless
  

• ca1bb72c
  by Jim MacArthur at 2018-09-21T10:53:11Z

  tests/artifactcache/push.py: Ignore return value of push_directory
  

• 55c93a82
  by Jim MacArthur at 2018-09-21T11:26:55Z

  Merge branch 'jmac/remote_exec_checkout_fix' into 'master'
  
  Remote exec: Remove early warning and check directory is not None
  
  See merge request BuildStream/buildstream!800

• e209beb0
  by Chandan Singh at 2018-09-21T13:10:08Z

  _stream.py: Ensure source-bundle's source directory exists
  
  Currently, `source-bundle` command is entirely broken as it tries to stage the
  sources in a directory that doesn't exist. Fix it by ensuring that we create
  the necessary directories before calling any methods that try to use those
  directories.
  
  This fix comes with a regression test to ensure that the basic use-case
  of `source-bundle` continues to work in future.
  
  Fixes https://gitlab.com/BuildStream/buildstream/issues/651.
  

• 0b000518
  by Chandan Singh at 2018-09-21T13:56:55Z

  Merge branch 'chandan/fix-source-bundle' into 'master'
  
  Fix source-bundle command
  
  Closes #651
  
  See merge request BuildStream/buildstream!807

• ef26043a
  by Chandan Singh at 2018-09-21T17:14:16Z

  setup.py: Make website the primary homepage
  

• 1b2aed40
  by Chandan Singh at 2018-09-21T17:40:11Z

  Merge branch 'chandan/update-project-homepage' into 'master'
  
  setup.py: Make website the primary homepage
  
  See merge request BuildStream/buildstream!795

• da7e038b
  by Jürg Billeter at 2018-09-24T13:58:55Z

  element.py: Fix cache check in non-strict mode
  
  The behavior of __assert_cached() was changed when the keystrength
  parameter was introduced. This restores the previous behavior.
  
  Fixes #607.
  
  Fixes: 3d308894 ("element.py: Cache the result of checking whether...")
  

• 86ea1173
  by Jürg Billeter at 2018-09-24T13:59:59Z

  tests/frontend/rebuild.py: Add strict/non-strict rebuild test
  
  Test rebuild that requires staging of weakly cached dependency.
  
  This is a regression test for #607.
  

• a76339de
  by Jürg Billeter at 2018-09-24T16:02:38Z

  Merge branch 'juerg/rebuild' into 'master'
  
  element.py: Fix cache check in non-strict mode
  
  Closes #607
  
  See merge request BuildStream/buildstream!822

• b199afe6
  by Jürg Billeter at 2018-09-25T09:01:51Z

  _artifactcache/cascache.py: Add _ensure_blob helper
  
  This adds directory objects to the local repository before downloading
  files in the directory. However, artifact references are still stored
  only after downloading the complete directory and thus, there won't be
  dangling references. This will anyway be required for partial download
  support.
  

• 7d199322
  by Jürg Billeter at 2018-09-25T09:01:51Z

  _artifactcache/cascache.py: Increase payload size limit for uploads
  
  gRPC can handle 1 MiB payloads. Increase size limit from 64 KiB to speed
  up uploads.`
  

• e2e24015
  by Jürg Billeter at 2018-09-25T09:01:51Z

  _artifactcache/casserver.py: Harmonize payload size limit
  
  Use 1 MiB as payload size limit on the server side for both individual
  downloads and batch uploads.
  

• 697d10f2
  by Jürg Billeter at 2018-09-25T09:01:51Z

  _artifactcache/cascache.py: Use BatchReadBlobs
  
  This uses BatchReadBlobs instead of individual blob download to speed up
  artifact pulling, if the server supports it.
  
  Fixes #554.
  

• 81c51dbf
  by Jürg Billeter at 2018-09-25T09:31:55Z

  Merge branch 'juerg/cas-batch' into 'master'
  
  _artifactcache/cascache.py: Use BatchReadBlobs
  
  Closes #554
  
  See merge request BuildStream/buildstream!813

• 2df7d140
  by Jim MacArthur at 2018-09-25T10:36:37Z

  sandbox.py: Stop caching get_virtual_directory if get_directory is used
  
  Also update the documentation for get_virtual_directory to make the
  implications of using both methods clearer.
  

• 62f59382
  by Jim MacArthur at 2018-09-25T10:36:37Z

  element.py/_assemble(): Call get_virtual_directory later.
  

• 8cea7b17
  by Jim MacArthur at 2018-09-25T10:58:40Z

  Merge branch 'jmac/stop-caching-vdirs' into 'master'
  
  Stop caching virtual directories if get_directory is used.
  
  See merge request BuildStream/buildstream!818

• affe04d4
  by Tom Pollard at 2018-09-25T17:08:02Z

  WIP: Don't pull artifact buildtrees by default
  
  - Set default pull to not include buildtree artifact dir
  - PullQueue configurable buildtree attribute
  - Add 'pullbuildtrees' option to existing artifact artifactcache()
    config
  - pullbuildtrees set on a per server basis from user conf
  - Add --pull-buildtrees flag to bst build cli
  - Add --pull-buildtrees flag to bst pull cli
  - Add helper function _fetch_subdir to cascache, to fetch buildtree
    or any other subdir digest
  - Make element._pull_pending not assume no need to process pull if
    artifact is cached if buildtrees are set to be pulled
  - Ensure cascache.py doesn't try to checkout/extract a dangling ref
  
  ToDo:
  - Tests
  

30 changed files:

• CONTRIBUTING.rst
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_artifactcache/casserver.py
• buildstream/_context.py
• buildstream/_exceptions.py
• buildstream/_frontend/app.py
• buildstream/_frontend/cli.py
• buildstream/_frontend/status.py
• buildstream/_fuse/hardlinks.py
• buildstream/_fuse/mount.py
• buildstream/_project.py
• buildstream/_scheduler/jobs/job.py
• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/fetchqueue.py
• buildstream/_scheduler/queues/pullqueue.py
• buildstream/_scheduler/queues/pushqueue.py
• buildstream/_scheduler/queues/queue.py
• buildstream/_scheduler/queues/trackqueue.py
• buildstream/_stream.py
• buildstream/element.py
• buildstream/sandbox/_mount.py
• buildstream/sandbox/_sandboxchroot.py
• buildstream/sandbox/_sandboxremote.py
• buildstream/sandbox/sandbox.py
• buildstream/source.py
• setup.py
• tests/artifactcache/config.py
• + tests/artifactcache/missing-certs/certificates/client.crt
• + tests/artifactcache/missing-certs/certificates/client.key

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.