[Notes] [Git][BuildStream/buildstream][coldtom/strip-rules] 132 commits: Improve documentation for artifact cache installation

[Thomas Coldrick <gitlab mg gitlab com>]

Title: GitLab

Thomas Coldrick pushed to branch coldtom/strip-rules at BuildStream / buildstream

Commits:

• 9b327eb6
  by Ben Brewer at 2018-09-04T14:41:23Z

  Improve documentation for artifact cache installation
  
  Remove ambiguity about systemd service files being separate.
  Add URL for more information about systemd service files.
  Add note about public keys being mandatory for self-signed certs.
  Make cert/key file naming consistent throughout document.
  

• 3e67e64a
  by Javier Jardón at 2018-09-04T16:43:13Z

  Merge branch 'benbrewer/install-artifacts-doc-improvements' into 'master'
  
  Improve documentation for artifact cache installation
  
  See merge request BuildStream/buildstream!777

• 3409609e
  by Daniel Silverstone at 2018-09-04T16:55:51Z

  jobs.py: Reduce FD leaks from queues and process objects
  
  The garbage collector can take too long to get around to cleaning
  up the Queue and Process instances in completed Job instances.  As
  such, FDs tend to leak and in very large projects this can result
  in running out of FDs before a build, fetch, track, or other process
  can complete.  This patch reduces the chance of that by only creating
  the queue when it's needed, and forcing the queue and process instances
  to be deleted when the parent is finished with them.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 7b32e1ec
  by Tristan Maat at 2018-09-04T17:20:55Z

  Merge branch 'danielsilverstone-ct/maybe-reduce-fd-leaks' into 'master'
  
  jobs.py: Reduce FD leaks from queues and process objects
  
  See merge request BuildStream/buildstream!778

• 936bb93a
  by Jim MacArthur at 2018-09-07T12:57:28Z

  cascache.py: Preparation for remote execution
  
  Refactor the push() and pull() implementations so that API additions
  needed for remote-execution is made easier.
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• b0f46545
  by Martin Blanchard at 2018-09-07T12:57:29Z

  cascache.py: Introduce new push helpers
  
  Add push_directory() and push_message() helpers along with a
  verify_digest_pushed().
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• 50bf3139
  by Martin Blanchard at 2018-09-07T12:57:29Z

  cascache.py: Introduce new pull helpers
  
  Add a pull_tree() helper.
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• 853e0af7
  by Martin Blanchard at 2018-09-07T12:57:29Z

  _casbaseddirectory.py: Add a method for hash recalculation
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• e7659c29
  by Jim MacArthur at 2018-09-07T12:57:29Z

  sandbox.py: Allow setting the virtual directory
  
  This is for use after remote execution has finished, since remote
  execution produces a new output directory rather than modifying
  the initial directory.
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• 7306a954
  by Jim MacArthur at 2018-09-07T12:57:29Z

  Add "remote-execution" project configuration option
  
  This just adds one option, "remote-execution/url". Affects multiple files.
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• c0ef7106
  by Jim MacArthur at 2018-09-07T12:57:29Z

  format_project.rst: Document remote-execution option
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• 43651af0
  by Jim MacArthur at 2018-09-07T12:57:29Z

  _sandboxremote.py: Implement the REAPI client
  
  The remote execution client is implemented as a remote sandbox that
  sends sources and build commands to a REAPI server and fetches results
  once remotely executed. New file.
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• da53c005
  by Jim MacArthur at 2018-09-07T12:57:29Z

  sandbox/__init__.py: Add SandboxRemote
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• c73896bb
  by Jim MacArthur at 2018-09-07T12:57:29Z

  element.py: Get the updated virtual directory after running
  
  Executing run() on a sandbox can now replace the virtual directory,
  since remote execution returns a potentially different directory rather
  than an update to the existing one. Call get_virtual_directory() again
  after running to accout for this.
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• 89b29b8b
  by Jim MacArthur at 2018-09-07T12:57:29Z

  element.py: Switch to SandboxRemote if config option is set
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• 116f80d2
  by Jim MacArthur at 2018-09-07T12:57:29Z

  autotools.py: Mark this as a BST_VIRTUAL_DIRECTORY plugin
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• 05956c71
  by Martin Blanchard at 2018-09-07T12:57:29Z

  cmake.py: Mark this as a BST_VIRTUAL_DIRECTORY plugin
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• 909b3be4
  by Martin Blanchard at 2018-09-07T12:57:29Z

  make.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• 556f2abb
  by Martin Blanchard at 2018-09-07T12:57:30Z

  meson.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• 71e2b543
  by Martin Blanchard at 2018-09-07T12:57:30Z

  qmake.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• d4c515d0
  by Martin Blanchard at 2018-09-07T12:57:30Z

  tests/artifactcache: Add a simple test project
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• 923d443b
  by Martin Blanchard at 2018-09-07T13:22:48Z

  tests/artifactcache: Add push unit-tests
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• 9ef53163
  by Martin Blanchard at 2018-09-07T13:22:57Z

  tests/artifactcache: Add pull unit-tests
  
  https://gitlab.com/BuildStream/buildstream/issues/454
  

• f52169b2
  by Qinusty at 2018-09-07T16:48:02Z

  Merge branch 'jmac/remote_execution_client' into 'master'
  
  Remote execution client
  
  See merge request BuildStream/buildstream!626

• 8326b0f7
  by Thomas Coldrick at 2018-09-08T17:54:22Z

  Upstream libtool .la file removal
  
  In freedesktop-sdk we add a script to our project.conf to remove
  libtool .la files from autotools projects after install, this seems
  like a sensible default, so we're attempting to send it upstream.
  

• 886a836e
  by Valentin David at 2018-09-08T20:07:29Z

  Merge branch 'coldtom/autotools-libtool' into 'master'
  
  Upstream freedesktop-sdk autotools config for libtool .la files
  
  See merge request BuildStream/buildstream!683

• 2174db01
  by Tristan Van Berkom at 2018-09-10T07:53:56Z

  tests/artifactcache/expiry.py: Fix test case expectations.
  

• e5847077
  by Tristan Van Berkom at 2018-09-10T07:53:56Z

  _scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE
  
  This is what the whole resource.py thing was created for, the
  cleanup job must have exclusive access to the cache, while the pull
  and build jobs which result in adding new artifacts, must only require
  shared access.
  
  This is a part of #623
  

• 98ab2b89
  by Tristan Van Berkom at 2018-09-10T07:53:56Z

  _scheduler/resources.py: Set CACHE max resources to 0
  
  This means there is no cap for shared resource requests.
  
  Together with the previous commit, this causes the cleanup
  job and the pull/build jobs to all require unlimited shared
  access to the CACHE resource; while only the cleanup job
  requires exclusive access to the resource.
  
  This is a part of #623
  

• deaecdc1
  by Tristan Van Berkom at 2018-09-10T07:53:56Z

  _scheduler/scheduler.py: Make CacheSizeJob() not require the CACHE exclusively
  
  This runs after every pull, and does not need the cache exclusively,
  only the cleanup job requires the cache exclusively.
  
  Without this, every time a cache_size job is queued, all pull and
  build jobs need to complete before cache_size job can run exclusively,
  which is not good.
  
  This is a part of #623
  

• 1f118ba4
  by Tristan Van Berkom at 2018-09-10T07:53:56Z

  element.py: Adding missing API documenting comment
  

• 26ccc1ba
  by Tristan Van Berkom at 2018-09-10T07:53:56Z

  _artifactcache: Making public methods public
  
  The artifact cache provides the following public methods for
  external callers, but was hiding them away as if they are private.
  
    o ArtifactCache.add_artifact_size()
  
    o ArtifactCache.set_cache_size()
  
  Mark these properly public
  

• 00784059
  by Tristan Van Berkom at 2018-09-10T07:53:56Z

  _scheduler: API fixes; _check_cache_size_real() -> check_cache_size()
  
  Here we have a very private looking _check_cache_size_real() function
  which no-one would ever want to call from outside of the _scheduler,
  especially given it's `_real()` prefix we should look for another
  outward facing API to use.
  
  However this is not private to the scheduler, and is intended to
  be called by the `Queue` implementations.
  
    o Renamed this to check_cache_size()
  
    o Moved it to the public API section of the Scheduler object
  
    o Added the missing API documenting comment
  
    o Also added the missing API documenting comment to the private
      `_run_cleanup()` callback which runs in response to completion
      of the cache size calculation job.
  
    o Also place the cleanup job logs into a cleanup subdirectory,
      for better symmetry with the cache_size jobs which now have
      their own subdirectory
  

• a3825ba6
  by Tristan Van Berkom at 2018-09-10T07:53:56Z

  _artifactcache/artifactcache.py: Removing unused variable.
  
  The ArtifactCache._local variable used to exist in order to
  use a special hack to allow absolute paths to a remote artifact
  cache, this was all for the purpose of testing.
  
  This has all gone away with the introduction of CAS, leaving behind
  a stale variable.
  

• 791f7dda
  by Tristan Van Berkom at 2018-09-10T07:53:56Z

  _artifactcache/artifactcache.py: Sealing away some the estimated size
  
  Previously, the API contract was to expose the estimated_size variable
  on the ArtifactCache instance for all to see, however it is only relevant
  to the ArtifactCache abstract class code. Subclasses were informed to
  update the estimated_size variable in their calculate_cache_size()
  implementation.
  
  To untangle this and hide away the estimated size, this commit
  does the following:
  
    o Introduces ArtifactCache.compute_cache_size() API for external
      callers
  
    o ArtifactCache.compute_cache_size() calls the abstract method
      for the CasCache subclass to implement
  
    o ArtifactCache.compute_cache_size() updates the private
      estimated_size variable
  
    o All direct callers to ArtifactCache.calculate_cache_size(),
      have been updated to use the ArtifactCache.compute_cache_size()
      method instead, which takes care of updating anything local
      to the ArtifactCache abstract class code (the estimated_size)
  

• cbfddc13
  by Tristan Van Berkom at 2018-09-10T07:53:56Z

  element.py: Remove _get_artifact_size()
  
  There is no justification to hold onto this state here.
  
  Instead, just make `Element._assemble()` return the size of the
  artifact it cached, and localize handling of that return value in
  the BuildQueue implementation where the value is observed.
  

• 8b748f2d
  by Tristan Van Berkom at 2018-09-10T07:53:57Z

  _scheduler/jobs/cachesizejob.py: Use Platform.get_platform() public accessor
  
  This was previously poking directly at the Platform._instance.
  
  Also, use the name 'artifacts' to hold the artifact cache to
  be consistent with other parts of the codebase, instead of
  calling it 'cache'.
  

• 742908ea
  by Tristan Van Berkom at 2018-09-10T07:53:57Z

  _scheduler/jobs/cleanupjob.py: Use Platform.get_platform() public accessor
  
  This was previously poking directly at the Platform._instance.
  
  Also, use the name 'artifacts' to hold the artifact cache to
  be consistent with other parts of the codebase, instead of
  calling it 'cache'.
  

• 335b2fdd
  by Tristan Van Berkom at 2018-09-10T07:53:57Z

  element.py: Remove _get_artifact_cache() accessor.
  
  The artifact cache is anyway a singleton, the Element itself
  has an internal handle on the artifact cache because it is
  needed very frequently; but an artifact cache is not element
  specific and should not be looked up by surrounding code
  on a per element basis.
  
  Updated _scheduler/queues/queue.py, _scheduler/queues/buildqueue.py
  and _scheduler/jobs/elementjob.py to get the artifact cache directly
  from the Platform
  

• 6db54c8f
  by Tristan Van Berkom at 2018-09-10T07:53:57Z

  _artifactcache: There shalt be only one cache size
  
  This does a lot of house cleaning, finally bringing cache
  cleanup logic to a level of comprehensibility.
  
  Changes in this commit include:
  
    o _artifactcache/artifactcache.py: _cache_size, _cache_quota and
      _cache_lower_threshold are now all private variables.
  
      get_approximate_cache_size() is now get_cache_size()
  
      Added get_quota_exceeded() for the purpose of safely checking
      if we have exceeded the quota.
  
      set_cache_size() now asserts that the passed size is not None,
      it is not acceptable to set a None size cache anymore.
  
    o _artifactcache/cascache.py: No longer set the ArtifactCache
      'cache_size' variable violently in the commit() method.
  
      Also the calculate_cache_size() method now unconditionally
      calculates the cache size, that is what it's for.
  
    o _scheduler/jobs/cachesizejob.py & _scheduler/jobs/cleanupjob.py:
  
      Now check the success status. Don't try to set the cache size
      in the case that the job was terminated.
  
    o _scheduler/jobs/elementjob.py & _scheduler/queues/queue.py:
  
      No longer passing around the cache size from child tasks,
      this happens only explicitly, not implicitly for all tasks.
  
    o _scheduler/queues/buildqueue.py & _scheduler/scheduler.py:
  
      Use get_quota_exceeded() accessor
  
  This is a part of #623
  

• 958ef8ef
  by Tristan Van Berkom at 2018-09-10T09:23:06Z

  Merge branch 'tristan/fix-cache-exclusivity' into 'master'
  
  _scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE
  
  See merge request BuildStream/buildstream!775

• 40f25512
  by Richard Maw at 2018-09-10T12:29:15Z

  subprocesses: Ensure PWD is set in process environment
  
  Naive getcwd implementations (such as in bash 4.4) can break
  when bind-mounts to different paths on the same filesystem are present,
  since the algorithm needs to know whether it's a mount-point
  to know whether it can trust the inode value from the readdir result
  or to use stat on the directory.
  
  Less naive implementations (such as in glibc) iterate again using stat
  in the case of not finding the directory because the inode in readdir was wrong,
  though a Linux-specific implementation could use name_to_handle_at.
  
  Letting the command know what directory it is in makes it unnecessary
  for it to call the faulty getcwd in the first place.
  

• 247686dc
  by richardmaw-codethink at 2018-09-10T14:47:48Z

  Merge branch 'richardmaw/subprocess-PWD' into 'master'
  
  Ensure PWD is set in process environment
  
  See merge request BuildStream/buildstream!782

• ef66daf5
  by Tiago Gomes at 2018-09-10T14:57:41Z

  tests: avoid setting max-jobs
  
  Setting "max-jobs" won't be allowed anymore in a following commit.
  

• a37bc6ce
  by Tiago Gomes at 2018-09-10T14:57:41Z

  docs: document protected variables
  
  And remove then from the defaults as they are dynamically set by
  BuildStream.
  

• af74a3f8
  by Tiago Gomes at 2018-09-10T14:57:41Z

  element: validate configuration variables
  
  Ensure that protected variables are not being redefined by the user.
  

• 75c55633
  by Tiago Gomes at 2018-09-10T14:57:41Z

  Add tests for validating configuration variables
  

• e0bb71b2
  by Tiago Gomes at 2018-09-10T15:27:53Z

  Merge branch 'tiagogomes/issue-287' into 'master'
  
  Add validation of configuration variables
  
  See merge request BuildStream/buildstream!678

• 47f3064a
  by Jürg Billeter at 2018-09-10T15:43:25Z

  _artifactcache/casserver.py: Implement Capabilities service
  

• 1a7fb3cb
  by Jürg Billeter at 2018-09-10T15:43:25Z

  _artifactcache/casserver.py: Implement BatchReadBlobs
  
  Fixes #632.
  

• b3ffcdc8
  by Jürg Billeter at 2018-09-10T16:07:30Z

  Merge branch 'juerg/cas-batch' into 'master'
  
  _artifactcache/casserver.py: Implement BatchReadBlobs
  
  Closes #632
  
  See merge request BuildStream/buildstream!785

• 6f925bcb
  by Javier Jardón at 2018-09-13T07:38:48Z

  source/install_source.rst: pip plugin depends on host pip
  

• c6155f8d
  by Javier Jardón at 2018-09-13T08:11:54Z

  Merge branch 'jjardon/pip_dependency' into 'master'
  
  source/install_source.rst: pip plugin depends on host pip
  
  See merge request BuildStream/buildstream!791

• 19838a07
  by Chandan Singh at 2018-09-13T10:58:38Z

  README.rst: Add status badges for PyPI release and Python versions
  
  The first badge will work fine right away while the second badge will
  show "not found" until a release is made after merging this branch:
  https://gitlab.com/BuildStream/buildstream/merge_requests/718.
  

• 3b81d451
  by Chandan Singh at 2018-09-13T12:27:59Z

  Merge branch 'chandan/pypi-badge' into 'master'
  
  README.rst: Add status badges for PyPI release and Python versions
  
  See merge request BuildStream/buildstream!719

• 55956762
  by Richard Maw at 2018-09-13T16:50:33Z

  element: Remove __cached field
  
  This can get out of sync with other two cache states,
  and we can do without it.
  

• fc7f83ac
  by richardmaw-codethink at 2018-09-13T17:14:31Z

  Merge branch 'richardmaw/element-cache-state-simplify' into 'master'
  
  Simplify element state by removing `__cached`
  
  See merge request BuildStream/buildstream!784

• 233a7d83
  by Richard Maw at 2018-09-14T08:53:14Z

  Handle sockets when copying trees into artifacts
  
  We can't include a socket in a CAS tree, but it's mostly meaningless to do so
  since there can't possibly be a process serving it.
  

• f06f234a
  by Richard Maw at 2018-09-14T08:53:14Z

  tests: cover builds that make sockets
  

• f86ab8f6
  by richardmaw-codethink at 2018-09-14T09:46:07Z

  Merge branch 'richardmaw/builddir-sockets' into 'master'
  
  Fix: While caching build artifact: "Cannot extract [path to socket file] into staging-area. Unsupported type."
  
  See merge request BuildStream/buildstream!783

• e7427462
  by Richard Maw at 2018-09-14T10:28:17Z

  tests: Use cli.configure for custom logging test
  
  Overriding the config with a custom config file on the command-line
  prevents it merging in the test-specific configuration
  and can permit it to attempt to initialise the user's cache.
  

• 800a8403
  by Richard Maw at 2018-09-14T10:28:17Z

  Tidy relative workspace tests
  
  They were moving the whole tmpdir to move the project repository.
  This moves the cache directories etc. as well, meaning cli.run can't find them.
  This was worked around by setting configure=False,
  but this has the side-effect of making use of the user's caches,
  which it should not be doing for reproducibility reasons.
  
  By changing the tempdir layout to have the project in a subdirectory
  we can move the project around for the relative workspace tests
  without losing track of the configured state directories,
  so we can leave configure=True and avoid touching the user's caches.
  

• d7152ef4
  by richardmaw-codethink at 2018-09-14T10:55:16Z

  Merge branch 'richardmaw/test-config-fixes' into 'master'
  
  Fix tests that attempt to access the home directory
  
  See merge request BuildStream/buildstream!780

• 160bb0c6
  by Tristan Van Berkom at 2018-09-14T12:07:46Z

  _artifactcache/artifactcache.py: Changes to mark_required_elements()
  
  This was previously append_required_artifacts(), which presumed that
  we knew at startup time what the cache keys of all elements in the
  loaded pipeline would be.
  
  This fixes unexpected deletions of required artifacts when
  dynamic tracking is enabled with `bst build --track-all target.bst`
  

• 39125d24
  by Tristan Van Berkom at 2018-09-14T12:07:46Z

  tests/artifactcache/expiry.py: Cleanup of test for required artifacts
  
  This commit renames test_never_delete_dependencies() to
  test_never_delete_required(), renders the test more readable by renaming
  some elements and reordering some statements and makes the comments more
  straight forward and accurate.
  

• f60558a3
  by Tristan Van Berkom at 2018-09-14T12:07:46Z

  tests/artifactcache/expiry.py: Assert the expected errors
  
  These tests were not checking that we fail for the expected reasons.
  
  Added `res.assert_task_error(ErrorDomain.ARTIFACT, 'cache-too-full')`
  where we expect to fail because the cache is too full.
  

• ce68fd27
  by Tristan Van Berkom at 2018-09-14T12:07:46Z

  testutils/repo/git.py: Added modify_file() method
  
  This allows one to modify a file in an existing git repo,
  as opposed to adding a new one.
  

• 532ec1eb
  by Tristan Van Berkom at 2018-09-14T12:07:46Z

  testutils/element_generators.py: Changing sized element functions
  
   * create_element_size()
  
     Now uses a git Repo object instead of a local source, and
     returns the repo.
  
   * update_element_size()
  
     Added this function which can now resize the expected output
     of an element generated with create_element_size(), useful
     to allow testing sized elements with the tracking feature.
  

• 20b797cb
  by Tristan Van Berkom at 2018-09-14T12:07:46Z

  tests/artifactcache/expiry.py: Added test_never_delete_required_track()
  
  Same test as test_never_delete_required(), except that this test ensures
  that we never delete required artifacts when their cache keys are
  discovered dynamically during the build.
  

• c2af0d51
  by Tristan Van Berkom at 2018-09-14T12:44:42Z

  Merge branch 'tristan/fix-required-artifacts' into 'master'
  
  Don't delete required artifacts when tracking is enabled
  
  See merge request BuildStream/buildstream!793

• 924cdc75
  by Tiago Gomes at 2018-09-14T15:32:01Z

  core: remove unused variable on Context
  

• 510ccbfd
  by Tiago Gomes at 2018-09-14T15:32:21Z

  artifactcache: fix oversight
  
  We want to check if some file is already cached here, not the parent
  directory.
  

• 82d4e2ac
  by Tiago Gomes at 2018-09-14T15:32:21Z

  scheduler: add missing copyright
  
  Tristan Maat created the original file, so he is added as the author.
  

• 18b37aab
  by Tiago Gomes at 2018-09-14T15:34:10Z

  artifactcache: improve _create_tree()
  
  * Rename it to _commit_directory() because… it is what it does; and
    also for symmetry with _fetch_directory().
  * Rename digest to dir_digest to make it clear this is a digest for a
    directory. A following commit will also reuse the same variable name
  * Document method.
  

• 32fad24f
  by Tiago Gomes at 2018-09-14T15:38:41Z

  artifactcache: improve _fetch_directory()
  
  * Rename tree to dir_digest to make it clear this is a Digest object,
    and not a Tree object.
  * Add documentation
  

• 46cbd889
  by Tiago Gomes at 2018-09-14T15:38:43Z

  artifactcache: rename get_quota_exceeded()
  

• 2fa92716
  by Tiago Gomes at 2018-09-14T15:38:43Z

  scheduler: minor cleanups to CleanupJob
  
  Remove unneeded cruft.
  

• c285f244
  by Tiago Gomes at 2018-09-14T16:08:07Z

  Merge branch 'tiagogomes/some-cleanups' into 'master'
  
  Bunch of cleanups
  
  See merge request BuildStream/buildstream!798

• 662d22a9
  by Tristan Van Berkom at 2018-09-16T08:54:00Z

  buildstream/_frontend/complete.py: Added missing click license compliance
  
  Somehow I missed this when originally forking the file from the click
  library, now noticing that we should have followed what was written
  in: https://github.com/pallets/click/blob/master/LICENSE
  

• a344407b
  by Richard Maw at 2018-09-16T10:22:59Z

  sandbox: deduplicate default environment logic
  

• c8fab440
  by Richard Maw at 2018-09-16T10:22:59Z

  Bump Artifact Version
  
  Since we now set PWD in the environment of builds
  existing builds may behave differently so must cache differently now.
  

• eb9481da
  by Richard Maw at 2018-09-16T10:22:59Z

  Cachekey update helper: Document ensuring . in PYTHONPATH
  
  The current directory isn't always in the python module search path,
  so we have to ensure it is for the script to work.
  
  Strictly speaking, the user may already have a modified PYTHONPATH
  at which point PYTHONPATH=".${PYTHONPATH+:$PYTHONPATH}" is necessary,
  but it's probably premature to overcomplicate the documentation like that
  before we discover it's a problem.
  

• 9b7b2a25
  by Tristan Van Berkom at 2018-09-16T10:50:36Z

  Merge branch 'richardmaw/subprocess-PWD' into 'master'
  
  Address post-merge review of Ensure PWD is set in process environment
  
  See merge request BuildStream/buildstream!788

• 8db62230
  by Adam Jones at 2018-09-17T07:39:06Z

  Add a section on how to raise an issue
  

• 6f0a3e98
  by Adam Jones at 2018-09-17T07:39:20Z

  Rename HACKING.rst to CONTRIBUTING.rst
  

• ade19151
  by Adam Jones at 2018-09-17T07:40:45Z

  update HACKING references
  

• b9c5ff7d
  by Tristan Van Berkom at 2018-09-17T08:09:12Z

  Merge branch 'tristan/contributing-guide' into 'master'
  
  Update contributing guide
  
  See merge request BuildStream/buildstream!801

• 727f2faa
  by Tristan Van Berkom at 2018-09-18T07:43:07Z

  _project.py: Fix option resolution in element & source overrides
  
  This ensures that option expressions are resolved in the project
  level overrides before attempting to composite them on the instantiated
  elements. Seems this is a regression from introducing the include
  directive.
  
  This fixes issue #658
  

• ffa0bb36
  by Tristan Van Berkom at 2018-09-18T07:47:44Z

  tests/format/optionoverrides.py: Added test for options in element overrides
  
  This is a regression test for issue #658
  

• f2ae46f8
  by Tristan Van Berkom at 2018-09-18T08:14:23Z

  Merge branch 'tristan/fix-override-options' into 'master'
  
  Fix override options
  
  Closes #658
  
  See merge request BuildStream/buildstream!802

• 7b117e40
  by Daniel Silverstone at 2018-09-18T08:41:32Z

  _artifactcache/artifactcache.py: Ensure no double-setup of remotes
  
  Since ArtifactCache.setup_remotes() can be expensive and should only
  happen once, this commit will assert() if it is called a second time
  on an artifact cache instance.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 345f5f49
  by Daniel Silverstone at 2018-09-18T08:41:32Z

  tests/artifactcache/pull.py: Do not double-initialize remotes
  
  The initialization of remotes is done by ArtifactCache.setup_remotes()
  and as such it was wrong for these tests to be calling
  CASCache.initialize_remotes() a second time.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• e32221b6
  by Daniel Silverstone at 2018-09-18T08:45:50Z

  sandbox/_sandboxremote.py: Acquire artifact cache via Platform
  
  The SandboxRemote used to construct its own CASCache which was
  considered dangerous.  This patch replaces that with acquisition of
  the cache via the Platform singleton, hopefully eliminating issues
  from having more than one artifact cache object in a single process.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• b587579f
  by Tristan Van Berkom at 2018-09-18T09:53:26Z

  Merge branch 'danielsilverstone-ct/fix-654' into 'master'
  
  sandbox/_sandboxremote.py: Acquire cache via Platform
  
  See merge request BuildStream/buildstream!797

• 30b41959
  by Tristan Van Berkom at 2018-09-18T09:56:45Z

  _artifactcache/artifactcache.py: Error out gracefully when push remote is mal-specified
  
  When configuring a push remote and specifying either the client-cert
  or the client-key, then both must be specified. This ensures we
  get an informative error instead of a stack trace and BUG.
  
  Fixes issue #625
  

• 41e8dc81
  by Tristan Van Berkom at 2018-09-18T09:56:45Z

  tests/artifactcache/config.py: Added test for invalid push remote configuration
  
  Test that we get the expected error when configuring a client-cert
  without client-key, or the inverse.
  

• 97071b6e
  by Tristan Van Berkom at 2018-09-18T10:16:43Z

  Merge branch 'tristan/fix-artifact-config-crash' into 'master'
  
  Fix artifact config crash
  
  Closes #625
  
  See merge request BuildStream/buildstream!804

• 081dcafa
  by Richard Maw at 2018-09-18T13:22:38Z

  fuse: Report the correct device number for devices
  
  This fixes all devices being mapped to the non-existant device 0,
  which prevents being able to use even safe devices like /dev/null
  through the hardlinks FUSE layer.
  

• d0425608
  by Richard Maw at 2018-09-18T13:22:38Z

  FUSE: Mount with -odev in chroot sandbox
  
  This is needed to permit access to the device nodes added to /dev
  on Linux when FUSE is used as root.
  
  The chroot sandbox only works with all privileges,
  so there's no explicit check for being root
  or having the appropriate capabilities.
  
  A check for whether it's running as root isn't needed on Linux with bubblewrap
  because /dev or its devices are mounted on top of the FUSE layer,
  so device nodes are accessed directly rather than through the FUSE layer.
  

• 8430fdc7
  by Richard Maw at 2018-09-18T13:22:38Z

  tests: test that integration commands can use /dev
  

• d8450166
  by Tristan Van Berkom at 2018-09-18T13:50:15Z

  Merge branch 'richardmaw/fix-chroot-sandbox-devices' into 'master'
  
  fix chroot sandbox devices
  
  See merge request BuildStream/buildstream!781

• ba4da3a4
  by knownexus at 2018-09-18T16:45:09Z

  Incorrect error when malformed project.conf
  
  bst build returns "missing project.conf" when a project.conf is invalid
  This results in an existing project with malformed yaml being dismissed
  and attempts to make a new project
  
  Added new exception
  

• 89a16e17
  by knownexus at 2018-09-18T16:45:09Z

  Adding test for Invalid Yaml
  

• 72b59021
  by Phillip Smyth at 2018-09-19T08:48:51Z

  Merge branch 'issue-642-Invalid_project.conf_seen_as_missing' into 'master'
  
  Incorrect error when malformed project.conf
  
  Closes #642
  
  See merge request BuildStream/buildstream!792

• ca811a4d
  by Josh Smith at 2018-09-19T09:25:21Z

  Rework Skipped usage
  
  The SKIPPED message type is now used to indicate the end of a task which
  was successful without having to perform the given task.
  
  This overhauls the use of `Queue.done()` and therefore queues do not
  need to provide a processed/skipped return value from `done()`. Instead
  this is replaced with the action of raising a `SkipJob` exception from
  within `Queue.process()`.
  

• 16fbc3e3
  by Josh Smith at 2018-09-19T09:25:21Z

  tests.py: Test skip on push
  
  Adds a test to ensure that BuildStream alerts the user of a skipped push
  when the remote already has the artifact cached.
  

• ab56f693
  by Josh Smith at 2018-09-19T09:25:21Z

  element.py: Remove redundant timed_activity
  
  This removes the timed_activity for an element _push action. This is
  unnecessary as the job is already being timed elsewhere.
  

• 198fbfe8
  by Josh Smith at 2018-09-19T09:25:21Z

  cascache.py: Move push/pull messaging to cascache
  
  Pulled/Pushed messages will no longer be produced from within
  element.py, instead they will be produced during CasCache push() and
  pull() appropriately.
  
  Message consistency has also been improved.
  

• 213f7794
  by Josh Smith at 2018-09-19T09:25:21Z

  cascache.py: Modify messaging API calls
  
  Both pulling and pushing INFO messages are now status messages.
  
  Calls to the messaging API through `self.context.message()` have now
  been switched to `element.info`.
  

• dddd6025
  by Qinusty at 2018-09-19T10:59:48Z

  Merge branch 'Qinusty/skipped-rework' into 'master'
  
  Add SkipJob for indicating a skipped activity
  
  See merge request BuildStream/buildstream!765

• 16d9c6e5
  by Tristan Van Berkom at 2018-09-19T11:08:33Z

  _frontend/status.py: Completely remove the blessings dependency from BuildStream
  
  This actually improves reliability of the status bar because we
  now disable it completely in the case that not all of the terminal
  escape sequences are supported on the given terminal.
  
  This replaces the few functions we were using, to move the cursor
  up one line, move it to the beginning of the line, and to clear a
  line, with low level functions provided by the curses module in
  the standard library.
  
  This change makes it easier for downstream distro package maintainers
  to package BuildStream, particularly on Fedora.
  
  Asides from changing _frontend/status.py, this commit includes the
  following changes:
  
    * _frontend/app.py: Use python isatty() function to determine if
                        we are connected to a tty, instead of relying
  		      on blessings.
  
    * setup.py: Remove the dependency on blessings.
  

• 7dd74b01
  by Tristan Van Berkom at 2018-09-19T11:27:55Z

  Merge branch 'tristan/remove-blessings' into 'master'
  
  _frontend/status.py: Completely remove the blessings dependency from BuildStream
  
  See merge request BuildStream/buildstream!808

• 335d5a45
  by Adam Jones at 2018-09-19T13:00:45Z

  Fix rst links for guideline links
  

• 667dd4ca
  by Javier Jardón at 2018-09-19T13:25:36Z

  Merge branch 'adamjones/contributing-links' into 'master'
  
  Fix rst link formatting for guideline links
  
  See merge request BuildStream/buildstream!811

• 68ef69e4
  by Tristan Van Berkom at 2018-09-21T05:20:46Z

  source.py: Fix re-instantiation
  
  This fixes an issue where the re-instantiated Source used with
  Source mirroring enabled is not completely initialized.
  
  Failing to load the ref from the project.refs file for instance,
  will result in a crash at `fetch` time.
  
  This fixes issue #666
  

• 662c729f
  by Tristan Van Berkom at 2018-09-21T05:59:30Z

  Merge branch 'tristan/fix-source-reinstantiation' into 'master'
  
  source.py: Fix re-instantiation
  
  Closes #666
  
  See merge request BuildStream/buildstream!817

• 461a0588
  by Jim MacArthur at 2018-09-21T10:53:11Z

  element.py: Remove early warning and check directory is not None
  
  The check for directory should have always been here. The early warning
  test doesn't work for `bst checkout` because caches haven't been configured
  at the point of the test, so I've had to remove it.
  

• aa9caaac
  by Jim MacArthur at 2018-09-21T10:53:11Z

  _sandboxremote.py: Ignore return value of push_directory
  
  The return value is always upload_vdir.ref.
  

• 2aae68c7
  by Jim MacArthur at 2018-09-21T10:53:11Z

  cascache.py: make push_remote raise exception if no push remotes exist
  
  Also add docstring to method and remove return value since it was useless
  

• ca1bb72c
  by Jim MacArthur at 2018-09-21T10:53:11Z

  tests/artifactcache/push.py: Ignore return value of push_directory
  

• 55c93a82
  by Jim MacArthur at 2018-09-21T11:26:55Z

  Merge branch 'jmac/remote_exec_checkout_fix' into 'master'
  
  Remote exec: Remove early warning and check directory is not None
  
  See merge request BuildStream/buildstream!800

• e209beb0
  by Chandan Singh at 2018-09-21T13:10:08Z

  _stream.py: Ensure source-bundle's source directory exists
  
  Currently, `source-bundle` command is entirely broken as it tries to stage the
  sources in a directory that doesn't exist. Fix it by ensuring that we create
  the necessary directories before calling any methods that try to use those
  directories.
  
  This fix comes with a regression test to ensure that the basic use-case
  of `source-bundle` continues to work in future.
  
  Fixes https://gitlab.com/BuildStream/buildstream/issues/651.
  

• 0b000518
  by Chandan Singh at 2018-09-21T13:56:55Z

  Merge branch 'chandan/fix-source-bundle' into 'master'
  
  Fix source-bundle command
  
  Closes #651
  
  See merge request BuildStream/buildstream!807

• ef26043a
  by Chandan Singh at 2018-09-21T17:14:16Z

  setup.py: Make website the primary homepage
  

• 1b2aed40
  by Chandan Singh at 2018-09-21T17:40:11Z

  Merge branch 'chandan/update-project-homepage' into 'master'
  
  setup.py: Make website the primary homepage
  
  See merge request BuildStream/buildstream!795

• da7e038b
  by Jürg Billeter at 2018-09-24T13:58:55Z

  element.py: Fix cache check in non-strict mode
  
  The behavior of __assert_cached() was changed when the keystrength
  parameter was introduced. This restores the previous behavior.
  
  Fixes #607.
  
  Fixes: 3d308894 ("element.py: Cache the result of checking whether...")
  

• 86ea1173
  by Jürg Billeter at 2018-09-24T13:59:59Z

  tests/frontend/rebuild.py: Add strict/non-strict rebuild test
  
  Test rebuild that requires staging of weakly cached dependency.
  
  This is a regression test for #607.
  

• a76339de
  by Jürg Billeter at 2018-09-24T16:02:38Z

  Merge branch 'juerg/rebuild' into 'master'
  
  element.py: Fix cache check in non-strict mode
  
  Closes #607
  
  See merge request BuildStream/buildstream!822

• 61acefbb
  by Thomas Coldrick at 2018-09-27T11:06:05Z

  Upstream freedesktop-sdk strip rules
  
  Adds a modified version of the freedesktop-sdk `strip-binaries` to
  buildstream default `project.conf`. We implemented a few changes to
  the current default, but the main change is to fix a bug due to
  permissions.
  
  The modifications I made to the freedesktop-sdk commands as they are
  at present are as follows:
  * Remove debugedit rules to include source debugging
  * Add a switch to allow failing on error, rather than this being
    the default behaviour.
  These changes mean that we should now have a sensible default for
  a wide variety of users.
  

• 0ab3e9a0
  by Thomas Coldrick at 2018-09-27T11:06:05Z

  Try with less fatal commands
  

• 6769ffa8
  by Thomas Coldrick at 2018-09-27T11:06:05Z

  Switch to using fatal-strip
  

• 0d409ae1
  by Thomas Coldrick at 2018-09-27T11:07:02Z

  tests/: Update expected cache keys
  

30 changed files:

• HACKING.rst → CONTRIBUTING.rst
• MANIFEST.in
• README.rst
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_artifactcache/casserver.py
• buildstream/_context.py
• buildstream/_exceptions.py
• buildstream/_frontend/app.py
• buildstream/_frontend/complete.py
• buildstream/_frontend/status.py
• buildstream/_fuse/hardlinks.py
• buildstream/_fuse/mount.py
• buildstream/_project.py
• buildstream/_scheduler/jobs/__init__.py
• buildstream/_scheduler/jobs/cachesizejob.py
• buildstream/_scheduler/jobs/cleanupjob.py
• buildstream/_scheduler/jobs/elementjob.py
• buildstream/_scheduler/jobs/job.py
• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/fetchqueue.py
• buildstream/_scheduler/queues/pullqueue.py
• buildstream/_scheduler/queues/pushqueue.py
• buildstream/_scheduler/queues/queue.py
• buildstream/_scheduler/queues/trackqueue.py
• buildstream/_scheduler/resources.py
• buildstream/_scheduler/scheduler.py
• buildstream/_stream.py
• buildstream/_versions.py
• buildstream/buildelement.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.