[Notes] [Git][BuildStream/buildstream][jonathan/workspace-fragment-create] 48 commits: setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52

[Jonathan Maw <gitlab mg gitlab com>]

Title: GitLab

Jonathan Maw pushed to branch jonathan/workspace-fragment-create at BuildStream / buildstream

Commits:

• 499c70fd
  by Valentin David at 2018-11-28T09:11:21Z

  setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52
  

• 3513580c
  by Valentin David at 2018-11-28T14:02:32Z

  Merge branch 'valentindavid/ruamel_0_15_41' into 'master'
  
  setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52
  
  See merge request BuildStream/buildstream!975

• 26cdee08
  by Valentin David at 2018-11-28T14:29:52Z

  Use f_bavail to query available space. Not f_bfree.
  
  f_bfree space might not be usable. In practice we see failures in big
  disks because f_bfree is over 2GB and f_bavail is 0. We get ENOSPC if
  writing on disk then.
  

• 58ca298f
  by Valentin David at 2018-11-28T14:29:52Z

  Make cache clients not fail when a blob is not available.
  
  We plan to make cache incomplete. That is some blobs are missing.  For
  most of cases we will delete references when requested if they are
  incomplete. But there will be corner cases where objects are removed
  after the reference is requested.
  

• 227fa26d
  by Valentin David at 2018-11-28T14:29:52Z

  Fix type error in RPC messages
  

• 5ef19a0b
  by Valentin David at 2018-11-28T14:29:52Z

  Avoid copying temporary file when adding object to CAS in server.
  
  The file is already a temporary file and does not need copy.  ENOSPC
  is thrown during that copy in issue #609.
  
  Fixes #678.
  

• 8d2946ff
  by Valentin David at 2018-11-28T14:29:52Z

  "Fallocate" object disk space to avoid getting NOSPC error when writing
  
  This locks the temporary object file so that cleanup does not need to
  be done for every write.
  

• b587953a
  by Valentin David at 2018-11-28T14:29:52Z

  Update mtimes of objects for requested references.
  
  This also remove references when some objects are missing. This is in
  preparation for the move from reference to object garbage collection.
  

• a64f667d
  by Valentin David at 2018-11-28T14:29:52Z

  Move cas server from ref-based to object-based garbage collection.
  

• 353b90dd
  by Valentin David at 2018-11-28T14:29:52Z

  Cleanup cache in cas server more agressively
  
  When there is less than 2GB left, it cleans up have 10GB available.
  These values are configurable.
  

• ba9afa98
  by Valentin David at 2018-11-28T14:29:52Z

  Lock cache cleanup in cas server
  
  Cleaning up in parallel might slow down the cleaning process
  

• 9a458402
  by Valentin David at 2018-11-29T08:47:40Z

  Merge branch 'valentindavid/cache_server_fill_up' into 'master'
  
  Fix cleanup of cache in server when disk is full
  
  Closes #678
  
  See merge request BuildStream/buildstream!830

• 47b7a9ba
  by Jim MacArthur at 2018-11-29T09:58:48Z

  _sandboxremote.py: Remove unnecessary tests.
  
  push_message and push_directory will both raise assertion failures if
  they fail to send the digest to the server. Checking the digest
  returned by each only tests that the content was hashed locally, which
  cannot reasonably fail.
  

• 4a8d0565
  by Jim MacArthur at 2018-11-29T10:42:31Z

  Merge branch 'jmac/no-verify-digests' into 'master'
  
  _sandboxremote.py: Remove unnecessary tests.
  
  Closes #786
  
  See merge request BuildStream/buildstream!976

• 2e78e0d1
  by Valentin David at 2018-11-29T13:18:17Z

  Add support for .netrc in remote/tar/zip plugins
  
  Fixes #723.
  

• a6144100
  by Valentin David at 2018-11-29T13:18:17Z

  .gitlab-ci.yml: Update image versions where forgotten
  

• f6c184f5
  by Valentin David at 2018-11-29T16:15:56Z

  Merge branch 'valentindavid/netrc' into 'master'
  
  Add support for .netrc in remote/tar/zip plugins
  
  Closes #723
  
  See merge request BuildStream/buildstream!908

• 1f31aefc
  by Angelos Evripiotis at 2018-11-30T09:34:38Z

  man/: update with changes since Apr 2018
  
  It seems that the regenerated man pages haven't been committed for some
  time. Add them all in one lump now.
  
  Had to manually strip the version number from the generated files.
  
  Had to manually fix the NAME field of bst.1 to not be 'bst - None'.
  
  Oddly had to invoke click-man directly in order to get the changes to
  be applied, the usual setup.py route mentioned in contributing.rst
  didn't work.
  
  Eyeballed the result of invoking man on each of the commands, seems
  good.
  

• 033a5ad9
  by Jürg Billeter at 2018-11-30T10:09:11Z

  Merge branch 'aevri/update_man' into 'master'
  
  man/: update with changes since Apr 2018
  
  See merge request BuildStream/buildstream!981

• 1123b9a1
  by Tom Pollard at 2018-11-30T13:30:35Z

  _stream.py: Ability to pull missing buildtrees outside of pull/build
  
  Adds helper function _buildtree_pull_required() to determine if a
  pullqueue should be constructed, for commands outside of bst pull
  and build where it is determined that an element's buildtree
  artifact is to be required given the respective semantics and
  config. Utilised in push() to attempt to mitigate skipping the push
  of partial elements without the user having to have preceded it with
  an explicit pull.
  
  cli.py: Add new behaviour to push command description
  
  element.py: Move _cached_buildtree() to be non local private method,
  use _KeyStrength types to reduce duplication.
  
  tests/integration/pullbuildtrees.py also updated to cover this
  use-case.
  

• ad293ed3
  by Jürg Billeter at 2018-11-30T14:10:06Z

  Merge branch 'tpollard/774' into 'master'
  
  _stream.py: Ability to pull missing buildtrees outside of pull/build
  
  Closes #774
  
  See merge request BuildStream/buildstream!978

• 0d634c21
  by Phillip Smyth at 2018-11-30T14:16:36Z

  plugin.py: Add BAD_ELEMENT_SUFFIX warning
  

• d97ef822
  by Phillip Smyth at 2018-11-30T14:16:36Z

  _project.py: Check for fatal_warnings earlier
  

• 59d442e5
  by Phillip Smyth at 2018-11-30T15:41:57Z

  loader: Added file suffix check and helper function
  
  _loader/loader.py: Added Suffix check and warning for named elements and add helper function
  

• 2d6ec07d
  by Phillip Smyth at 2018-11-30T15:41:57Z

  cli.py: no autocomplete on files without .bst
  

• ac0bc4c0
  by Phillip Smyth at 2018-11-30T16:50:35Z

  Added tests for .bst suffix and completions
  
  buildcheckout.py: Add mandatory .bst suffix tests
  completions.py: Add test for fail on invalid suffix
  Added required files for testing
  integration/source-determinism.py: renamed test elements to end with .bst
  loader/__init__.py: initialised context properly with load and a message handler
  

• d4bc662a
  by Phillip Smyth at 2018-11-30T16:50:50Z

  NEWS: Added update about the Mandatory .bst change
  

• 7cea464a
  by Jürg Billeter at 2018-11-30T17:21:37Z

  Merge branch 'mandatory_suffix' into 'master'
  
  Mandatory .bst suffix
  
  See merge request BuildStream/buildstream!967

• cc5ef68c
  by Tristan Van Berkom at 2018-12-03T08:41:42Z

  MANIFEST.in: Include the sources for rebuilding the documentation session HTML
  
  This was previously missing, causing our CI to forget to rebuild the sessions
  when updating the documentation.
  
  This fixes issue #778
  

• 87ba19d7
  by Tristan Van Berkom at 2018-12-03T09:19:54Z

  Merge branch 'tristan/fix-doc-session-rebuilds' into 'master'
  
  MANIFEST.in: Include the sources for rebuilding the documentation session HTML
  
  Closes #778
  
  See merge request BuildStream/buildstream!984

• 1b24148f
  by Chandan Singh at 2018-12-03T09:46:13Z

  Ensure `--deps=none` option works for `bst checkout`
  
  Currently, `bst checkout --deps none` command always produces empty
  output. Fix this issue and add regression test for the same.
  
  - element_enums.py: Add Scope.NONE.
  
  - element.py: Ensure Scope.NONE works correctly in addition to
    Scope.RUN/Scope.ALL in Element.dependencies() and Element.search().
  
  - tests/frontend/buildcheckout.py: Fix tests for `--deps none`.
  
  Fixes #670.
  

• 8ac1e757
  by Chandan Singh at 2018-12-03T09:46:14Z

  Add `--deps build` option to `bst checkout`
  
  As discussed in
  https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
  add `--deps build` option to `bst checkout`. This will allow users to
  checkout the all build dependencies of a given element using a single
  command.
  
  - _frontend/cli.py: Add `--deps build` option for `bst checkout`.
  
  - element.py: Support `deps='build'` in Element._prepare_sandbox().
  
  - tests/frontend/buildcheckout.py: Ensure `--deps build` works as
    expected.
  

• ac60f47a
  by Benjamin Schubert at 2018-12-03T09:46:14Z

  Refactor and simplify _prepare_sandbox for elements
  
  Before we would have a intricate logics with multiple arguments
  that might get ignored.
  
  This simplifies the design and introduces a bool `shell` instead
  of having two different variables concerned about scope
  

• b1430fb5
  by Tristan Van Berkom at 2018-12-03T11:01:54Z

  Merge branch 'chandan/bst-checkout-build' into 'master'
  
  Add `--deps build` option to `bst checkout`
  
  Closes #670
  
  See merge request BuildStream/buildstream!819

• 9022a778
  by Javier Jardón at 2018-12-03T11:14:55Z

  buildstream/plugins/sources/git.py: warn about checkout by default
  
  See #783
  

• 4462892e
  by Tristan Van Berkom at 2018-12-03T11:45:07Z

  Merge branch 'jjardon/doc_git_submodules' into 'master'
  
  git: warn about checkout submodules  by default
  
  See merge request BuildStream/buildstream!983

• 7805a7ce
  by Jonathan Maw at 2018-12-03T15:40:38Z

  utils.py: Add a helper for searching upwards for files
  
  i.e. with a given directory and filename, check parent directories until
  either a directory with the filename is found, or you reach the root of
  the filesystem.
  
  This is a part of #222
  

• 89147d23
  by Jonathan Maw at 2018-12-03T15:55:29Z

  _workspaces.py: Add a WorkspaceProject object
  
  WorkspaceProject contains all the information needed to refer back to a
  project from its workspace (currently this is the project path and the
  element used to create this workspace)
  
  This is stored within a WorkspaceProjectCache object, which keeps
  WorkspaceProjects around so they don't need to be loaded from disk
  repeatedly.
  
  This is a part of #222
  

• 1d73d3ef
  by Jonathan Maw at 2018-12-03T15:55:29Z

  _project.py: Rename _ensure_project_dir to _find_project_dir
  
  This is a part of #222
  

• d1e8ffc0
  by Jonathan Maw at 2018-12-03T15:55:29Z

  cli.py: Use utils' search upwards helper when searching for project.conf
  
  This is a part of #222
  

• 172fb64e
  by Jonathan Maw at 2018-12-03T15:55:29Z

  _project.py: Find project from workspace if outside of a project
  
  Searching for the workspace file is preferred to searching for the
  project file for the moment, because it avoids the circumstance of
  the workspaces being inside a directory where a higher-level directory
  may have project.conf (but may not be a buildstream project).
  
  This is a part of #222
  

• 87c4b748
  by Jonathan Maw at 2018-12-03T15:55:29Z

  _stream.py: Opened workspaces store the project that opened them
  
  This is a part of #222
  

• 73ae72c5
  by Jonathan Maw at 2018-12-03T15:55:29Z

  _stream.py: workspace_close deletes the stored project reference
  
  This is a part of #222
  

• 3948af12
  by Jonathan Maw at 2018-12-03T15:55:29Z

  _stream.py: Re-create the .bstproject when resetting the workspace
  
  This is a part of #222
  

• d7acd734
  by Jonathan Maw at 2018-12-03T15:55:29Z

  _workspaces.py: Do not include .bstproject.yaml in the cache key
  
  This is a part of #222
  

• f21af132
  by Jonathan Maw at 2018-12-03T15:55:30Z

  cli: Interactively warn if the user is trying to close the workspace they're using to load the project
  
  This involves changes in:
  * _stream.py:
    * Add the helper Stream.workspace_is_required()
  * userconfig.yaml:
    * Add a default value for prompt.really-workspace-close-project-inaccessible
  * _context.py:
    * Load the prompt 'really-workspace-close-project-inaccessible' from
      user config.
  * cli.py:
    * If buildstream is invoked interactively, prompt the user to confirm
      that they want to close the workspace they're using to load this
      project.
  
  This is a part of #222
  

• 2c0c0df3
  by Jonathan Maw at 2018-12-03T15:55:30Z

  tests: Test bst commands from an external workspace
  
  This is a part of #222
  

• 06d7f84d
  by Jonathan Maw at 2018-12-03T15:55:30Z

  NEWS: Add an entry for being able to run commands from a workspace
  
  This is a part of #222
  

30 changed files:

• .gitlab-ci.yml
• MANIFEST.in
• NEWS
• buildstream/_artifactcache/cascache.py
• buildstream/_artifactcache/casserver.py
• buildstream/_context.py
• buildstream/_frontend/cli.py
• buildstream/_loader/loader.py
• buildstream/_project.py
• buildstream/_stream.py
• buildstream/_workspaces.py
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/plugin.py
• buildstream/plugins/sources/_downloadablefilesource.py
• buildstream/plugins/sources/git.py
• buildstream/sandbox/_sandboxremote.py
• buildstream/types.py
• buildstream/utils.py
• dev-requirements.txt
• + man/bst-artifact-server.1
• man/bst-build.1
• man/bst-checkout.1
• man/bst-fetch.1
• + man/bst-help.1
• man/bst-init.1
• man/bst-pull.1
• man/bst-push.1
• man/bst-shell.1
• man/bst-show.1

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.