[Notes] [Git][BuildStream/buildstream][valentindavid/deterministic-source] 36 commits: PullQueue: fix resource used

[Valentin David <gitlab mg gitlab com>]

Title: GitLab

Valentin David pushed to branch valentindavid/deterministic-source at BuildStream / buildstream

Commits:

• 4d6dbabe
  by Tiago Gomes at 2018-08-06T15:03:25Z

  PullQueue: fix resource used
  
  Improve fetchers, builders and pushers documentation.
  
  Closes #550.
  

• a8073264
  by Tiago Gomes at 2018-08-06T16:55:03Z

  Merge branch 'tiagogomes/issue-550' into 'master'
  
  PullQueue: fix resource used
  
  Closes #550
  
  See merge request BuildStream/buildstream!611

• 3d26ff6e
  by Ed Baunton at 2018-08-07T05:04:50Z

  remote.py: Add support for marking downloaded files executable
  
  Add an optional flag to make files executable after having downloaded them.
  
  Instead of leaving the permissioning of downloaded file in remote.py up
  to the user's umask; expressly set permissions to 0644 or 0755 if
  executable.
  
  Bump format version to 13.
  

• 60a552a5
  by Tristan Van Berkom at 2018-08-07T06:13:53Z

  Merge branch 'edbaunton/executable-remote-source' into 'master'
  
  remote.py: Add support for marking downloaded files executable
  
  See merge request BuildStream/buildstream!581

• 01ff0177
  by Javier Jardón at 2018-08-07T07:29:30Z

  source/install_linux_distro.rst: Make clearer ArchLinux packages available
  

• 9b469fe6
  by Tristan Van Berkom at 2018-08-07T09:12:46Z

  Merge branch 'jjardon/doc_arch_releases' into 'master'
  
  source/install_linux_distro.rst: Make clearer ArchLinux packages available
  
  See merge request BuildStream/buildstream!613

• 289de2ef
  by Adam Jones at 2018-08-07T09:24:58Z

  Add systemd service file examples to artifacts documentation
  
  change code style to ini
  

• e42325b9
  by Javier Jardón at 2018-08-07T10:29:46Z

  Merge branch 'adamjones/systemd-cas' into 'master'
  
  Include systemd file examples for google-cas cache
  
  Closes #419
  
  See merge request BuildStream/buildstream!524

• 6d349610
  by Javier Jardón at 2018-08-07T10:53:23Z

  source/install_linux_distro.rst: Fix formatting
  

• 493d19d2
  by James Ennis at 2018-08-07T11:08:36Z

  install_linux_distro.rst: Change install instructions for python-arpy package on Arch
  
    - python-arpy is unobtainable via pacman
    - Also a minor change to how we display install instructions
    from Fedora's dnf package manager
  

• fc9869e2
  by James Ennis at 2018-08-07T11:56:21Z

  Merge branch 'jennis/change_arch_install_instructions' into 'master'
  
  Change install instructions for python-arpy package on Arch
  
  See merge request BuildStream/buildstream!614

• 42aa3999
  by William Salmon at 2018-08-07T13:41:02Z

  Add warning to git track if track and ref are not present
  
  This is to address https://gitlab.com/BuildStream/buildstream/issues/471 that
  documented unhelpfull behavour when tracking git sources.
  

• 2ceb5dec
  by Will Salmon at 2018-08-07T14:46:36Z

  Merge branch 'willsalmon/trackWarning' into 'master'
  
  Add warning to git track if track and ref are not present
  
  Closes #471
  
  See merge request BuildStream/buildstream!580

• eee4b674
  by Jürg Billeter at 2018-08-07T15:36:35Z

  Revert "Restrict version of pylint"
  
  This reverts commit 4f168b9b6a02216e2fae24d758ae6b778e545869.
  
  The latest version of pytest_pylint works fine with pylint 2, which
  means there is no longer a reason to restrict the pylint version.
  
  pylint 2 is required for Python 3.7.
  

• ea27e389
  by Jürg Billeter at 2018-08-07T15:36:35Z

  _artifactcache/cascache.py: Fix for PEP 479 / Python 3.7
  
  Do not rely on `StopIteration` bubbling up.
  
  https://www.python.org/dev/peps/pep-0479/
  

• fa5a59f0
  by Jürg Billeter at 2018-08-07T16:48:21Z

  Merge branch 'juerg/python3.7' into 'master'
  
  Python 3.7 support
  
  See merge request BuildStream/buildstream!615

• ed653fbc
  by Chandan Singh at 2018-08-08T00:06:28Z

  .gitlab-ci.yml: Remove unused script "install.sh"
  

• 65f382f1
  by Chandan Singh at 2018-08-08T10:52:32Z

  Merge branch 'chandan/remove-dead-code' into 'master'
  
  .gitlab-ci.yml: Remove unused script "install.sh"
  
  See merge request BuildStream/buildstream!617

• c68dcab8
  by Tiago Gomes at 2018-08-09T09:18:43Z

  cascache: move tmp directory one level up
  
  The CAS uses a temp directory while manipulating the cache, temporary
  files can be added and removed while adding artifacts to the cache here.
  
  Since calculation of the cache size happens in parallel to artifact
  cache additions, this causes race conditions in the size calculation
  job, as we end up calling `stat` on temporary files which are being
  removed in parallel.
  
  Handle this by moving the temporary directory out of the way, and avoid
  considering the tmp directory when calculating the cache size
  

• 35ab0335
  by Tiago Gomes at 2018-08-09T10:16:24Z

  Merge branch 'tiagogomes/issue-520' into 'master'
  
  Fix race condition when calculating disk usage
  
  See merge request BuildStream/buildstream!600

• 8aa33e23
  by Valentin David at 2018-08-09T12:49:17Z

  Keep original flags for create in SafeHardlinks.
  
  When open(2) is used with flags O_CREAT|O_RDWR, the file descriptor
  must be readable. Unfortunately O_RDWR was not passed which made
  read fail with EBADF and mmap to signal SIGBUS.
  
  This issue happened with man-db for example.
  
  Fixes #143.
  

• ef7810f3
  by Valentin David at 2018-08-09T14:06:50Z

  Merge branch 'valentindavid/fuse-create-flags' into 'master'
  
  Keep original flags for create in SafeHardlinks.
  
  Closes #143
  
  See merge request BuildStream/buildstream!624

• 2d061173
  by Javier Jardón at 2018-08-09T16:31:15Z

  buildstream/_project.py:
  
  Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for
  systems where not all CPU can be used (in particular, Docker, CI etc).
  
  As suggested in the multiprocessing docs
  (http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count)
  using len(os.sched_getaffinity(0)) might be better.
  

• 4e1488ee
  by Javier Jardón at 2018-08-09T16:31:15Z

  buildstream/_project.py: Restrict max-jobs
  
  even if the machine have mores cores available
  
  Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227
  

• de955834
  by Javier Jardón at 2018-08-09T17:22:24Z

  Merge branch 'jjardon/max-jobs' into 'master'
  
  buildstream/_project.py: Restrict max-jobs
  
  See merge request BuildStream/buildstream!620

• 97595585
  by Jim MacArthur at 2018-08-09T17:35:19Z

  setup.py: Pin ruamel.yaml version to <= 0.15
  

• a602365c
  by Tristan Van Berkom at 2018-08-09T20:03:52Z

  Merge branch 'jmac/pin-ruamel-version' into 'master'
  
  Pin ruamel.yaml version to <= 0.15
  
  See merge request BuildStream/buildstream!630

• db0478ab
  by Phillip Smyth at 2018-08-10T10:17:47Z

  Implementing relative workspaces
  
  This fixes #191
  
  A note has been added to NEWS explaining backwards
  compatibility issues
  

• 04cee9a9
  by Phillip Smyth at 2018-08-10T11:12:31Z

  Merge branch 'relative_workspaces' into 'master'
  
  Patch for issue #191 support relative workspaces
  
  Closes #191
  
  See merge request BuildStream/buildstream!504

• 5dcecbad
  by Valentin David at 2018-08-10T12:18:55Z

  Set environment in bwrap command line instead of its environment
  
  Fixes #498
  

• 2e8db54e
  by Valentin David at 2018-08-10T12:53:19Z

  Merge branch 'valentindavid/498_bwrap_environment' into 'master'
  
  Set environment in bwrap command line instead of its environment
  
  Closes #498
  
  See merge request BuildStream/buildstream!565

• a250b5f7
  by Valentin David at 2018-08-10T14:21:41Z

  buildstream/plugins/sources/local.py: Make staging deterministic.
  
  Instead of copying metadata on files staged by local, we manually set
  mode to 0755 or 0644 depending on whether user execution was enabled
  on source file.
  
  This makes file modes deterministic independently on the way source
  was distributed.
  
  Non-deterministic mode copying all metadata can still be enabled by
  disable 'deterministic' Boolean configuration on the plugin.
  
  Fixes #527.
  

• 1b9c9277
  by Valentin David at 2018-08-10T14:21:41Z

  Use deterministic umask when staging sources.
  
  This fix is applied to plugins bzr, git, patch.
  
  Fixes #543 #544 #555.
  

• 1950f1d2
  by Valentin David at 2018-08-10T14:21:41Z

  buildstream/plugins/sources/zip.py: Fix non-determism in staging.
  
  Staging could end-up with file with different rights depending on the
  umask. The extracted files need to get their access rights fixed.
  

• b881de3a
  by Valentin David at 2018-08-10T14:22:02Z

  Add some integration tests for source plugin determinism.
  

• c6e42f81
  by Valentin David at 2018-08-10T14:22:02Z

  Bump BST_CORE_ARTIFACT_VERSION for deterministic source plugins
  

30 changed files:

• .gitlab-ci.yml
• NEWS
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_artifactcache/casserver.py
• buildstream/_frontend/widget.py
• buildstream/_fuse/fuse.py
• buildstream/_fuse/hardlinks.py
• buildstream/_pipeline.py
• buildstream/_project.py
• buildstream/_scheduler/queues/pullqueue.py
• buildstream/_stream.py
• buildstream/_versions.py
• buildstream/_workspaces.py
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/plugins/sources/git.py
• buildstream/plugins/sources/local.py
• buildstream/plugins/sources/remote.py
• buildstream/plugins/sources/zip.py
• buildstream/sandbox/_sandboxbwrap.py
• buildstream/utils.py
• doc/source/install_artifacts.rst
• doc/source/install_linux_distro.rst
• setup.py
• tests/cachekey/project/elements/build1.expected
• tests/cachekey/project/elements/build2.expected
• tests/cachekey/project/elements/compose1.expected
• tests/cachekey/project/elements/compose2.expected
• tests/cachekey/project/elements/compose3.expected

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.