[Notes] [Git][BuildStream/buildstream][tiagogomes/issue-573] 19 commits: Allow source plugins to access previous sources

[Tiago Gomes <gitlab mg gitlab com>]

Title: GitLab

Tiago Gomes pushed to branch tiagogomes/issue-573 at BuildStream / buildstream

Commits:

• 9d2442b2
  by Chandan Singh at 2018-08-15T17:29:07Z

  Allow source plugins to access previous sources
  
  Source plugin implementations can now specify that they need access to
  previously staged sources by specifying
  `BST_REQUIRES_PREVIOUS_SOURCES_TRACK` and/or
  `BST_REQUIRES_PREVIOUS_SOURCES_FETCH`, corresponding to access at `track`
  and `fetch` times respectively.
  
  Fixes #381.
  
  Replaces !505. For relevant discussion, see this discussion:
  https://gitlab.com/BuildStream/buildstream/merge_requests/505#note_83780747
  

• a7a82650
  by Chandan Singh at 2018-08-15T18:23:04Z

  Add pip source plugin
  
  `pip` source plugin can stage python packages that are either specified
  directly in the element definition or picked up from `requirements.txt`
  from previous sources. In order to support the latter use-case
  (which is also the primary motivation for this plugin), this plugin
  requires access to previous sources and hence is an example of a
  Source Transform source.
  
  Also, bump `BST_FORMAT_VERSION` as this patch adds a new core plugin.
  

• 4a2dd6af
  by Chandan Singh at 2018-08-15T18:58:58Z

  Add NEWS entry for Source Transform and pip source
  

• d4706096
  by Chandan Singh at 2018-08-15T20:21:01Z

  Merge branch 'chandan/sourcetransform' into 'master'
  
  Allow source plugins to access previous sources
  
  Closes #381
  
  See merge request BuildStream/buildstream!568

• c0431f15
  by Valentin David at 2018-08-16T07:43:07Z

  Fix ostree repository mirroring
  
  Ostree mirrors were not sharing the same local repository, so it was
  impossible the request refs from the right local repository when data
  was fetched from a mirror rather than upstream.
  
  Instead of having several repository with one remote each, we now
  have one repository with several remotes.
  
  This fixes #538.
  

• 04f83679
  by Tristan Van Berkom at 2018-08-16T08:54:27Z

  Merge branch 'valentindavid/fallback_mirror_ostree' into 'master'
  
  Fix ostree repository mirroring
  
  Closes #538
  
  See merge request BuildStream/buildstream!658

• 177c4264
  by William Salmon at 2018-08-16T09:52:38Z

  Trying to mitigate a mtime granularity braking the cache tests
  
  This patch mitigates the granularity of the mtimes used on the default
  gitlab runners, Allowing the test suite to pass on these runners.
  

• 01c4ac57
  by Tom Pollard at 2018-08-16T11:01:45Z

  Merge branch 'willsalmon/CacheExpiryTest' into 'master'
  
  Trying to mitigate a file system issue
  
  See merge request BuildStream/buildstream!595

• 0066d701
  by Tiago Gomes at 2018-08-16T12:33:01Z

  Use http instead of https for the gnu ftp mirror
  
  https seems broken on the Debian image:
  
  START   autotools/hello/42930621-fetch.499.log
  START   Fetching https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz
  FAILURE Fetching https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz
  FAILURE tar source at hello.bst [line 16 column 2]: Error mirroring https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz: <urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:720)>
  

• 898aa8a0
  by Tiago Gomes at 2018-08-16T13:43:06Z

  Merge branch 'tiagogomes/disable-https-gnu-repo' into 'master'
  
  Use http instead of https for the gnu ftp mirror
  
  See merge request BuildStream/buildstream!676

• b62c361c
  by Tiago Gomes at 2018-08-16T13:54:54Z

  cascache: use errno module
  
  os.errno does no longer work with Python 3.7
  
  Closes #577.
  

• 1e3e2a93
  by Tiago Gomes at 2018-08-16T15:00:12Z

  Merge branch 'tiagogomes/issue-577' into 'master'
  
  cascache: use errno module
  
  Closes #577
  
  See merge request BuildStream/buildstream!674

• ba992444
  by Tiago Gomes at 2018-08-16T15:01:39Z

  cascache: rename _fetch_directory and _checkout
  
  For symmetry with _create_tree.
  

• 5804cb1f
  by Tiago Gomes at 2018-08-16T15:01:39Z

  cascache: fix oversight
  

• 07a78d29
  by Tiago Gomes at 2018-08-16T15:01:39Z

  artifactcache: return bytes required to cache artifact
  
  Track number of bytes required to commit or pull an artifact and return
  this data on pull() and commit(). This takes deduplication into account.
  

• bb1b696b
  by Tiago Gomes at 2018-08-16T15:01:39Z

  element: use ArtifactCache.commit() return value
  
  The ArtifactCache.commit() return value is more accurate for what
  __artifact_size is being used for, as it takes the deduplication done by
  the CAS cache into account.
  

• f5afbeda
  by Tiago Gomes at 2018-08-16T15:01:39Z

  Revamp calculation of cache size code
  
  Before this commit, the cache size was being calculated every time an
  artifact was committed to or pulled to the artifact cache. This is a
  very IO demanding operation that doesn't scale as the cache gets bigger.
  
  Now that the commit() and pull() methods of the ArtifactCache return the
  number of bytes added, we can rework the code to calculate the size of
  The cache only once at startup and to dynamically update it afterwards.
  
  The ArtifactClean.clean() is also changed to return the number of bytes
  removed instead of the new size of the cache, so that we no longer need
  the ArtifactClean.clean._set_cache_size() method.
  
  The CacheSize job is no longer used and is removed.
  
  Fixes #573.
  

• 179bbe66
  by Tiago Gomes at 2018-08-16T15:01:39Z

  Move quota checking code into a separate method
  
  Instead of running `utils._get_dir_size(self.artifactdir)` to get the
  size of the cache, it would be better to reuse
  ArtifactCache.get_cache_size(), otherwise we would have to fully
  calculate the size of the artifact cache twice at startup.
  
  As the ArtifactCache is not yet instantiated when Context.load() runs,
  move most of the quota checking code into a separate function which can
  be called after the ArtifactCache is instantiated.
  

• 6e61474b
  by Tiago Gomes at 2018-08-16T15:01:39Z

  artifactcache: add some logging to cleanup operation
  

30 changed files:

• NEWS
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_artifactcache/casserver.py
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_loader/loader.py
• buildstream/_scheduler/jobs/__init__.py
• buildstream/_scheduler/jobs/cleanupjob.py
• buildstream/_scheduler/jobs/elementjob.py
• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/fetchqueue.py
• buildstream/_scheduler/queues/pullqueue.py
• buildstream/_scheduler/queues/queue.py
• buildstream/_scheduler/scheduler.py
• buildstream/_versions.py
• buildstream/element.py
• buildstream/plugins/sources/ostree.py
• + buildstream/plugins/sources/pip.py
• buildstream/source.py
• buildstream/storage/_casbaseddirectory.py
• doc/examples/autotools/project.conf
• doc/source/core_plugins.rst
• tests/artifactcache/expiry.py
• + tests/cachekey/project/sources/pip1.bst
• + tests/cachekey/project/sources/pip1.expected
• tests/cachekey/project/target.bst
• tests/cachekey/project/target.expected
• tests/frontend/mirror.py
• tests/frontend/push.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.