[Notes] [Git][BuildStream/buildstream][Qinusty/553-backport-1.2] 7 commi

Qinusty pushed to branch Qinusty/553-backport-1.2 at BuildStream / buildstream

Commits:

16221c6b

by Josh Smith at 2018-08-06T09:49:13Z

_context.py: Cache size is now restricted to available disk space

This address issue #491.

When attempting to run buildstream with a configuration specifying
a cache quota larger than your available disk space, buildstream
will alert the user and exit.

Note:
This takes into consideration your current cache usage and
therefore restricts the overall size of your artifact cache folder.

dc88c4e2

by Tristan Van Berkom at 2018-08-06T10:34:32Z

Merge branch 'Qinusty/563-cache-quota-restriction' into 'bst-1.2'

Backport cache quota restriction to 1.2

See merge request BuildStream/buildstream!607

38cf132e

by Tristan Van Berkom at 2018-08-06T10:34:49Z

NEWS: Adding news entry for the include directive work.

0c57cdda

by Tristan Van Berkom at 2018-08-06T10:34:49Z

man: Updating man pages for the upcoming 1.1.5 release

84cba737

by Josh Smith at 2018-08-06T10:42:33Z

element.py: Make Element._get_brief_display_key() available to Internal API

This change will allow for more detailed info messages throughout
the CASCache.

23909122

by Josh Smith at 2018-08-06T10:42:33Z

cascache.py: Add push info detailing which remote is being pushed to

Work towards fixing issue #275

2a42b5f6

by Josh Smith at 2018-08-06T10:42:33Z

cascache.py: Add pull info detailing which remote is being pulled from

Work towards fixing issue #275

22 changed files:

NEWS
buildstream/_artifactcache/cascache.py
buildstream/_context.py
buildstream/element.py
buildstream/utils.py
man/bst-build.1
man/bst-checkout.1
man/bst-fetch.1
man/bst-help.1
man/bst-init.1
man/bst-pull.1
man/bst-push.1
man/bst-shell.1
man/bst-show.1
man/bst-source-bundle.1
man/bst-track.1
man/bst-workspace-close.1
man/bst-workspace-list.1
man/bst-workspace-open.1
man/bst-workspace-reset.1
man/bst-workspace.1
man/bst.1

Changes:

NEWS

@@ -11,6 +11,9 @@ buildstream 1.1.5
    o Added new `remote` source plugin for downloading file blobs
 +  o Add support for the new include '(@)' directive in project.conf and .bst files
++
++
  =================
  buildstream 1.1.4
  =================

buildstream/_artifactcache/cascache.py

@@ -221,6 +221,8 @@ class CASCache(ArtifactCache):
              try:
                  remote.init()
 +                element.info("Pulling {} <- {}".format(element._get_brief_display_key(), remote.spec.url))
++
                  request = buildstream_pb2.GetReferenceRequest()
                  request.key = ref
                  response = remote.ref_storage.GetReference(request)
@@ -263,6 +265,8 @@ class CASCache(ArtifactCache):
          for remote in push_remotes:
              remote.init()
 +            element.info("Pushing {} -> {}".format(element._get_brief_display_key(), remote.spec.url))
++
              try:
                  for ref in refs:
                      tree = self.resolve_ref(ref)
@@ -276,6 +280,8 @@ class CASCache(ArtifactCache):
                          if response.digest.hash == tree.hash and response.digest.size_bytes == tree.size_bytes:
                              # ref is already on the server with the same tree
 +                            element.info("Skipping {}, remote ({}) already has artifact cached".format(
 +                                element._get_brief_display_key(), remote.spec.url))
                              continue
                      except grpc.RpcError as e:

buildstream/_context.py

@@ -197,29 +197,55 @@ class Context():
                              "\nValid values are, for example: 800M 10G 1T 50%\n"
                              .format(str(e))) from e
 -        # If we are asked not to set a quota, we set it to the maximum
 -        # disk space available minus a headroom of 2GB, such that we
 -        # at least try to avoid raising Exceptions.
 +        # Headroom intended to give BuildStream a bit of leeway.
 +        # This acts as the minimum size of cache_quota and also
 +        # is taken from the user requested cache_quota.
+         #
 -        # Of course, we might still end up running out during a build
 -        # if we end up writing more than 2G, but hey, this stuff is
 -        # already really fuzzy.
 -        #
 -        if cache_quota is None:
 -            stat = os.statvfs(artifactdir_volume)
 -            # Again, the artifact directory may not yet have been
 -            # created
 -            if not os.path.exists(self.artifactdir):
 -                cache_size = 0
 -            else:
 -                cache_size = utils._get_dir_size(self.artifactdir)
 -            cache_quota = cache_size + stat.f_bsize * stat.f_bavail
+-
          if 'BST_TEST_SUITE' in os.environ:
              headroom = 0
          else:
              headroom = 2e9
 +        stat = os.statvfs(artifactdir_volume)
 +        available_space = (stat.f_bsize * stat.f_bavail)
++
 +        # Again, the artifact directory may not yet have been created yet
 +        #
 +        if not os.path.exists(self.artifactdir):
 +            cache_size = 0
 +        else:
 +            cache_size = utils._get_dir_size(self.artifactdir)
++
 +        # Ensure system has enough storage for the cache_quota
 +        #
 +        # If cache_quota is none, set it to the maximum it could possibly be.
 +        #
 +        # Also check that cache_quota is atleast as large as our headroom.
 +        #
 +        if cache_quota is None:  # Infinity, set to max system storage
 +            cache_quota = cache_size + available_space
 +        if cache_quota < headroom:  # Check minimum
 +            raise LoadError(LoadErrorReason.INVALID_DATA,
 +                            "Invalid cache quota ({}): ".format(utils._pretty_size(cache_quota)) +
 +                            "BuildStream requires a minimum cache quota of 2G.")
 +        elif cache_quota > cache_size + available_space:  # Check maximum
 +            raise LoadError(LoadErrorReason.INVALID_DATA,
 +                            ("Your system does not have enough available " +
 +                             "space to support the cache quota specified.\n" +
 +                             "You currently have:\n" +
 +                             "- {used} of cache in use at {local_cache_path}\n" +
 +                             "- {available} of available system storage").format(
 +                                 used=utils._pretty_size(cache_size),
 +                                 local_cache_path=self.artifactdir,
 +                                 available=utils._pretty_size(available_space)))
++
 +        # Place a slight headroom (2e9 (2GB) on the cache_quota) into
 +        # cache_quota to try and avoid exceptions.
 +        #
 +        # Of course, we might still end up running out during a build
 +        # if we end up writing more than 2G, but hey, this stuff is
 +        # already really fuzzy.
 +        #
          self.cache_quota = cache_quota - headroom
          self.cache_lower_threshold = self.cache_quota / 2

buildstream/element.py

@@ -618,7 +618,7 @@ class Element(Plugin):
          # Time to use the artifact, check once more that it's there
          self.__assert_cached()
 -        with self.timed_activity("Staging {}/{}".format(self.name, self.__get_brief_display_key())):
 +        with self.timed_activity("Staging {}/{}".format(self.name, self._get_brief_display_key())):
              # Get the extracted artifact
              artifact_base, _ = self.__extract()
              artifact = os.path.join(artifact_base, 'files')
@@ -1157,6 +1157,19 @@ class Element(Plugin):
          length = min(len(cache_key), context.log_key_length)
          return (cache_key, cache_key[0:length], dim_key)
 +    # _get_brief_display_key()
 +    #
 +    # Returns an abbreviated cache key for display purposes
 +    #
 +    # Returns:
 +    #    (str): An abbreviated hex digest cache key for this Element
 +    #
 +    # Question marks are returned if information for the cache key is missing.
 +    #
 +    def _get_brief_display_key(self):
 +        _, display_key, _ = self._get_display_key()
 +        return display_key
++
      # _preflight():
+     #
      # A wrapper for calling the abstract preflight() method on
@@ -1631,7 +1644,7 @@ class Element(Plugin):
              return False
          # Notify successfull download
 -        display_key = self.__get_brief_display_key()
 +        display_key = self._get_brief_display_key()
          self.info("Downloaded artifact {}".format(display_key))
          return True
@@ -1671,14 +1684,14 @@ class Element(Plugin):
              self.warn("Not pushing tainted artifact.")
              return False
 -        with self.timed_activity("Pushing artifact"):
 +        display_key = self._get_brief_display_key()
 +        with self.timed_activity("Pushing artifact {}".format(display_key)):
              # Push all keys used for local commit
              pushed = self.__artifacts.push(self, self.__get_cache_keys_for_commit())
              if not pushed:
                  return False
              # Notify successful upload
 -            display_key = self.__get_brief_display_key()
              self.info("Pushed artifact {}".format(display_key))
              return True
@@ -1954,19 +1967,6 @@ class Element(Plugin):
      def __can_build_incrementally(self):
          return bool(self._get_workspace())
 -    # __get_brief_display_key():
 -    #
 -    # Returns an abbreviated cache key for display purposes
 -    #
 -    # Returns:
 -    #    (str): An abbreviated hex digest cache key for this Element
 -    #
 -    # Question marks are returned if information for the cache key is missing.
 -    #
 -    def __get_brief_display_key(self):
 -        _, display_key, _ = self._get_display_key()
 -        return display_key
+-
      # __prepare():
+     #
      # Internal method for calling public abstract prepare() method.
@@ -1989,7 +1989,7 @@ class Element(Plugin):
      # Raises an error if the artifact is not cached.
+     #
      def __assert_cached(self):
 -        assert self._cached(), "{}: Missing artifact {}".format(self, self.__get_brief_display_key())
 +        assert self._cached(), "{}: Missing artifact {}".format(self, self._get_brief_display_key())
      # __get_tainted():
+     #

buildstream/utils.py

@@ -612,6 +612,27 @@ def _parse_size(size, volume):
      return int(num) * 1024**units.index(unit)
 +# _pretty_size()
 +#
 +# Converts a number of bytes into a string representation in KB, MB, GB, TB
 +# represented as K, M, G, T etc.
 +#
 +# Args:
 +#   size (int): The size to convert in bytes.
 +#   dec_places (int): The number of decimal places to output to.
 +#
 +# Returns:
 +#   (str): The string representation of the number of bytes in the largest
 +def _pretty_size(size, dec_places=0):
 +    psize = size
 +    unit = 'B'
 +    for unit in ('B', 'K', 'M', 'G', 'T'):
 +        if psize < 1024:
 +            break
 +        else:
 +            psize /= 1024
 +    return "{size:g}{unit}".format(size=round(psize, dec_places), unit=unit)
++
  # A sentinel to be used as a default argument for functions that need
  # to distinguish between a kwarg set to None and an unset kwarg.
  _sentinel = object()

man/bst-build.1

 -.TH "BST BUILD" "1" "18-Jul-2018" "" "bst build Manual"
 +.TH "BST BUILD" "1" "06-Aug-2018" "" "bst build Manual"
  .SH NAME
  bst\-build \- Build elements in a pipeline
  .SH SYNOPSIS

man/bst-checkout.1

 -.TH "BST CHECKOUT" "1" "18-Jul-2018" "" "bst checkout Manual"
 +.TH "BST CHECKOUT" "1" "06-Aug-2018" "" "bst checkout Manual"
  .SH NAME
  bst\-checkout \- Checkout a built artifact
  .SH SYNOPSIS
@@ -22,5 +22,4 @@ Whether to run integration commands
  Checkout hardlinks instead of copies (handle with care)
  .TP
  \fB\-\-tar\fP
 -Create a tarball from the artifact contents instead of a file tree. If
 -LOCATION is '-', the tarball will be dumped to the standard output.
 +Create a tarball from the artifact contents instead of a file tree. If LOCATION is '-', the tarball will be dumped to the standard output.
 \ No newline at end of file

man/bst-fetch.1

 -.TH "BST FETCH" "1" "18-Jul-2018" "" "bst fetch Manual"
 +.TH "BST FETCH" "1" "06-Aug-2018" "" "bst fetch Manual"
  .SH NAME
  bst\-fetch \- Fetch sources in a pipeline
  .SH SYNOPSIS

man/bst-help.1

 -.TH "BST HELP" "1" "18-Jul-2018" "" "bst help Manual"
 +.TH "BST HELP" "1" "06-Aug-2018" "" "bst help Manual"
  .SH NAME
  bst\-help \- Print usage information
  .SH SYNOPSIS

man/bst-init.1

 -.TH "BST INIT" "1" "18-Jul-2018" "" "bst init Manual"
 +.TH "BST INIT" "1" "06-Aug-2018" "" "bst init Manual"
  .SH NAME
  bst\-init \- Initialize a new BuildStream project
  .SH SYNOPSIS
@@ -18,7 +18,7 @@ interactive session.
  The project name to use
  .TP
  \fB\-\-format\-version\fP INTEGER
 -The required format version (default: 9)
 +The required format version (default: 12)
  .TP
  \fB\-\-element\-path\fP PATH
  The subdirectory to store elements in (default: elements)

man/bst-pull.1

 -.TH "BST PULL" "1" "18-Jul-2018" "" "bst pull Manual"
 +.TH "BST PULL" "1" "06-Aug-2018" "" "bst pull Manual"
  .SH NAME
  bst\-pull \- Pull a built artifact
  .SH SYNOPSIS

man/bst-push.1

 -.TH "BST PUSH" "1" "18-Jul-2018" "" "bst push Manual"
 +.TH "BST PUSH" "1" "06-Aug-2018" "" "bst push Manual"
  .SH NAME
  bst\-push \- Push a built artifact
  .SH SYNOPSIS

man/bst-shell.1

 -.TH "BST SHELL" "1" "18-Jul-2018" "" "bst shell Manual"
 +.TH "BST SHELL" "1" "06-Aug-2018" "" "bst shell Manual"
  .SH NAME
  bst\-shell \- Shell into an element's sandbox environment
  .SH SYNOPSIS

man/bst-show.1

 -.TH "BST SHOW" "1" "18-Jul-2018" "" "bst show Manual"
 +.TH "BST SHOW" "1" "06-Aug-2018" "" "bst show Manual"
  .SH NAME
  bst\-show \- Show elements in the pipeline
  .SH SYNOPSIS

man/bst-source-bundle.1

 -.TH "BST SOURCE-BUNDLE" "1" "18-Jul-2018" "" "bst source-bundle Manual"
 +.TH "BST SOURCE-BUNDLE" "1" "06-Aug-2018" "" "bst source-bundle Manual"
  .SH NAME
  bst\-source-bundle \- Produce a build bundle to be manually executed
  .SH SYNOPSIS

man/bst-track.1

 -.TH "BST TRACK" "1" "18-Jul-2018" "" "bst track Manual"
 +.TH "BST TRACK" "1" "06-Aug-2018" "" "bst track Manual"
  .SH NAME
  bst\-track \- Track new source references
  .SH SYNOPSIS

man/bst-workspace-close.1

 -.TH "BST WORKSPACE CLOSE" "1" "18-Jul-2018" "" "bst workspace close Manual"
 +.TH "BST WORKSPACE CLOSE" "1" "06-Aug-2018" "" "bst workspace close Manual"
  .SH NAME
  bst\-workspace\-close \- Close workspaces
  .SH SYNOPSIS

man/bst-workspace-list.1

 -.TH "BST WORKSPACE LIST" "1" "18-Jul-2018" "" "bst workspace list Manual"
 +.TH "BST WORKSPACE LIST" "1" "06-Aug-2018" "" "bst workspace list Manual"
  .SH NAME
  bst\-workspace\-list \- List open workspaces
  .SH SYNOPSIS

man/bst-workspace-open.1

 -.TH "BST WORKSPACE OPEN" "1" "18-Jul-2018" "" "bst workspace open Manual"
 +.TH "BST WORKSPACE OPEN" "1" "06-Aug-2018" "" "bst workspace open Manual"
  .SH NAME
  bst\-workspace\-open \- Open a new workspace
  .SH SYNOPSIS

man/bst-workspace-reset.1

 -.TH "BST WORKSPACE RESET" "1" "18-Jul-2018" "" "bst workspace reset Manual"
 +.TH "BST WORKSPACE RESET" "1" "06-Aug-2018" "" "bst workspace reset Manual"
  .SH NAME
  bst\-workspace\-reset \- Reset a workspace to its original state
  .SH SYNOPSIS

man/bst-workspace.1

 -.TH "BST WORKSPACE" "1" "18-Jul-2018" "" "bst workspace Manual"
 +.TH "BST WORKSPACE" "1" "06-Aug-2018" "" "bst workspace Manual"
  .SH NAME
  bst\-workspace \- Manipulate developer workspaces
  .SH SYNOPSIS
@@ -7,22 +7,22 @@ bst\-workspace \- Manipulate developer workspaces
  .SH DESCRIPTION
  Manipulate developer workspaces
  .SH COMMANDS
 -.PP
 -\fBopen\fP
 -  Open a new workspace
 -  See \fBbst workspace-open(1)\fP for full documentation on the \fBopen\fP command.
+-
  .PP
  \fBreset\fP
    Reset a workspace to its original state
    See \fBbst workspace-reset(1)\fP for full documentation on the \fBreset\fP command.
 +.PP
 +\fBclose\fP
 +  Close workspaces
 +  See \fBbst workspace-close(1)\fP for full documentation on the \fBclose\fP command.
++
  .PP
  \fBlist\fP
    List open workspaces
    See \fBbst workspace-list(1)\fP for full documentation on the \fBlist\fP command.
  .PP
 -\fBclose\fP
 -  Close workspaces
 -  See \fBbst workspace-close(1)\fP for full documentation on the \fBclose\fP command.
 +\fBopen\fP
 +  Open a new workspace
 +  See \fBbst workspace-open(1)\fP for full documentation on the \fBopen\fP command.

man/bst.1

 -.TH "BST" "1" "18-Jul-2018" "" "bst Manual"
 +.TH "BST" "1" "06-Aug-2018" "" "bst Manual"
  .SH NAME
  bst \- Build and manipulate BuildStream projects...
  .SH SYNOPSIS
@@ -61,31 +61,29 @@ Elements must be rebuilt when their dependencies have changed
  .TP
  \fB\-o,\fP \-\-option OPTION VALUE
  Specify a project option
 +.TP
 +\fB\-\-default\-mirror\fP TEXT
 +The mirror to fetch from first, before attempting other mirrors
  .SH COMMANDS
  .PP
 -\fBsource-bundle\fP
 -  Produce a build bundle to be manually executed
 -  See \fBbst-source-bundle(1)\fP for full documentation on the \fBsource-bundle\fP command.
+-
 -.PP
 -\fBhelp\fP
 -  Print usage information
 -  See \fBbst-help(1)\fP for full documentation on the \fBhelp\fP command.
 +\fBshow\fP
 +  Show elements in the pipeline
 +  See \fBbst-show(1)\fP for full documentation on the \fBshow\fP command.
  .PP
 -\fBinit\fP
 -  Initialize a new BuildStream project
 -  See \fBbst-init(1)\fP for full documentation on the \fBinit\fP command.
 +\fBpush\fP
 +  Push a built artifact
 +  See \fBbst-push(1)\fP for full documentation on the \fBpush\fP command.
  .PP
 -\fBtrack\fP
 -  Track new source references
 -  See \fBbst-track(1)\fP for full documentation on the \fBtrack\fP command.
 +\fBcheckout\fP
 +  Checkout a built artifact
 +  See \fBbst-checkout(1)\fP for full documentation on the \fBcheckout\fP command.
  .PP
 -\fBpull\fP
 -  Pull a built artifact
 -  See \fBbst-pull(1)\fP for full documentation on the \fBpull\fP command.
 +\fBfetch\fP
 +  Fetch sources in a pipeline
 +  See \fBbst-fetch(1)\fP for full documentation on the \fBfetch\fP command.
  .PP
  \fBworkspace\fP
@@ -98,9 +96,19 @@ Specify a project option
    See \fBbst-build(1)\fP for full documentation on the \fBbuild\fP command.
  .PP
 -\fBcheckout\fP
 -  Checkout a built artifact
 -  See \fBbst-checkout(1)\fP for full documentation on the \fBcheckout\fP command.
 +\fBtrack\fP
 +  Track new source references
 +  See \fBbst-track(1)\fP for full documentation on the \fBtrack\fP command.
++
 +.PP
 +\fBsource-bundle\fP
 +  Produce a build bundle to be manually executed
 +  See \fBbst-source-bundle(1)\fP for full documentation on the \fBsource-bundle\fP command.
++
 +.PP
 +\fBinit\fP
 +  Initialize a new BuildStream project
 +  See \fBbst-init(1)\fP for full documentation on the \fBinit\fP command.
  .PP
  \fBshell\fP
@@ -108,16 +116,11 @@ Specify a project option
    See \fBbst-shell(1)\fP for full documentation on the \fBshell\fP command.
  .PP
 -\fBfetch\fP
 -  Fetch sources in a pipeline
 -  See \fBbst-fetch(1)\fP for full documentation on the \fBfetch\fP command.
+-
 -.PP
 -\fBpush\fP
 -  Push a built artifact
 -  See \fBbst-push(1)\fP for full documentation on the \fBpush\fP command.
 +\fBpull\fP
 +  Pull a built artifact
 +  See \fBbst-pull(1)\fP for full documentation on the \fBpull\fP command.
  .PP
 -\fBshow\fP
 -  Show elements in the pipeline
 -  See \fBbst-show(1)\fP for full documentation on the \fBshow\fP command.
 +\fBhelp\fP
 +  Print usage information
 +  See \fBbst-help(1)\fP for full documentation on the \fBhelp\fP command.

[Notes] [Git][BuildStream/buildstream][Qinusty/553-backport-1.2] 7 commits: _context.py: Cache size is now restricted to available disk space

Qinusty pushed to branch Qinusty/553-backport-1.2 at BuildStream / buildstream

Commits:

22 changed files:

Changes: