[Notes] [Git][BuildStream/buildstream][Qinusty/491] 7 commits: cascache.

Qinusty pushed to branch Qinusty/491 at BuildStream / buildstream

Commits:

fb222ba1

by Josh Smith at 2018-07-27T08:50:45Z

cascache.py: Raise ArtifactError on grpc error

This aims to fix issue #502.

48916b8a

by Tristan Maat at 2018-07-27T10:22:41Z

Merge branch 'Qinusty/502' into 'master'

cascache.py: Raise ArtifactError on grpc error

Closes #502

See merge request BuildStream/buildstream!576

23e080b9
by James Ennis at 2018-07-27T11:13:50Z
```
README.md: Minor corrections
```

196cfffc

by James Ennis at 2018-07-27T11:13:50Z

README.md: Direct users towards the tutorial before existing projects:

  This patch also removes displaying the projects that live within
  the buildstream-examples repository, as we wish to deprecate this
  repo and migrate the two examples.
  - #438 and #439

  In addition to this, both examples do not 'build' in their current
  state.

2b93574e

by James Ennis at 2018-07-27T11:13:50Z

main_install.rst: Add note to refer Windows and macOS users to docker

6ea97b17

by Javier Jardón at 2018-07-27T11:54:20Z

Merge branch 'jennis/doc_improvements' into 'master'

Refer readers to our tutorial before referring them to existing bst projects

See merge request BuildStream/buildstream!578

a555522e

by Josh Smith at 2018-07-27T13:00:07Z

_context.py: Cache size is now restricted to available disk space

This address issue #491.

When attempting to run buildstream with a configuration specifying
a cache quota larger than your available disk space, buildstream
will alert the user and exit.

Note:
This takes into consideration your current cache usage and
therefore restricts the overall size of your artifact cache folder.

Changes:

README.rst

@@ -25,7 +25,7 @@ BuildStream offers the following advantages:
  * **Declarative build instructions/definitions**
 -  BuildStream provides a a flexible and extensible framework for the modelling
 +  BuildStream provides a flexible and extensible framework for the modelling
    of software build pipelines in a declarative YAML format, which allows you to
    manipulate filesystem data in a controlled, reproducible sandboxed environment.
@@ -61,25 +61,29 @@ How does BuildStream work?
  ==========================
  BuildStream operates on a set of YAML files (.bst files), as follows:
 -* loads the YAML files which describe the target(s) and all dependencies
 -* evaluates the version information and build instructions to calculate a build
 +* Loads the YAML files which describe the target(s) and all dependencies.
 +* Evaluates the version information and build instructions to calculate a build
    graph for the target(s) and all dependencies and unique cache-keys for each
 -  element
 -* retrieves elements from cache if they are already built, or builds them in a
 -  sandboxed environment using the instructions declared in the .bst files
 -* transforms/configures and/or deploys the resulting target(s) based on the
 +  element.
 +* Retrieves previously built elements (artifacts) from a local/remote cache, or
 +  builds the elements in a sandboxed environment using the instructions declared
 +  in the .bst files.
 +* Transforms/configures and/or deploys the resulting target(s) based on the
    instructions declared in the .bst files.
  How can I get started?
  ======================
 -The easiest way to get started is to explore some existing .bst files, for example:
 +To start using BuildStream, first,
 +`install <https://buildstream.gitlab.io/buildstream/main_install.html>`_
 +BuildStream onto your machine and then follow our
 +`tutorial <https://buildstream.gitlab.io/buildstream/using_tutorial.html>`_.
++
 +We also recommend exploring some existing BuildStream projects:
  * https://gitlab.gnome.org/GNOME/gnome-build-meta/
  * https://gitlab.com/freedesktop-sdk/freedesktop-sdk
  * https://gitlab.com/baserock/definitions
 -* https://gitlab.com/BuildStream/buildstream-examples/tree/master/build-x86image
 -* https://gitlab.com/BuildStream/buildstream-examples/tree/master/netsurf-flatpak
  If you have any questions please ask on our `#buildstream <irc://irc.gnome.org/buildstream>`_ channel in `irc.gnome.org <irc://irc.gnome.org>`_

buildstream/_artifactcache/cascache.py

@@ -240,7 +240,8 @@ class CASCache(ArtifactCache):
              except grpc.RpcError as e:
                  if e.code() != grpc.StatusCode.NOT_FOUND:
 -                    raise
 +                    raise ArtifactError("Failed to pull artifact {}: {}".format(
 +                        element._get_brief_display_key(), e)) from e
          return False
@@ -285,6 +286,7 @@ class CASCache(ArtifactCache):
                      except grpc.RpcError as e:
                          if e.code() != grpc.StatusCode.NOT_FOUND:
 +                            # Intentionally re-raise RpcError for outer except block.
                              raise
                      missing_blobs = {}

buildstream/_context.py

@@ -197,29 +197,55 @@ class Context():
                              "\nValid values are, for example: 800M 10G 1T 50%\n"
                              .format(str(e))) from e
 -        # If we are asked not to set a quota, we set it to the maximum
 -        # disk space available minus a headroom of 2GB, such that we
 -        # at least try to avoid raising Exceptions.
 +        # Headroom intended to give BuildStream a bit of leeway.
 +        # This acts as the minimum size of cache_quota and also
 +        # is taken from the user requested cache_quota.
+         #
 -        # Of course, we might still end up running out during a build
 -        # if we end up writing more than 2G, but hey, this stuff is
 -        # already really fuzzy.
 -        #
 -        if cache_quota is None:
 -            stat = os.statvfs(artifactdir_volume)
 -            # Again, the artifact directory may not yet have been
 -            # created
 -            if not os.path.exists(self.artifactdir):
 -                cache_size = 0
 -            else:
 -                cache_size = utils._get_dir_size(self.artifactdir)
 -            cache_quota = cache_size + stat.f_bsize * stat.f_bavail
+-
          if 'BST_TEST_SUITE' in os.environ:
              headroom = 0
          else:
              headroom = 2e9
 +        stat = os.statvfs(artifactdir_volume)
 +        available_space = (stat.f_bsize * stat.f_bavail)
++
 +        # Again, the artifact directory may not yet have been created yet
 +        #
 +        if not os.path.exists(self.artifactdir):
 +            cache_size = 0
 +        else:
 +            cache_size = utils._get_dir_size(self.artifactdir)
++
 +        # Ensure system has enough storage for the cache_quota
 +        #
 +        # If cache_quota is none, set it to the maximum it could possibly be.
 +        #
 +        # Also check that cache_quota is atleast as large as our headroom.
 +        #
 +        if cache_quota is None:  # Infinity, set to max system storage
 +            cache_quota = cache_size + available_space
 +        if cache_quota < headroom:  # Check minimum
 +            raise LoadError(LoadErrorReason.INVALID_DATA,
 +                            "Invalid cache quota ({}): ".format(utils._pretty_size(cache_quota)) +
 +                            "BuildStream requires a minimum cache quota of 2G.")
 +        elif cache_quota > cache_size + available_space:  # Check maximum
 +            raise LoadError(LoadErrorReason.INVALID_DATA,
 +                            ("Your system does not have enough available " +
 +                             "space to support the cache quota specified.\n" +
 +                             "You currently have:\n" +
 +                             "- {used} of cache in use at {local_cache_path}\n" +
 +                             "- {available} of available system storage").format(
 +                                 used=utils._pretty_size(cache_size),
 +                                 local_cache_path=self.artifactdir,
 +                                 available=utils._pretty_size(available_space)))
++
 +        # Place a slight headroom (2e9 (2GB) on the cache_quota) into
 +        # cache_quota to try and avoid exceptions.
 +        #
 +        # Of course, we might still end up running out during a build
 +        # if we end up writing more than 2G, but hey, this stuff is
 +        # already really fuzzy.
 +        #
          self.cache_quota = cache_quota - headroom
          self.cache_lower_threshold = self.cache_quota / 2

buildstream/utils.py

@@ -608,6 +608,27 @@ def _parse_size(size, volume):
      return int(num) * 1024**units.index(unit)
 +# _pretty_size()
 +#
 +# Converts a number of bytes into a string representation in KB, MB, GB, TB
 +# represented as K, M, G, T etc.
 +#
 +# Args:
 +#   size (int): The size to convert in bytes.
 +#   dec_places (int): The number of decimal places to output to.
 +#
 +# Returns:
 +#   (str): The string representation of the number of bytes in the largest
 +def _pretty_size(size, dec_places=0):
 +    psize = size
 +    unit = 'B'
 +    for unit in ('B', 'K', 'M', 'G', 'T'):
 +        if psize < 1024:
 +            break
 +        else:
 +            psize /= 1024
 +    return "{size:g}{unit}".format(size=round(psize, dec_places), unit=unit)
++
  # A sentinel to be used as a default argument for functions that need
  # to distinguish between a kwarg set to None and an unset kwarg.
  _sentinel = object()

doc/source/main_install.rst

  Install
  =======
 -This section covers how to install BuildStream onto your machine, how to run BuildStream inside a docker image and also how to configure an artifact server.
 +This section covers how to install BuildStream onto your machine, how to run
 +BuildStream inside a docker image and also how to configure an artifact server.
 +.. note::
++
 +   BuildStream is not currently supported natively on macOS and Windows. Windows
 +   and macOS users should refer to :ref:`docker`.
  .. toctree::
     :maxdepth: 2

[Notes] [Git][BuildStream/buildstream][Qinusty/491] 7 commits: cascache.py: Raise ArtifactError on grpc error

Qinusty pushed to branch Qinusty/491 at BuildStream / buildstream

Commits:

5 changed files:

Changes: