[Notes] [Git][BuildStream/buildstream][jennis/docs

richardmaw-codethink pushed to branch jennis/docs_patch at BuildStream / buildstream

Commits:

9a045080

by Josh Smith at 2018-11-06T13:23:19Z

utils: Fix _pretty_size() for sizes > 1024T

d3a07e6b
by Josh Smith at 2018-11-06T13:23:19Z
```
Add regression test for _pretty_size
```

0c09fb9c

by richardmaw-codethink at 2018-11-06T14:12:20Z

Merge branch 'Qinusty/unit-test-utils' into 'master'

Fix issue with _pretty_size with large numbers of bytes

See merge request BuildStream/buildstream!799

b4eec489

by Jim MacArthur at 2018-11-06T16:44:14Z

_sandboxremote.py: Use the standard SandboxError.

Replaces the custom one which was erroneously added during development.
Fixes #746.

1f7acf74

by Jim MacArthur at 2018-11-06T17:16:31Z

Merge branch 'jmac/remote-sandbox-errors' into 'master'

_sandboxremote.py: Use the standard SandboxError

Closes #746

See merge request BuildStream/buildstream!931

8e64ccef

by James Ennis at 2018-11-07T11:06:11Z

using_config.rst: Update our instructions for how we declare caches in user config

09ab676d

by James Ennis at 2018-11-07T11:06:11Z

format_project.rst: Update instructions regarding how we declare remote caches in project.conf

f514124f

by James Ennis at 2018-11-07T11:06:11Z

using_configuring_artifact_server.rst: Remove examples and redirect to user config/project config

7 changed files:

buildstream/sandbox/_sandboxremote.py
buildstream/utils.py
doc/source/format_project.rst
doc/source/using_config.rst
doc/source/using_configuring_artifact_server.rst
+ tests/testutils/mock_os.py
+ tests/utils/misc.py

Changes:

buildstream/sandbox/_sandboxremote.py

@@ -28,10 +28,7 @@ from ..storage._filebaseddirectory import FileBasedDirectory
  from ..storage._casbaseddirectory import CasBasedDirectory
  from .._protos.build.bazel.remote.execution.v2 import remote_execution_pb2, remote_execution_pb2_grpc
  from .._protos.google.rpc import code_pb2
+-
+-
 -class SandboxError(Exception):
 -    pass
 +from .._exceptions import SandboxError
  # SandboxRemote()

buildstream/utils.py

@@ -634,7 +634,7 @@ def _parse_size(size, volume):
  # _pretty_size()
+ #
 -# Converts a number of bytes into a string representation in KB, MB, GB, TB
 +# Converts a number of bytes into a string representation in KiB, MiB, GiB, TiB
  # represented as K, M, G, T etc.
+ #
  # Args:
@@ -646,10 +646,11 @@ def _parse_size(size, volume):
  def _pretty_size(size, dec_places=0):
      psize = size
      unit = 'B'
 -    for unit in ('B', 'K', 'M', 'G', 'T'):
 +    units = ('B', 'K', 'M', 'G', 'T')
 +    for unit in units:
          if psize < 1024:
              break
 -        else:
 +        elif unit != units[-1]:
              psize /= 1024
      return "{size:g}{unit}".format(size=round(psize, dec_places), unit=unit)

doc/source/format_project.rst

@@ -190,19 +190,34 @@ for more detail.
  Artifact server
  ~~~~~~~~~~~~~~~
  If you have setup an :ref:`artifact server <artifacts>` for your
 -project then it is convenient to configure this in your ``project.conf``
 +project then it is convenient to configure the following in your ``project.conf``
  so that users need not have any additional configuration to communicate
  with an artifact share.
  .. code:: yaml
 +  #
 +  # Artifacts
 +  #
    artifacts:
 +    # A remote cache from which to download prebuilt artifacts
 +    - url: https://foo.com/artifacts:11001
 +      server.cert: server.crt
 +    # A remote cache from which to upload/download built/prebuilt artifacts
 +    - url: https://foo.com/artifacts:11002
 +      server-cert: server.crt
 +      client-cert: client.crt
 +      client-key: client.key
 -    # A url from which to download prebuilt artifacts
 -    url: https://foo.com/artifacts
 +.. note::
++
 +    You can also specify a list of different caches here; earlier entries in the
 +    list will have higher priority than later ones.
++
 +The use of ports are required to distinguish between pull only access and
 +push/pull access. For information regarding the server/client certificates
 +and keys, please see: :ref:`Key pair for the server <server_authentication>`.
 -You can also specify a list of caches here; earlier entries in the list
 -will have higher priority than later ones.
  Remote execution
  ~~~~~~~~~~~~~~~~

doc/source/using_config.rst

@@ -32,38 +32,75 @@ the supported configurations on a project wide basis are listed here.
  Artifact server
  ~~~~~~~~~~~~~~~
 -The project you build will often specify a :ref:`remote artifact cache
 -<artifacts>` already, but you may want to specify extra caches. There are two
 -ways to do this.  You can add one or more global caches:
 +Although project's often specify a :ref:`remote artifact cache <artifacts>` in
 +their ``project.conf``, you may also want to specify extra caches.
 -**Example**
 +Assuming that your host/server is reachable on the internet as ``artifacts.com``
 +(for example), there are two ways to declare remote caches in your user
 +configuration:
++
 +1. Adding global caches:
  .. code:: yaml
 +   #
 +   # Artifacts
 +   #
     artifacts:
 -     url: https://artifacts.com/artifacts
 +     # Add a cache to pull from
 +     - url: https://artifacts.com/artifacts:11001
 +       server-cert: server.crt
 +     # Add a cache to push/pull to/from
 +     - url: https://artifacts.com/artifacts:11002
 +       server-cert: server.crt
 +       client-cert: client.crt
 +       client-key: client.key
 +       push: true
 +     # Add another cache to pull from
 +     - url: https://anothercache.com/artifacts:8080
 +       server-cert: another_server.crt
++
 +.. note::
 -Caches listed there will be considered lower priority than those specified
 -by the project configuration.
 +    Caches declared here will be used by **all** BuildStream project's on the user's
 +    machine and are considered a lower priority than those specified in the project
 +    configuration.
 -You can also add project-specific caches:
 -**Example**
 +2. Specifying caches for a specific project within the user configuration:
  .. code:: yaml
     projects:
       project-name:
         artifacts:
 -         - url: https://artifacts.com/artifacts1
 -         - url: ssh://user artifacts com/artifacts2
 +         # Add a cache to pull from
 +         - url: https://artifacts.com/artifacts:11001
 +           server-cert: server.crt
 +         # Add a cache to push/pull to/from
 +         - url: https://artifacts.com/artifacts:11002
 +           server-cert: server.crt
 +           client-cert: client.crt
 +           client-key: client.key
             push: true
 +         # Add another cache to pull from
 +         - url: https://ourprojectcache.com/artifacts:8080
 +           server-cert: project_server.crt
++
++
 +.. note::
++
 +    Caches listed here will be considered a higher priority than those specified
 +    by the project. Furthermore, for a given list of URLs, earlier entries will
 +    have higher priority.
++
++
 +Notice that the use of different ports for the same server distinguishes between
 +pull only access and push/pull access. For information regarding this and the
 +server/client certificates and keys, please see:
 +:ref:`Key pair for the server <server_authentication>`.
 -Caches listed here will be considered higher priority than those specified
 -by the project.
 -If you give a list of URLs, earlier entries in the list will have higher
 -priority than later ones.
  Strict build plan
  ~~~~~~~~~~~~~~~~~

doc/source/using_configuring_artifact_server.rst

@@ -98,6 +98,8 @@ Command reference
     :prog: bst-artifact-server
 +.. _server_authentication:
++
  Key pair for the server
  ~~~~~~~~~~~~~~~~~~~~~~~
@@ -237,52 +239,12 @@ We can then check if the services are successfully running with:
  For more information on systemd services see:
  `Creating Systemd Service Files <https://www.devdungeon.com/content/creating-systemd-service-files>`_.
 -User configuration
 -~~~~~~~~~~~~~~~~~~
 -The user configuration for artifacts is documented with the rest
 -of the :ref:`user configuration documentation <user_config>`.
+-
 -Note that for self-signed certificates, the public key fields are mandatory.
+-
 -Assuming you have the same setup used in this document, and that your
 -host is reachable on the internet as ``artifacts.com`` (for example),
 -then a user can use the following user configuration:
+-
 -Pull-only:
+-
 -.. code:: yaml
+-
 -   #
 -   #    Artifacts
 -   #
 -   artifacts:
+-
 -     url: https://artifacts.com:11001
+-
 -     # Optional server certificate if not trusted by system root certificates
 -     server-cert: server.crt
 +Declaring remote artifact caches
 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 +Remote artifact caches can be declared within either:
 -Pull and push:
+-
 -.. code:: yaml
+-
 -   #
 -   #    Artifacts
 -   #
 -   artifacts:
+-
 -     url: https://artifacts.com:11002
+-
 -     # Optional server certificate if not trusted by system root certificates
 -     server-cert: server.crt
+-
 -     # Optional client key pair for authentication
 -     client-key: client.key
 -     client-cert: client.crt
+-
 -     push: true
+-
 -.. note::
 +1. The :ref:`project configuration <project_essentials_artifacts>`, or
 +2. The :ref:`user configuration <config_artifacts>`.
 -    Equivalent statements can be delcared in a project's configuration file
 -    (the ``project.conf``).
 +Please follow the above links to see examples showing how we declare remote
 +caches in both the project configuration and the user configuration, respectively.

tests/testutils/mock_os.py

 +from contextlib import contextmanager
 +import os
++
++
 +# MockAttributeResult
 +#
 +# A class to take a dictionary of kwargs and make them accessible via
 +# attributes of the object.
 +#
 +class MockAttributeResult(dict):
 +    __getattr__ = dict.get
++
++
 +# mock_statvfs():
 +#
 +# Gets a function which mocks statvfs and returns a statvfs result with the kwargs accessible.
 +#
 +# Returns:
 +#    func(path) -> object: object will have all the kwargs accessible via object.kwarg
 +#
 +# Example:
 +#    statvfs = mock_statvfs(f_blocks=10)
 +#    result = statvfs("regardless/of/path")
 +#    assert result.f_blocks == 10 # True
 +def mock_statvfs(**kwargs):
 +    def statvfs(path):
 +        return MockAttributeResult(kwargs)
 +    return statvfs
++
++
 +# monkey_patch()
 +#
 +# with monkey_patch("statvfs", custom_statvfs):
 +#    assert os.statvfs == custom_statvfs # True
 +# assert os.statvfs == custom_statvfs # False
 +#
 +@contextmanager
 +def monkey_patch(to_patch, patched_func):
 +    orig = getattr(os, to_patch)
 +    setattr(os, to_patch, patched_func)
 +    try:
 +        yield
 +    finally:
 +        setattr(os, to_patch, orig)

tests/utils/misc.py

 +from buildstream import _yaml
 +from ..testutils import mock_os
 +from ..testutils.runcli import cli
++
 +import os
 +import pytest
++
++
 +KiB = 1024
 +MiB = (KiB * 1024)
 +GiB = (MiB * 1024)
 +TiB = (GiB * 1024)
++
++
 +def test_parse_size_over_1024T(cli, tmpdir):
 +    BLOCK_SIZE = 4096
 +    cli.configure({
 +        'cache': {
 +            'quota': 2048 * TiB
 +        }
 +    })
 +    project = tmpdir.join("main")
 +    os.makedirs(str(project))
 +    _yaml.dump({'name': 'main'}, str(project.join("project.conf")))
++
 +    bavail = (1025 * TiB) / BLOCK_SIZE
 +    patched_statvfs = mock_os.mock_statvfs(f_bavail=bavail, f_bsize=BLOCK_SIZE)
 +    with mock_os.monkey_patch("statvfs", patched_statvfs):
 +        result = cli.run(project, args=["build", "file.bst"])
 +        assert "1025T of available system storage" in result.stderr

[Notes] [Git][BuildStream/buildstream][jennis/docs_patch] 8 commits: utils: Fix _pretty_size() for sizes > 1024T

richardmaw-codethink pushed to branch jennis/docs_patch at BuildStream / buildstream

Commits:

7 changed files:

Changes: