[Notes] [Git][BuildStream/buildstream][jmac/make_writable

Jim MacArthur pushed to branch jmac/make_writable_combination at BuildStream / buildstream

Commits:

8e64ccef

by James Ennis at 2018-11-07T11:06:11Z

using_config.rst: Update our instructions for how we declare caches in user config

09ab676d

by James Ennis at 2018-11-07T11:06:11Z

format_project.rst: Update instructions regarding how we declare remote caches in project.conf

f514124f

by James Ennis at 2018-11-07T11:06:11Z

using_configuring_artifact_server.rst: Remove examples and redirect to user config/project config

029ba17d

by richardmaw-codethink at 2018-11-07T11:32:20Z

Merge branch 'jennis/docs_patch' into 'master'

Update our instructions/examples regarding how we declare remote caches

See merge request BuildStream/buildstream!932

261e2cd3

by Jim MacArthur at 2018-11-07T12:02:43Z

element.py: Use _force_rmtree instead of custom code.

There are arguments for and against using onerror - see issue #735
for details.

4 changed files:

buildstream/element.py
doc/source/format_project.rst
doc/source/using_config.rst
doc/source/using_configuring_artifact_server.rst

Changes:

buildstream/element.py

@@ -1410,16 +1410,9 @@ class Element(Plugin):
              finally:
                  # Staging may produce directories with less than 'rwx' permissions
 -                # for the owner, which will break tempfile, so we need to use chmod
 -                # occasionally.
 -                def make_dir_writable(fn, path, excinfo):
 -                    os.chmod(os.path.dirname(path), 0o777)
 -                    if os.path.isdir(path):
 -                        os.rmdir(path)
 -                    else:
 -                        os.remove(path)
 -                shutil.rmtree(temp_staging_directory, onerror=make_dir_writable)
+-
 +                # for the owner, which breaks tempfile. _force_rmtree will deal
 +                # with these.
 +                utils._force_rmtree(temp_staging_directory)
          # Ensure deterministic mtime of sources at build time
          vdirectory.set_deterministic_mtime()
          # Ensure deterministic owners of sources at build time

doc/source/format_project.rst

@@ -190,19 +190,34 @@ for more detail.
  Artifact server
  ~~~~~~~~~~~~~~~
  If you have setup an :ref:`artifact server <artifacts>` for your
 -project then it is convenient to configure this in your ``project.conf``
 +project then it is convenient to configure the following in your ``project.conf``
  so that users need not have any additional configuration to communicate
  with an artifact share.
  .. code:: yaml
 +  #
 +  # Artifacts
 +  #
    artifacts:
 +    # A remote cache from which to download prebuilt artifacts
 +    - url: https://foo.com/artifacts:11001
 +      server.cert: server.crt
 +    # A remote cache from which to upload/download built/prebuilt artifacts
 +    - url: https://foo.com/artifacts:11002
 +      server-cert: server.crt
 +      client-cert: client.crt
 +      client-key: client.key
 -    # A url from which to download prebuilt artifacts
 -    url: https://foo.com/artifacts
 +.. note::
++
 +    You can also specify a list of different caches here; earlier entries in the
 +    list will have higher priority than later ones.
++
 +The use of ports are required to distinguish between pull only access and
 +push/pull access. For information regarding the server/client certificates
 +and keys, please see: :ref:`Key pair for the server <server_authentication>`.
 -You can also specify a list of caches here; earlier entries in the list
 -will have higher priority than later ones.
  Remote execution
  ~~~~~~~~~~~~~~~~

doc/source/using_config.rst

@@ -32,38 +32,75 @@ the supported configurations on a project wide basis are listed here.
  Artifact server
  ~~~~~~~~~~~~~~~
 -The project you build will often specify a :ref:`remote artifact cache
 -<artifacts>` already, but you may want to specify extra caches. There are two
 -ways to do this.  You can add one or more global caches:
 +Although project's often specify a :ref:`remote artifact cache <artifacts>` in
 +their ``project.conf``, you may also want to specify extra caches.
 -**Example**
 +Assuming that your host/server is reachable on the internet as ``artifacts.com``
 +(for example), there are two ways to declare remote caches in your user
 +configuration:
++
 +1. Adding global caches:
  .. code:: yaml
 +   #
 +   # Artifacts
 +   #
     artifacts:
 -     url: https://artifacts.com/artifacts
 +     # Add a cache to pull from
 +     - url: https://artifacts.com/artifacts:11001
 +       server-cert: server.crt
 +     # Add a cache to push/pull to/from
 +     - url: https://artifacts.com/artifacts:11002
 +       server-cert: server.crt
 +       client-cert: client.crt
 +       client-key: client.key
 +       push: true
 +     # Add another cache to pull from
 +     - url: https://anothercache.com/artifacts:8080
 +       server-cert: another_server.crt
++
 +.. note::
 -Caches listed there will be considered lower priority than those specified
 -by the project configuration.
 +    Caches declared here will be used by **all** BuildStream project's on the user's
 +    machine and are considered a lower priority than those specified in the project
 +    configuration.
 -You can also add project-specific caches:
 -**Example**
 +2. Specifying caches for a specific project within the user configuration:
  .. code:: yaml
     projects:
       project-name:
         artifacts:
 -         - url: https://artifacts.com/artifacts1
 -         - url: ssh://user artifacts com/artifacts2
 +         # Add a cache to pull from
 +         - url: https://artifacts.com/artifacts:11001
 +           server-cert: server.crt
 +         # Add a cache to push/pull to/from
 +         - url: https://artifacts.com/artifacts:11002
 +           server-cert: server.crt
 +           client-cert: client.crt
 +           client-key: client.key
             push: true
 +         # Add another cache to pull from
 +         - url: https://ourprojectcache.com/artifacts:8080
 +           server-cert: project_server.crt
++
++
 +.. note::
++
 +    Caches listed here will be considered a higher priority than those specified
 +    by the project. Furthermore, for a given list of URLs, earlier entries will
 +    have higher priority.
++
++
 +Notice that the use of different ports for the same server distinguishes between
 +pull only access and push/pull access. For information regarding this and the
 +server/client certificates and keys, please see:
 +:ref:`Key pair for the server <server_authentication>`.
 -Caches listed here will be considered higher priority than those specified
 -by the project.
 -If you give a list of URLs, earlier entries in the list will have higher
 -priority than later ones.
  Strict build plan
  ~~~~~~~~~~~~~~~~~

doc/source/using_configuring_artifact_server.rst

@@ -98,6 +98,8 @@ Command reference
     :prog: bst-artifact-server
 +.. _server_authentication:
++
  Key pair for the server
  ~~~~~~~~~~~~~~~~~~~~~~~
@@ -237,52 +239,12 @@ We can then check if the services are successfully running with:
  For more information on systemd services see:
  `Creating Systemd Service Files <https://www.devdungeon.com/content/creating-systemd-service-files>`_.
 -User configuration
 -~~~~~~~~~~~~~~~~~~
 -The user configuration for artifacts is documented with the rest
 -of the :ref:`user configuration documentation <user_config>`.
+-
 -Note that for self-signed certificates, the public key fields are mandatory.
+-
 -Assuming you have the same setup used in this document, and that your
 -host is reachable on the internet as ``artifacts.com`` (for example),
 -then a user can use the following user configuration:
+-
 -Pull-only:
+-
 -.. code:: yaml
+-
 -   #
 -   #    Artifacts
 -   #
 -   artifacts:
+-
 -     url: https://artifacts.com:11001
+-
 -     # Optional server certificate if not trusted by system root certificates
 -     server-cert: server.crt
 +Declaring remote artifact caches
 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 +Remote artifact caches can be declared within either:
 -Pull and push:
+-
 -.. code:: yaml
+-
 -   #
 -   #    Artifacts
 -   #
 -   artifacts:
+-
 -     url: https://artifacts.com:11002
+-
 -     # Optional server certificate if not trusted by system root certificates
 -     server-cert: server.crt
+-
 -     # Optional client key pair for authentication
 -     client-key: client.key
 -     client-cert: client.crt
+-
 -     push: true
+-
 -.. note::
 +1. The :ref:`project configuration <project_essentials_artifacts>`, or
 +2. The :ref:`user configuration <config_artifacts>`.
 -    Equivalent statements can be delcared in a project's configuration file
 -    (the ``project.conf``).
 +Please follow the above links to see examples showing how we declare remote
 +caches in both the project configuration and the user configuration, respectively.

[Notes] [Git][BuildStream/buildstream][jmac/make_writable_combination] 5 commits: using_config.rst: Update our instructions for how we declare caches in user config

Jim MacArthur pushed to branch jmac/make_writable_combination at BuildStream / buildstream

Commits:

4 changed files:

Changes: