[Notes] [Git][BuildStream/buildstream][tristan/fix-cache-exclusivity] 7

Tristan Van Berkom pushed to branch tristan/fix-cache-exclusivity at BuildStream / buildstream

Commits:

9b327eb6

by Ben Brewer at 2018-09-04T14:41:23Z

Improve documentation for artifact cache installation

Remove ambiguity about systemd service files being separate.
Add URL for more information about systemd service files.
Add note about public keys being mandatory for self-signed certs.
Make cert/key file naming consistent throughout document.

3e67e64a

by Javier Jardón at 2018-09-04T16:43:13Z

Merge branch 'benbrewer/install-artifacts-doc-improvements' into 'master'

Improve documentation for artifact cache installation

See merge request BuildStream/buildstream!777

3409609e

by Daniel Silverstone at 2018-09-04T16:55:51Z

jobs.py: Reduce FD leaks from queues and process objects

The garbage collector can take too long to get around to cleaning
up the Queue and Process instances in completed Job instances.  As
such, FDs tend to leak and in very large projects this can result
in running out of FDs before a build, fetch, track, or other process
can complete.  This patch reduces the chance of that by only creating
the queue when it's needed, and forcing the queue and process instances
to be deleted when the parent is finished with them.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

7b32e1ec

by Tristan Maat at 2018-09-04T17:20:55Z

Merge branch 'danielsilverstone-ct/maybe-reduce-fd-leaks' into 'master'

jobs.py: Reduce FD leaks from queues and process objects

See merge request BuildStream/buildstream!778

470305a7

by Tristan Van Berkom at 2018-09-05T12:02:02Z

_scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE

This is what the whole resource.py thing was created for, the
cleanup job must have exclusive access to the cache, while the pull
and build jobs which result in adding new artifacts, must only require
shared access.

This is a part of #623

0b51f1f0

by Tristan Van Berkom at 2018-09-05T12:02:02Z

_scheduler/resources.py: Set CACHE max resources to 0

This means there is no cap for shared resource requests.

Together with the previous commit, this causes the cleanup
job and the pull/build jobs to all require unlimited shared
access to the CACHE resource; while only the cleanup job
requires exclusive access to the resource.

This is a part of #623

ba89fa8f

by Tristan Van Berkom at 2018-09-05T12:02:02Z

_scheduler/scheduler.py: Make CacheSizeJob() not require the CACHE exclusively

This runs after every pull, and does not need the cache exclusively,
only the cleanup job requires the cache exclusively.

Without this, every time a cache_size job is queued, all pull and
build jobs need to complete before cache_size job can run exclusively,
which is not good.

6 changed files:

buildstream/_scheduler/jobs/job.py
buildstream/_scheduler/queues/buildqueue.py
buildstream/_scheduler/queues/pullqueue.py
buildstream/_scheduler/resources.py
buildstream/_scheduler/scheduler.py
doc/source/install_artifacts.rst

Changes:

buildstream/_scheduler/jobs/job.py

@@ -109,7 +109,7 @@ class Job():
          # Private members
+         #
          self._scheduler = scheduler            # The scheduler
 -        self._queue = multiprocessing.Queue()  # A message passing queue
 +        self._queue = None                     # A message passing queue
          self._process = None                   # The Process object
          self._watcher = None                   # Child process watcher
          self._listening = False                # Whether the parent is currently listening
@@ -130,6 +130,8 @@ class Job():
+     #
      def spawn(self):
 +        self._queue = multiprocessing.Queue()
++
          self._tries += 1
          self._parent_start_listening()
@@ -552,6 +554,9 @@ class Job():
          self.parent_complete(returncode == RC_OK, self._result)
          self._scheduler.job_completed(self, returncode == RC_OK)
 +        # Force the deletion of the queue and process objects to try and clean up FDs
 +        self._queue = self._process = None
++
      # _parent_process_envelope()
+     #
      # Processes a message Envelope deserialized form the message queue.

buildstream/_scheduler/queues/buildqueue.py

@@ -32,7 +32,7 @@ class BuildQueue(Queue):
      action_name = "Build"
      complete_name = "Built"
 -    resources = [ResourceType.PROCESS]
 +    resources = [ResourceType.PROCESS, ResourceType.CACHE]
      def __init__(self, *args, **kwargs):
          super().__init__(*args, **kwargs)

buildstream/_scheduler/queues/pullqueue.py

@@ -29,7 +29,7 @@ class PullQueue(Queue):
      action_name = "Pull"
      complete_name = "Pulled"
 -    resources = [ResourceType.DOWNLOAD]
 +    resources = [ResourceType.DOWNLOAD, ResourceType.CACHE]
      def process(self, element):
          # returns whether an artifact was downloaded or not

buildstream/_scheduler/resources.py

@@ -8,7 +8,7 @@ class ResourceType():
  class Resources():
      def __init__(self, num_builders, num_fetchers, num_pushers):
          self._max_resources = {
 -            ResourceType.CACHE: 1,
 +            ResourceType.CACHE: 0,
              ResourceType.DOWNLOAD: num_fetchers,
              ResourceType.PROCESS: num_builders,
              ResourceType.UPLOAD: num_pushers

buildstream/_scheduler/scheduler.py

@@ -332,7 +332,6 @@ class Scheduler():
          job = CacheSizeJob(self, 'cache_size', 'cache_size/cache_size',
                             resources=[ResourceType.CACHE,
                                        ResourceType.PROCESS],
 -                           exclusive_resources=[ResourceType.CACHE],
                             complete_cb=self._run_cleanup)
          self.schedule_jobs([job])

doc/source/install_artifacts.rst

@@ -161,13 +161,13 @@ Below are two examples of how to run the cache server as a systemd service, one
     [Service]
     Environment="LC_ALL=C.UTF-8"
 -   ExecStart=/usr/local/bin/bst-artifact-server --port 11001 --server-key {{certs_path}}/privkey.pem --
 -   server-cert {{certs_path}}/fullchain.pem {{artifacts_path}}
 +   ExecStart=/usr/local/bin/bst-artifact-server --port 11001 --server-key {{certs_path}}/server.key --server-cert {{certs_path}}/server.crt {{artifacts_path}}
     User=artifacts
     [Install]
     WantedBy=multi-user.target
 +.. code:: ini
+    #
     # Pull/Push
@@ -178,9 +178,7 @@ Below are two examples of how to run the cache server as a systemd service, one
     [Service]
     Environment="LC_ALL=C.UTF-8"
 -   ExecStart=/usr/local/bin/bst-artifact-server --port 11002 --server-key {{certs_path}}/privkey.pem --
 -   server-cert {{certs_path}}/fullchain.pem --client-certs /home/artifacts/authorized.crt --enable-push /
 -   {{artifacts_path}}
 +   ExecStart=/usr/local/bin/bst-artifact-server --port 11002 --server-key {{certs_path}}/server.key --server-cert {{certs_path}}/server.crt --client-certs {{certs_path}}/authorized.crt --enable-push {{artifacts_path}}
     User=artifacts
     [Install]
@@ -188,11 +186,16 @@ Below are two examples of how to run the cache server as a systemd service, one
  Here we define when systemd should start the service, which is after the networking stack has been started, we then define how to run the cache with the desired configuration, under the artifacts user. The {{ }} are there to denote where you should change these files to point to your desired locations.
 +For more information on systemd services see:
 +`Creating Systemd Service Files <https://www.devdungeon.com/content/creating-systemd-service-files>`_.
++
  User configuration
  ~~~~~~~~~~~~~~~~~~
  The user configuration for artifacts is documented with the rest
  of the :ref:`user configuration documentation <user_config>`.
 +Note that for self-signed certificates, the public key fields are mandatory.
++
  Assuming you have the same setup used in this document, and that your
  host is reachable on the internet as ``artifacts.com`` (for example),
  then a user can use the following user configuration:

[Notes] [Git][BuildStream/buildstream][tristan/fix-cache-exclusivity] 7 commits: Improve documentation for artifact cache installation

Tristan Van Berkom pushed to branch tristan/fix-cache-exclusivity at BuildStream / buildstream

Commits:

6 changed files:

Changes: