[Notes] [Git][BuildStream/buildstream][danielsilverstone-ct/maybe-reduce

Tristan Maat pushed to branch danielsilverstone-ct/maybe-reduce-fd-leaks at BuildStream / buildstream

Commits:

9b327eb6

by Ben Brewer at 2018-09-04T14:41:23Z

Improve documentation for artifact cache installation

Remove ambiguity about systemd service files being separate.
Add URL for more information about systemd service files.
Add note about public keys being mandatory for self-signed certs.
Make cert/key file naming consistent throughout document.

3e67e64a

by Javier Jardón at 2018-09-04T16:43:13Z

Merge branch 'benbrewer/install-artifacts-doc-improvements' into 'master'

Improve documentation for artifact cache installation

See merge request BuildStream/buildstream!777

3409609e

by Daniel Silverstone at 2018-09-04T16:55:51Z

jobs.py: Reduce FD leaks from queues and process objects

The garbage collector can take too long to get around to cleaning
up the Queue and Process instances in completed Job instances.  As
such, FDs tend to leak and in very large projects this can result
in running out of FDs before a build, fetch, track, or other process
can complete.  This patch reduces the chance of that by only creating
the queue when it's needed, and forcing the queue and process instances
to be deleted when the parent is finished with them.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

Changes:

buildstream/_scheduler/jobs/job.py

@@ -109,7 +109,7 @@ class Job():
          # Private members
+         #
          self._scheduler = scheduler            # The scheduler
 -        self._queue = multiprocessing.Queue()  # A message passing queue
 +        self._queue = None                     # A message passing queue
          self._process = None                   # The Process object
          self._watcher = None                   # Child process watcher
          self._listening = False                # Whether the parent is currently listening
@@ -130,6 +130,8 @@ class Job():
+     #
      def spawn(self):
 +        self._queue = multiprocessing.Queue()
++
          self._tries += 1
          self._parent_start_listening()
@@ -552,6 +554,9 @@ class Job():
          self.parent_complete(returncode == RC_OK, self._result)
          self._scheduler.job_completed(self, returncode == RC_OK)
 +        # Force the deletion of the queue and process objects to try and clean up FDs
 +        self._queue = self._process = None
++
      # _parent_process_envelope()
+     #
      # Processes a message Envelope deserialized form the message queue.

doc/source/install_artifacts.rst

@@ -161,13 +161,13 @@ Below are two examples of how to run the cache server as a systemd service, one
     [Service]
     Environment="LC_ALL=C.UTF-8"
 -   ExecStart=/usr/local/bin/bst-artifact-server --port 11001 --server-key {{certs_path}}/privkey.pem --
 -   server-cert {{certs_path}}/fullchain.pem {{artifacts_path}}
 +   ExecStart=/usr/local/bin/bst-artifact-server --port 11001 --server-key {{certs_path}}/server.key --server-cert {{certs_path}}/server.crt {{artifacts_path}}
     User=artifacts
     [Install]
     WantedBy=multi-user.target
 +.. code:: ini
+    #
     # Pull/Push
@@ -178,9 +178,7 @@ Below are two examples of how to run the cache server as a systemd service, one
     [Service]
     Environment="LC_ALL=C.UTF-8"
 -   ExecStart=/usr/local/bin/bst-artifact-server --port 11002 --server-key {{certs_path}}/privkey.pem --
 -   server-cert {{certs_path}}/fullchain.pem --client-certs /home/artifacts/authorized.crt --enable-push /
 -   {{artifacts_path}}
 +   ExecStart=/usr/local/bin/bst-artifact-server --port 11002 --server-key {{certs_path}}/server.key --server-cert {{certs_path}}/server.crt --client-certs {{certs_path}}/authorized.crt --enable-push {{artifacts_path}}
     User=artifacts
     [Install]
@@ -188,11 +186,16 @@ Below are two examples of how to run the cache server as a systemd service, one
  Here we define when systemd should start the service, which is after the networking stack has been started, we then define how to run the cache with the desired configuration, under the artifacts user. The {{ }} are there to denote where you should change these files to point to your desired locations.
 +For more information on systemd services see:
 +`Creating Systemd Service Files <https://www.devdungeon.com/content/creating-systemd-service-files>`_.
++
  User configuration
  ~~~~~~~~~~~~~~~~~~
  The user configuration for artifacts is documented with the rest
  of the :ref:`user configuration documentation <user_config>`.
 +Note that for self-signed certificates, the public key fields are mandatory.
++
  Assuming you have the same setup used in this document, and that your
  host is reachable on the internet as ``artifacts.com`` (for example),
  then a user can use the following user configuration:

[Notes] [Git][BuildStream/buildstream][danielsilverstone-ct/maybe-reduce-fd-leaks] 3 commits: Improve documentation for artifact cache installation

Tristan Maat pushed to branch danielsilverstone-ct/maybe-reduce-fd-leaks at BuildStream / buildstream

Commits:

2 changed files:

Changes: