[Notes] [Git][BuildStream/buildstream][tiagogomes/issue-520] 4 commits:

Tiago Gomes pushed to branch tiagogomes/issue-520 at BuildStream / buildstream

Commits:

289de2ef

by Adam Jones at 2018-08-07T09:24:58Z

Add systemd service file examples to artifacts documentation

change code style to ini

e42325b9

by Javier Jardón at 2018-08-07T10:29:46Z

Merge branch 'adamjones/systemd-cas' into 'master'

Include systemd file examples for google-cas cache

Closes #419

See merge request BuildStream/buildstream!524

7e939e3c

by Tiago Gomes at 2018-08-07T10:36:20Z

elementjob: stop calculating cache size

Calculating the cache size on the child_process_data() is racy, as this
runs in the main process which does not obtain exclusive access to the
cache resource.

This calculation also seems unnecessary, as it will triggered when
BuildQueue.done() is called.

4fece773

by Tiago Gomes at 2018-08-07T10:36:20Z

artifactcache: turn method safe concurrently

Change ArtifactCache.get_approximate_cache_size() to be safe to use
concurrently without the caller having to have exclusive access to the
cache resource.

5 changed files:

buildstream/_artifactcache/artifactcache.py
buildstream/_scheduler/jobs/elementjob.py
buildstream/_scheduler/queues/buildqueue.py
buildstream/_scheduler/queues/queue.py
doc/source/install_artifacts.rst

Changes:

buildstream/_artifactcache/artifactcache.py

@@ -276,11 +276,6 @@ class ArtifactCache():
      #     (int) An approximation of the artifact cache size.
+     #
      def get_approximate_cache_size(self):
 -        # If we don't currently have an estimate, figure out the real
 -        # cache size.
 -        if self.estimated_size is None:
 -            self.estimated_size = self.calculate_cache_size()
+-
          return self.estimated_size
      ################################################
@@ -534,10 +529,8 @@ class ArtifactCache():
      # ArtifactCache.estimated_size.
+     #
      def _add_artifact_size(self, artifact_size):
 -        if not self.estimated_size:
 -            self.estimated_size = self.calculate_cache_size()
+-
 -        self.estimated_size += artifact_size
 +        if self.estimated_size:
 +            self.estimated_size += artifact_size
      # _set_cache_size()
+     #

buildstream/_scheduler/jobs/elementjob.py

@@ -109,13 +109,11 @@ class ElementJob(Job):
          data = {}
          workspace = self._element._get_workspace()
 -        artifact_size = self._element._get_artifact_size()
 -        cache_size = self._element._get_artifact_cache().calculate_cache_size()
+-
          if workspace is not None:
              data['workspace'] = workspace.to_dict()
++
 +        artifact_size = self._element._get_artifact_size()
          if artifact_size is not None:
              data['artifact_size'] = artifact_size
 -        data['cache_size'] = cache_size
          return data

buildstream/_scheduler/queues/buildqueue.py

+ #
 -#  Copyright (C) 2016 Codethink Limited
 +#  Copyright (C) 2018 Codethink Limited
+ #
  #  This program is free software; you can redistribute it and/or
  #  modify it under the terms of the GNU Lesser General Public
@@ -92,13 +92,16 @@ class BuildQueue(Queue):
              return
          artifact_size = job.child_data.get('artifact_size', False)
 +        if not artifact_size:
 +            return
 -        if artifact_size:
 -            cache = element._get_artifact_cache()
 -            cache._add_artifact_size(artifact_size)
 +        cache = element._get_artifact_cache()
 +        cache._add_artifact_size(artifact_size)
 -            if cache.get_approximate_cache_size() > self._scheduler.context.cache_quota:
 -                self._scheduler._check_cache_size_real()
 +        approx_size = cache.get_approximate_cache_size()
 +        if not approx_size or (approx_size >
 +                self._scheduler.context.cache_quota):
 +            self._scheduler._check_cache_size_real()
      def done(self, job, element, result, success):

buildstream/_scheduler/queues/queue.py

@@ -301,8 +301,6 @@ class Queue():
          # Update values that need to be synchronized in the main task
          # before calling any queue implementation
          self._update_workspaces(element, job)
 -        if job.child_data:
 -            element._get_artifact_cache().cache_size = job.child_data.get('cache_size')
          # Give the result of the job to the Queue implementor,
          # and determine if it should be considered as processed

doc/source/install_artifacts.rst

@@ -143,6 +143,50 @@ Instance with push and requiring client authentication:
      bst-artifact-server --port 11002 --server-key server.key --server-cert server.crt --client-certs authorized.crt --enable-push /home/artifacts/artifacts
 +Managing the cache with systemd
 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
++
 +It is better to run the cache as a systemd service, especially if it is running on a dedicated server, as this will allow systemd to manage the cache, incase the server ever encounters any issues.
++
 +Below are two examples of how to run the cache server as a systemd service, one is for pull only and the other is configured for push & pull.
++
 +.. code:: ini
++
 +   #
 +   # Pull
 +   #
 +   [Unit]
 +   Description=Buildstream Artifact pull server
 +   After=remote-fs.target network-online.target
++
 +   [Service]
 +   Environment="LC_ALL=C.UTF-8"
 +   ExecStart=/usr/local/bin/bst-artifact-server --port 11001 --server-key {{certs_path}}/privkey.pem --
 +   server-cert {{certs_path}}/fullchain.pem {{artifacts_path}}
 +   User=artifacts
++
 +   [Install]
 +   WantedBy=multi-user.target
++
++
 +   #
 +   # Pull/Push
 +   #
 +   [Unit]
 +   Description=Buildstream Artifact pull/push server
 +   After=remote-fs.target network-online.target
++
 +   [Service]
 +   Environment="LC_ALL=C.UTF-8"
 +   ExecStart=/usr/local/bin/bst-artifact-server --port 11002 --server-key {{certs_path}}/privkey.pem --
 +   server-cert {{certs_path}}/fullchain.pem --client-certs /home/artifacts/authorized.crt --enable-push /
 +   {{artifacts_path}}
 +   User=artifacts
++
 +   [Install]
 +   WantedBy=multi-user.target
++
 +Here we define when systemd should start the service, which is after the networking stack has been started, we then define how to run the cache with the desired configuration, under the artifacts user. The {{ }} are there to denote where you should change these files to point to your desired locations.
  User configuration
  ~~~~~~~~~~~~~~~~~~

[Notes] [Git][BuildStream/buildstream][tiagogomes/issue-520] 4 commits: Add systemd service file examples to artifacts documentation

Tiago Gomes pushed to branch tiagogomes/issue-520 at BuildStream / buildstream

Commits:

5 changed files:

Changes: