[Notes] [Git][BuildStream/buildstream][willsalmon/trackWarning] 8 commit

Will Salmon pushed to branch willsalmon/trackWarning at BuildStream / buildstream

Commits:

01ff0177

by Javier Jardón at 2018-08-07T07:29:30Z

source/install_linux_distro.rst: Make clearer ArchLinux packages available

9b469fe6

by Tristan Van Berkom at 2018-08-07T09:12:46Z

Merge branch 'jjardon/doc_arch_releases' into 'master'

source/install_linux_distro.rst: Make clearer ArchLinux packages available

See merge request BuildStream/buildstream!613

289de2ef

by Adam Jones at 2018-08-07T09:24:58Z

Add systemd service file examples to artifacts documentation

change code style to ini

e42325b9

by Javier Jardón at 2018-08-07T10:29:46Z

Merge branch 'adamjones/systemd-cas' into 'master'

Include systemd file examples for google-cas cache

Closes #419

See merge request BuildStream/buildstream!524

6d349610

by Javier Jardón at 2018-08-07T10:53:23Z

source/install_linux_distro.rst: Fix formatting

493d19d2

by James Ennis at 2018-08-07T11:08:36Z

install_linux_distro.rst: Change install instructions for python-arpy package on Arch

  - python-arpy is unobtainable via pacman
  - Also a minor change to how we display install instructions
  from Fedora's dnf package manager

fc9869e2

by James Ennis at 2018-08-07T11:56:21Z

Merge branch 'jennis/change_arch_install_instructions' into 'master'

Change install instructions for python-arpy package on Arch

See merge request BuildStream/buildstream!614

42aa3999

by William Salmon at 2018-08-07T13:41:02Z

Add warning to git track if track and ref are not present

This is to address https://gitlab.com/BuildStream/buildstream/issues/471 that
documented unhelpfull behavour when tracking git sources.

Changes:

buildstream/_pipeline.py

@@ -358,10 +358,24 @@ class Pipeline():
                      inconsistent.append(element)
          if inconsistent:
 -            detail = "Exact versions are missing for the following elements\n" + \
 -                     "Try tracking these elements first with `bst track`\n\n"
 +            detail = "Exact versions are missing for the following elements:\n\n"
++
 +            missingTrack = 0
              for element in inconsistent:
 -                detail += "  " + element._get_full_name() + "\n"
 +                detail += "  " + element._get_full_name()
 +                for source in element.sources():
 +                    if not source._get_consistency() and not source.get_ref():
 +                        if hasattr(source, 'tracking') and source.tracking is None:
 +                            detail += ": Source {} is missing ref and track. ".format(source._get_full_name()) + \
 +                                      "Please specify a ref or branch/tag to track."
 +                            missingTrack = 1
++
 +                detail += "\n"
++
 +            if missingTrack:
 +                detail += "\nThen track these elements with `bst track`\n"
 +            else:
 +                detail += "\nTry tracking these elements first with `bst track`\n"
              raise PipelineError("Inconsistent pipeline", detail=detail, reason="inconsistent-pipeline")
      #############################################################

buildstream/plugins/sources/git.py

@@ -363,6 +363,12 @@ class GitSource(Source):
          # If self.tracking is not specified it's not an error, just silently return
          if not self.tracking:
 +            # Is there a better way to check if a ref is given.
 +            if self.mirror.ref is None:
 +                detail = 'Without a tracking branch ref can not be updated. Please ' + \
 +                         'provide a ref or a track.'
 +                raise SourceError("{}: No track or ref".format(self),
 +                                  detail=detail, reason="track-attempt-no-track")
              return None
          with self.timed_activity("Tracking {} from {}"

doc/source/install_artifacts.rst

@@ -143,6 +143,50 @@ Instance with push and requiring client authentication:
      bst-artifact-server --port 11002 --server-key server.key --server-cert server.crt --client-certs authorized.crt --enable-push /home/artifacts/artifacts
 +Managing the cache with systemd
 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
++
 +It is better to run the cache as a systemd service, especially if it is running on a dedicated server, as this will allow systemd to manage the cache, incase the server ever encounters any issues.
++
 +Below are two examples of how to run the cache server as a systemd service, one is for pull only and the other is configured for push & pull.
++
 +.. code:: ini
++
 +   #
 +   # Pull
 +   #
 +   [Unit]
 +   Description=Buildstream Artifact pull server
 +   After=remote-fs.target network-online.target
++
 +   [Service]
 +   Environment="LC_ALL=C.UTF-8"
 +   ExecStart=/usr/local/bin/bst-artifact-server --port 11001 --server-key {{certs_path}}/privkey.pem --
 +   server-cert {{certs_path}}/fullchain.pem {{artifacts_path}}
 +   User=artifacts
++
 +   [Install]
 +   WantedBy=multi-user.target
++
++
 +   #
 +   # Pull/Push
 +   #
 +   [Unit]
 +   Description=Buildstream Artifact pull/push server
 +   After=remote-fs.target network-online.target
++
 +   [Service]
 +   Environment="LC_ALL=C.UTF-8"
 +   ExecStart=/usr/local/bin/bst-artifact-server --port 11002 --server-key {{certs_path}}/privkey.pem --
 +   server-cert {{certs_path}}/fullchain.pem --client-certs /home/artifacts/authorized.crt --enable-push /
 +   {{artifacts_path}}
 +   User=artifacts
++
 +   [Install]
 +   WantedBy=multi-user.target
++
 +Here we define when systemd should start the service, which is after the networking stack has been started, we then define how to run the cache with the desired configuration, under the artifacts user. The {{ }} are there to denote where you should change these files to point to your desired locations.
  User configuration
  ~~~~~~~~~~~~~~~~~~

doc/source/install_linux_distro.rst

@@ -57,9 +57,20 @@ Install the dependencies with::
  For the default plugins::
    sudo pacman -S \
 -      bzr git lzip ostree patch python-arpy python-gobject
 +      bzr git lzip ostree patch python-gobject
 +The package *python-arpy* is required by the deb source plugin. This is not
 +obtainable via `pacman`, you must get *python-arpy* from AUR:
 +https://aur.archlinux.org/packages/python-arpy/
++
 +To install::
++
 +  wget https://aur.archlinux.org/cgit/aur.git/snapshot/python-arpy.tar.gz
 +  tar -xvf python-arpy.tar.gz
 +  cd python-arpy
 +  makepkg -si
++
  Debian
  ++++++
  Install the dependencies with::
@@ -216,17 +227,18 @@ Installing from distro packages
  Arch Linux
  ~~~~~~~~~~
 -Install `buildstream <https://aur.archlinux.org/packages/buildstream>`_
 -from `AUR <https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages>`_.
 -Alternatively, use
 -`buildstream-git <https://aur.archlinux.org/packages/buildstream-git>`_
 -for the lastest version of the development branch.
 +Packages for Arch exist in `AUR <https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages>`_.
 +Two different package versions are available:
++
 +* Latest release: `buildstream <https://aur.archlinux.org/packages/buildstream>`_
 +* Latest development snapshot: `buildstream-git <https://aur.archlinux.org/packages/buildstream-git>`_
++
  Fedora
  ~~~~~~
  BuildStream is not yet in the official Fedora repositories, but you can
 -install it from a Copr:
 +install it from a Copr::
    sudo dnf copr enable bochecha/buildstream
    sudo dnf install buildstream

tests/sources/git.py

@@ -359,3 +359,45 @@ def test_submodule_track_ignore_inconsistent(cli, tmpdir, datafiles):
      # Assert that we are just fine without it, and emit a warning to the user.
      assert "Ignoring inconsistent submodule" in result.stderr
++
++
 +@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
 +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
 +def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
++
 +    # Create the repo from 'repofiles' subdir
 +    repo = create_repo('git', str(tmpdir))
 +    ref = repo.create(os.path.join(project, 'repofiles'))
++
 +    # Write out our test target
 +    gitsource = repo.source_config(ref=None)
 +    gitsource.pop('track')
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            gitsource
 +        ]
 +    }
++
 +    _yaml.dump(element, os.path.join(project, 'target.bst'))
++
 +    # Track will encounter an inconsistent submodule without any ref
 +    result = cli.run(project=project, args=['track', 'target.bst'])
 +    result.assert_main_error(ErrorDomain.STREAM, None)
 +    result.assert_task_error(ErrorDomain.SOURCE, 'track-attempt-no-track')
++
 +    # Assert that we are just fine without it, and emit a warning to the user.
 +    assert "FAILURE git source at" in result.stderr
 +    assert "Without a tracking branch ref can not be updated. Please " + \
 +        "provide a ref or a track." in result.stderr
++
 +    # Track will encounter an inconsistent submodule without any ref
 +    result = cli.run(project=project, args=['build', 'target.bst'])
 +    result.assert_main_error(ErrorDomain.PIPELINE, 'inconsistent-pipeline')
 +    result.assert_task_error(None, None)
++
 +    # Assert that we are just fine without it, and emit a warning to the user.
 +    assert "Exact versions are missing for the following elements" in result.stderr
 +    assert "is missing ref and track." in result.stderr
 +    assert "Then track these elements with `bst track`" in result.stderr

[Notes] [Git][BuildStream/buildstream][willsalmon/trackWarning] 8 commits: source/install_linux_distro.rst: Make clearer ArchLinux packages available

Will Salmon pushed to branch willsalmon/trackWarning at BuildStream / buildstream

Commits:

5 changed files:

Changes: