[Notes] [Git][BuildStream/buildstream][tiagogomes/issue-500] 7 commits: Add remote source plugin



Title: GitLab

Tiago Gomes pushed to branch tiagogomes/issue-500 at BuildStream / buildstream

Commits:

24 changed files:

Changes:

  • HACKING.rst
    ... ... @@ -294,7 +294,7 @@ committed with that.
    294 294
     To do this, first ensure you have ``click_man`` installed, possibly
    
    295 295
     with::
    
    296 296
     
    
    297
    -  pip install --user click_man
    
    297
    +  pip3 install --user click_man
    
    298 298
     
    
    299 299
     Then, in the toplevel directory of buildstream, run the following::
    
    300 300
     
    
    ... ... @@ -450,7 +450,7 @@ To run the tests, just type::
    450 450
     At the toplevel.
    
    451 451
     
    
    452 452
     When debugging a test, it can be desirable to see the stdout
    
    453
    -and stderr generated by a test, to do this use the --addopts
    
    453
    +and stderr generated by a test, to do this use the ``--addopts``
    
    454 454
     function to feed arguments to pytest as such::
    
    455 455
     
    
    456 456
       ./setup.py test --addopts -s
    
    ... ... @@ -530,7 +530,7 @@ tool.
    530 530
     
    
    531 531
     Python provides `cProfile <https://docs.python.org/3/library/profile.html>`_
    
    532 532
     which gives you a list of all functions called during execution and how much
    
    533
    -time was spent in each function. Here is an example of running `bst --help`
    
    533
    +time was spent in each function. Here is an example of running ``bst --help``
    
    534 534
     under cProfile:
    
    535 535
     
    
    536 536
         python3 -m cProfile -o bst.cprofile -- $(which bst) --help
    

  • buildstream/_versions.py
    ... ... @@ -23,7 +23,7 @@
    23 23
     # This version is bumped whenever enhancements are made
    
    24 24
     # to the `project.conf` format or the core element format.
    
    25 25
     #
    
    26
    -BST_FORMAT_VERSION = 9
    
    26
    +BST_FORMAT_VERSION = 10
    
    27 27
     
    
    28 28
     
    
    29 29
     # The base BuildStream artifact version
    

  • buildstream/plugins/sources/remote.py
    1
    +#
    
    2
    +#  Copyright Bloomberg Finance LP
    
    3
    +#
    
    4
    +#  This program is free software; you can redistribute it and/or
    
    5
    +#  modify it under the terms of the GNU Lesser General Public
    
    6
    +#  License as published by the Free Software Foundation; either
    
    7
    +#  version 2 of the License, or (at your option) any later version.
    
    8
    +#
    
    9
    +#  This library is distributed in the hope that it will be useful,
    
    10
    +#  but WITHOUT ANY WARRANTY; without even the implied warranty of
    
    11
    +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
    
    12
    +#  Lesser General Public License for more details.
    
    13
    +#
    
    14
    +#  You should have received a copy of the GNU Lesser General Public
    
    15
    +#  License along with this library. If not, see <http://www.gnu.org/licenses/>.
    
    16
    +#
    
    17
    +#  Authors:
    
    18
    +#        Ed Baunton <ebaunton1 bloomberg net>
    
    19
    +
    
    20
    +"""
    
    21
    +remote - stage files from remote urls
    
    22
    +=====================================
    
    23
    +
    
    24
    +**Usage:**
    
    25
    +
    
    26
    +.. code:: yaml
    
    27
    +
    
    28
    +   # Specify the remote source kind
    
    29
    +   kind: remote
    
    30
    +
    
    31
    +   # Optionally specify a relative staging directory
    
    32
    +   # directory: path/to/stage
    
    33
    +
    
    34
    +   # Optionally specify a relative staging filename.
    
    35
    +   # If not specified, the basename of the url will be used.
    
    36
    +   # filename: customfilename
    
    37
    +
    
    38
    +   # Specify the url. Using an alias defined in your project
    
    39
    +   # configuration is encouraged. 'bst track' will update the
    
    40
    +   # sha256sum in 'ref' to the downloaded file's sha256sum.
    
    41
    +   url: upstream:foo
    
    42
    +
    
    43
    +   # Specify the ref. It's a sha256sum of the file you download.
    
    44
    +   ref: 6c9f6f68a131ec6381da82f2bff978083ed7f4f7991d931bfa767b7965ebc94b
    
    45
    +
    
    46
    +.. note::
    
    47
    +
    
    48
    +   The ``remote`` plugin is available since :ref:`format version 10 <project_format_version>`
    
    49
    +
    
    50
    +"""
    
    51
    +import os
    
    52
    +from buildstream import SourceError, utils
    
    53
    +from ._downloadablefilesource import DownloadableFileSource
    
    54
    +
    
    55
    +
    
    56
    +class RemoteSource(DownloadableFileSource):
    
    57
    +    # pylint: disable=attribute-defined-outside-init
    
    58
    +
    
    59
    +    def configure(self, node):
    
    60
    +        super().configure(node)
    
    61
    +
    
    62
    +        self.filename = self.node_get_member(node, str, 'filename', os.path.basename(self.url))
    
    63
    +
    
    64
    +        if os.sep in self.filename:
    
    65
    +            raise SourceError('{}: filename parameter cannot contain directories'.format(self),
    
    66
    +                              reason="filename-contains-directory")
    
    67
    +        self.node_validate(node, DownloadableFileSource.COMMON_CONFIG_KEYS + ['filename'])
    
    68
    +
    
    69
    +    def get_unique_key(self):
    
    70
    +        return super().get_unique_key() + [self.filename]
    
    71
    +
    
    72
    +    def stage(self, directory):
    
    73
    +        # Same as in local plugin, don't use hardlinks to stage sources, they
    
    74
    +        # are not write protected in the sandbox.
    
    75
    +        dest = os.path.join(directory, self.filename)
    
    76
    +        with self.timed_activity("Staging remote file to {}".format(dest)):
    
    77
    +            utils.safe_copy(self._get_mirror_file(), dest)
    
    78
    +
    
    79
    +
    
    80
    +def setup():
    
    81
    +    return RemoteSource

  • buildstream/utils.py
    ... ... @@ -802,6 +802,12 @@ def _process_list(srcdir, destdir, filelist, actionfunc, result,
    802 802
                 os.mknod(destpath, file_stat.st_mode, file_stat.st_rdev)
    
    803 803
                 os.chmod(destpath, file_stat.st_mode)
    
    804 804
     
    
    805
    +        elif stat.S_ISFIFO(mode):
    
    806
    +            os.mkfifo(destpath, mode)
    
    807
    +
    
    808
    +        elif stat.S_ISSOCK(mode):
    
    809
    +            pass
    
    810
    +
    
    805 811
             else:
    
    806 812
                 # Unsupported type.
    
    807 813
                 raise UtilError('Cannot extract {} into staging-area. Unsupported type.'.format(srcpath))
    

  • doc/source/core_plugins.rst
    ... ... @@ -50,6 +50,7 @@ Sources
    50 50
        :maxdepth: 1
    
    51 51
     
    
    52 52
        sources/local
    
    53
    +   sources/remote
    
    53 54
        sources/tar
    
    54 55
        sources/zip
    
    55 56
        sources/git
    

  • doc/source/examples/flatpak-autotools.rst
    ... ... @@ -58,7 +58,7 @@ it's important to note two things:
    58 58
     * **source**: We only want to extract the ``files/`` directory from the SDK,
    
    59 59
     
    
    60 60
       This is becase Flatpak runtimes dont start at the root of the OSTree checkout,
    
    61
    -  instead the actual files start in the ``files//`` subdirectory
    
    61
    +  instead the actual files start in the ``files/`` subdirectory
    
    62 62
     
    
    63 63
     * **target**: The content we've extracted should be staged at ``/usr``
    
    64 64
     
    

  • doc/source/format_declaring.rst
    ... ... @@ -400,7 +400,7 @@ dependency and that all referenced variables are declared, the following is fine
    400 400
        It should be noted that variable resolution only happens after all
    
    401 401
        :ref:`Element Composition <format_composition>` has already taken place.
    
    402 402
     
    
    403
    -   This is to say that overriding ``%{version}`` at a higher priority will effect
    
    403
    +   This is to say that overriding ``%{version}`` at a higher priority will affect
    
    404 404
        the final result of ``%{release-text}``.
    
    405 405
     
    
    406 406
     
    

  • doc/source/format_project.rst
    ... ... @@ -548,7 +548,7 @@ The defaults environment for the build sandbox is defined here.
    548 548
          PATH: /usr/bin:/bin:/usr/sbin:/sbin
    
    549 549
     
    
    550 550
     Additionally, the special ``environment-nocache`` list which specifies
    
    551
    -which environment variables do not effect build output, and are thus
    
    551
    +which environment variables do not affect build output, and are thus
    
    552 552
     not considered in the calculation of artifact keys can be defined here.
    
    553 553
     
    
    554 554
     .. code:: yaml
    
    ... ... @@ -558,7 +558,7 @@ not considered in the calculation of artifact keys can be defined here.
    558 558
     
    
    559 559
     Note that the ``environment-nocache`` list only exists so that we can
    
    560 560
     control parameters such as ``make -j ${MAXJOBS}``, allowing us to control
    
    561
    -the number of jobs for a given build without effecting the resulting
    
    561
    +the number of jobs for a given build without affecting the resulting
    
    562 562
     cache key.
    
    563 563
     
    
    564 564
     
    

  • doc/source/install_docker.rst
    ... ... @@ -38,8 +38,8 @@ be able to integrate with Bash tab-completion if you invoke it in this way.
    38 38
     
    
    39 39
     Two Docker volumes are set up by the ``bst-here`` script:
    
    40 40
     
    
    41
    - * buildstream-cache -- mounted at ``~/.cache/buildstream``
    
    42
    - * buildstream-config -- mounted at ``~/.config/``
    
    41
    + * ``buildstream-cache --`` mounted at ``~/.cache/buildstream``
    
    42
    + * ``buildstream-config --`` mounted at ``~/.config/``
    
    43 43
     
    
    44 44
     These are necessary so that your BuildStream cache and configuration files
    
    45 45
     persist between invocations of ``bst-here``.

  • doc/source/install_linux_distro.rst
    ... ... @@ -77,17 +77,17 @@ setup as described `here <https://backports.debian.org/Instructions/>`_
    77 77
     
    
    78 78
     By adding the following line to your sources.list::
    
    79 79
     
    
    80
    -  deb http://ftp.debian.org/debian stretch-backports main
    
    80
    +  deb http://deb.debian.org/debian stretch-backports main
    
    81 81
     
    
    82 82
     And then running::
    
    83 83
     
    
    84
    -  sudo apt-get update
    
    84
    +  sudo apt update
    
    85 85
     
    
    86 86
     At this point you should be able to get the system requirements for the default plugins with::
    
    87 87
     
    
    88
    -  sudo apt-get install \
    
    88
    +  sudo apt install \
    
    89 89
           bzr git lzip patch python3-arpy python3-gi
    
    90
    -  sudo apt-get install -t stretch-backports \
    
    90
    +  sudo apt install -t stretch-backports \
    
    91 91
           gir1.2-ostree-1.0 ostree
    
    92 92
     
    
    93 93
     Buster or Sid
    
    ... ... @@ -177,8 +177,8 @@ checkout::
    177 177
       cd /path/to/buildstream
    
    178 178
       git pull --rebase
    
    179 179
     
    
    180
    -If you did not specify the ``-e`` option at install time, you will
    
    181
    -need to cleanly reinstall BuildStream::
    
    180
    +If you did not specify the ``-e`` option at install time or the dependancies
    
    181
    +have changed, you will need to cleanly reinstall BuildStream::
    
    182 182
     
    
    183 183
       pip3 uninstall buildstream
    
    184 184
       cd /path/to/buildstream
    

  • doc/source/tutorial/autotools.rst
    ... ... @@ -109,7 +109,7 @@ changes the output.
    109 109
     
    
    110 110
     Observe where the variables are declared in the :ref:`builtin defaults
    
    111 111
     <project_builtin_defaults>` and :mod:`autotools <elements.autotools>` element
    
    112
    -documentation, and how overriding these affects the resolved set of variables.
    
    112
    +documentation, and how overriding these effects the resolved set of variables.
    
    113 113
     
    
    114 114
     
    
    115 115
     Using the project
    

  • tests/sources/remote.py
    1
    +import os
    
    2
    +import pytest
    
    3
    +
    
    4
    +from buildstream._exceptions import ErrorDomain
    
    5
    +from buildstream import _yaml
    
    6
    +from tests.testutils import cli
    
    7
    +
    
    8
    +DATA_DIR = os.path.join(
    
    9
    +    os.path.dirname(os.path.realpath(__file__)),
    
    10
    +    'remote',
    
    11
    +)
    
    12
    +
    
    13
    +
    
    14
    +def generate_project(project_dir, tmpdir):
    
    15
    +    project_file = os.path.join(project_dir, "project.conf")
    
    16
    +    _yaml.dump({
    
    17
    +        'name': 'foo',
    
    18
    +        'aliases': {
    
    19
    +            'tmpdir': "file:///" + str(tmpdir)
    
    20
    +        }
    
    21
    +    }, project_file)
    
    22
    +
    
    23
    +
    
    24
    +# Test that without ref, consistency is set appropriately.
    
    25
    +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'no-ref'))
    
    26
    +def test_no_ref(cli, tmpdir, datafiles):
    
    27
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    
    28
    +    generate_project(project, tmpdir)
    
    29
    +    assert cli.get_element_state(project, 'target.bst') == 'no reference'
    
    30
    +
    
    31
    +
    
    32
    +# Here we are doing a fetch on a file that doesn't exist. target.bst
    
    33
    +# refers to 'file' but that file is not present.
    
    34
    +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'missing-file'))
    
    35
    +def test_missing_file(cli, tmpdir, datafiles):
    
    36
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    
    37
    +    generate_project(project, tmpdir)
    
    38
    +
    
    39
    +    # Try to fetch it
    
    40
    +    result = cli.run(project=project, args=[
    
    41
    +        'fetch', 'target.bst'
    
    42
    +    ])
    
    43
    +
    
    44
    +    result.assert_main_error(ErrorDomain.STREAM, None)
    
    45
    +    result.assert_task_error(ErrorDomain.SOURCE, None)
    
    46
    +
    
    47
    +
    
    48
    +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'path-in-filename'))
    
    49
    +def test_path_in_filename(cli, tmpdir, datafiles):
    
    50
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    
    51
    +    generate_project(project, tmpdir)
    
    52
    +
    
    53
    +    # Try to fetch it
    
    54
    +    result = cli.run(project=project, args=[
    
    55
    +        'fetch', 'target.bst'
    
    56
    +    ])
    
    57
    +
    
    58
    +    # The bst file has a / in the filename param
    
    59
    +    result.assert_main_error(ErrorDomain.SOURCE, "filename-contains-directory")
    
    60
    +
    
    61
    +
    
    62
    +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'single-file'))
    
    63
    +def test_simple_file_build(cli, tmpdir, datafiles):
    
    64
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    
    65
    +    generate_project(project, tmpdir)
    
    66
    +    checkoutdir = os.path.join(str(tmpdir), "checkout")
    
    67
    +
    
    68
    +    # Try to fetch it
    
    69
    +    result = cli.run(project=project, args=[
    
    70
    +        'fetch', 'target.bst'
    
    71
    +    ])
    
    72
    +    result.assert_success()
    
    73
    +
    
    74
    +    result = cli.run(project=project, args=[
    
    75
    +        'build', 'target.bst'
    
    76
    +    ])
    
    77
    +    result.assert_success()
    
    78
    +
    
    79
    +    result = cli.run(project=project, args=[
    
    80
    +        'checkout', 'target.bst', checkoutdir
    
    81
    +    ])
    
    82
    +    result.assert_success()
    
    83
    +    # Note that the url of the file in target.bst is actually /dir/file
    
    84
    +    # but this tests confirms we take the basename
    
    85
    +    assert(os.path.exists(os.path.join(checkoutdir, 'file')))
    
    86
    +
    
    87
    +
    
    88
    +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'single-file-custom-name'))
    
    89
    +def test_simple_file_custom_name_build(cli, tmpdir, datafiles):
    
    90
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    
    91
    +    generate_project(project, tmpdir)
    
    92
    +    checkoutdir = os.path.join(str(tmpdir), "checkout")
    
    93
    +
    
    94
    +    # Try to fetch it
    
    95
    +    result = cli.run(project=project, args=[
    
    96
    +        'fetch', 'target.bst'
    
    97
    +    ])
    
    98
    +    result.assert_success()
    
    99
    +
    
    100
    +    result = cli.run(project=project, args=[
    
    101
    +        'build', 'target.bst'
    
    102
    +    ])
    
    103
    +    result.assert_success()
    
    104
    +
    
    105
    +    result = cli.run(project=project, args=[
    
    106
    +        'checkout', 'target.bst', checkoutdir
    
    107
    +    ])
    
    108
    +    result.assert_success()
    
    109
    +    assert(not os.path.exists(os.path.join(checkoutdir, 'file')))
    
    110
    +    assert(os.path.exists(os.path.join(checkoutdir, 'custom-file')))
    
    111
    +
    
    112
    +
    
    113
    +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'unique-keys'))
    
    114
    +def test_unique_key(cli, tmpdir, datafiles):
    
    115
    +    '''This test confirms that the 'filename' parameter is honoured when it comes
    
    116
    +    to generating a cache key for the source.
    
    117
    +    '''
    
    118
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    
    119
    +    generate_project(project, tmpdir)
    
    120
    +    assert cli.get_element_state(project, 'target.bst') == "fetch needed"
    
    121
    +    assert cli.get_element_state(project, 'target-custom.bst') == "fetch needed"
    
    122
    +    # Try to fetch it
    
    123
    +    result = cli.run(project=project, args=[
    
    124
    +        'fetch', 'target.bst'
    
    125
    +    ])
    
    126
    +
    
    127
    +    # We should download the file only once
    
    128
    +    assert cli.get_element_state(project, 'target.bst') == 'buildable'
    
    129
    +    assert cli.get_element_state(project, 'target-custom.bst') == 'buildable'
    
    130
    +
    
    131
    +    # But the cache key is different because the 'filename' is different.
    
    132
    +    assert cli.get_element_key(project, 'target.bst') != \
    
    133
    +        cli.get_element_key(project, 'target-custom.bst')

  • tests/sources/remote/missing-file/target.bst
    1
    +kind: autotools
    
    2
    +description: The kind of this element is irrelevant.
    
    3
    +sources:
    
    4
    +- kind: remote
    
    5
    +  url: tmpdir:/file
    
    6
    +  ref: abcdef
    
    7
    +  filename: filename

  • tests/sources/remote/no-ref/file
    1
    +filecontent

  • tests/sources/remote/no-ref/target.bst
    1
    +kind: autotools
    
    2
    +description: The kind of this element is irrelevant.
    
    3
    +sources:
    
    4
    +- kind: remote
    
    5
    +  url: tmpdir:/file

  • tests/sources/remote/path-in-filename/dir/file

  • tests/sources/remote/path-in-filename/target.bst
    1
    +kind: import
    
    2
    +description: test
    
    3
    +sources:
    
    4
    +- kind: remote
    
    5
    +  url: tmpdir:/dir/file
    
    6
    +  ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    
    7
    +  filename: path/to/file

  • tests/sources/remote/single-file-custom-name/dir/file

  • tests/sources/remote/single-file-custom-name/target.bst
    1
    +kind: import
    
    2
    +description: test
    
    3
    +sources:
    
    4
    +- kind: remote
    
    5
    +  url: tmpdir:/dir/file
    
    6
    +  ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    
    7
    +  filename: custom-file

  • tests/sources/remote/single-file/dir/file

  • tests/sources/remote/single-file/target.bst
    1
    +kind: import
    
    2
    +description: test
    
    3
    +sources:
    
    4
    +- kind: remote
    
    5
    +  url: tmpdir:/dir/file
    
    6
    +  ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

  • tests/sources/remote/unique-keys/dir/file

  • tests/sources/remote/unique-keys/target-custom.bst
    1
    +kind: import
    
    2
    +description: test
    
    3
    +sources:
    
    4
    +- kind: remote
    
    5
    +  url: tmpdir:/dir/file
    
    6
    +  ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    
    7
    +  filename: some-custom-file

  • tests/sources/remote/unique-keys/target.bst
    1
    +kind: import
    
    2
    +description: test
    
    3
    +sources:
    
    4
    +- kind: remote
    
    5
    +  url: tmpdir:/dir/file
    
    6
    +  ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855



  • [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]