GNOME.org
 

[Notes] [Git][BuildStream/buildstream][jjardon/use_freedesktop-sdk_18_08] 10 commits: Sandbox: CWD was not being created for workspaces

Title: GitLab

Javier Jardón pushed to branch jjardon/use_freedesktop-sdk_18_08 at BuildStream / buildstream

Commits:

26 changed files:

Changes:

  • NEWS
    ... ... @@ -27,6 +27,9 @@ buildstream 1.3.1
    27 27 
       o Generate Docker images from built artifacts using
    28 28 
         `contrib/bst-docker-import` script.
    29 29
    30 
    +  o Added Documentation on how to create out of source builds. This includes the
    31 
    +    new the `conf-root` variable to make the process easier. And there has been
    32 
    +    a bug fix to workspaces so they can be build in workspaces too.
    30 33
    31 34 
     =================
    32 35 
     buildstream 1.1.5

  • buildstream/_versions.py
    ... ... @@ -23,7 +23,7 @@
    23 23 
     # This version is bumped whenever enhancements are made
    24 24 
     # to the `project.conf` format or the core element format.
    25 25 
     #
    26 
    -BST_FORMAT_VERSION = 16
    26 
    +BST_FORMAT_VERSION = 17
    27 27
    28 28
    29 29 
     # The base BuildStream artifact version

  • buildstream/buildelement.py
    ... ... @@ -23,6 +23,50 @@ BuildElement - Abstract class for build elements
    23 23 
     The BuildElement class is a convenience element one can derive from for
    24 24 
     implementing the most common case of element.
    25 25
    26 
    +Built-in functionality
    27 
    +----------------------
    28 
    +
    29 
    +The BuildElement base class provides built in functionality that could be
    30 
    +overridden by the individual plugins.
    31 
    +
    32 
    +This section will give a brief summary of how some of the common features work,
    33 
    +some of them or the variables they use will be further detailed in the following
    34 
    +sections.
    35 
    +
    36 
    +Location for running commands
    37 
    +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    38 
    +The ``command-subdir`` variable sets where the build commands will be executed,
    39 
    +if the directory does not exist it will be created, it is defined relative to
    40 
    +the buildroot.
    41 
    +
    42 
    +Location for configuring the project
    43 
    +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    44 
    +The ``conf-root`` is defined by default as ``.`` and is the location that
    45 
    +specific build element can use to look for build configuration files. This is
    46 
    +used by elements such as autotools, cmake, distutils, meson, pip and qmake.
    47 
    +
    48 
    +The configuration commands are run in ``command-subdir`` and by default
    49 
    +``conf-root`` is ``.`` so if ``conf-root`` is not set the configuration files
    50 
    +in ``command-subdir`` will be used.
    51 
    +
    52 
    +By setting ``conf-root`` to ``"%{build-root}/Source/conf_location"`` and your
    53 
    +source elements ``directory`` variable to ``Source`` then the configuration
    54 
    +files in the directory ``conf_location`` with in your Source will be used.
    55 
    +The current working directory when your configuration command is run will still
    56 
    +be wherever you set your ``command-subdir`` to be, regardless of where the
    57 
    +configure scripts are set with ``conf-root``.
    58 
    +
    59 
    +.. note::
    60 
    +
    61 
    +   The ``conf-root`` variable is available since :ref:`format version 17 <project_format_version>`
    62 
    +
    63 
    +Install Location
    64 
    +~~~~~~~~~~~~~~~~
    65 
    +
    66 
    +You should not change the ``install-root`` variable as it is a special
    67 
    +writeable location in the sandbox but it is useful when writing custom
    68 
    +install instructions as it may need to be supplied as the ``DESTDIR``, please
    69 
    +see the :mod:`cmake <elements.cmake>` build element for example.
    26 70
    27 71 
     Abstract method implementations
    28 72 
     -------------------------------

  • buildstream/data/projectconfig.yaml
    ... ... @@ -38,6 +38,9 @@ variables:
    38 38 
       # normally staged
    39 39 
       build-root: /buildstream/%{project-name}/%{element-name}
    40 40
    41 
    +  # Indicates where the build system should look for configuration files
    42 
    +  conf-root: .
    43 
    +
    41 44 
       # Indicates the build installation directory in the sandbox
    42 45 
       install-root: /buildstream-install
    43 46

  • buildstream/plugins/elements/autotools.yaml
    ... ... @@ -6,11 +6,11 @@ variables:
    6 6 
         export NOCONFIGURE=1;
    7 7
    8 8 
         if [ -x %{conf-cmd} ]; then true;
    9 
    -    elif [ -x autogen ]; then ./autogen;
    10 
    -    elif [ -x autogen.sh ]; then ./autogen.sh;
    11 
    -    elif [ -x bootstrap ]; then ./bootstrap;
    12 
    -    elif [ -x bootstrap.sh ]; then ./bootstrap.sh;
    13 
    -    else autoreconf -ivf;
    9 
    +    elif [ -x %{conf-root}/autogen ]; then %{conf-root}/autogen;
    10 
    +    elif [ -x %{conf-root}/autogen.sh ]; then %{conf-root}/autogen.sh;
    11 
    +    elif [ -x %{conf-root}/bootstrap ]; then %{conf-root}/bootstrap;
    12 
    +    elif [ -x %{conf-root}/bootstrap.sh ]; then %{conf-root}/bootstrap.sh;
    13 
    +    else autoreconf -ivf %{conf-root};
    14 14 
         fi
    15 15
    16 16 
       # Project-wide extra arguments to be passed to `configure`
    ... ... @@ -22,7 +22,8 @@ variables:
    22 22 
       # For backwards compatibility only, do not use.
    23 23 
       conf-extra: ''
    24 24
    25 
    -  conf-cmd: ./configure
    25 
    +  conf-cmd: "%{conf-root}/configure"
    26 
    +
    26 27 
       conf-args: |
    27 28
    28 29 
         --prefix=%{prefix} \

  • buildstream/plugins/elements/cmake.yaml
    ... ... @@ -23,7 +23,7 @@ variables:
    23 23
    24 24 
       cmake: |
    25 25
    26 
    -    cmake -B%{build-dir} -H. -G"%{generator}" %{cmake-args}
    26 
    +    cmake -B%{build-dir} -H"%{conf-root}" -G"%{generator}" %{cmake-args}
    27 27
    28 28 
       make: cmake --build %{build-dir} -- ${JOBS}
    29 29 
       make-install: env DESTDIR="%{install-root}" cmake --build %{build-dir} --target install

  • buildstream/plugins/elements/distutils.yaml
    ... ... @@ -8,7 +8,7 @@ variables:
    8 8
    9 9 
       python-build: |
    10 10
    11 
    -    %{python} setup.py build
    11 
    +    %{python} %{conf-root}/setup.py build
    12 12
    13 13 
       install-args: |
    14 14
    ... ... @@ -17,7 +17,7 @@ variables:
    17 17
    18 18 
       python-install: |
    19 19
    20 
    -    %{python} setup.py install %{install-args}
    20 
    +    %{python} %{conf-root}/setup.py install %{install-args}
    21 21
    22 22
    23 23 
     config:

  • buildstream/plugins/elements/meson.yaml
    ... ... @@ -28,7 +28,7 @@ variables:
    28 28 
         --mandir=%{mandir} \
    29 29 
         --infodir=%{infodir} %{meson-extra} %{meson-global} %{meson-local}
    30 30
    31 
    -  meson: meson %{build-dir} %{meson-args}
    31 
    +  meson: meson %{conf-root} %{build-dir} %{meson-args}
    32 32
    33 33 
       ninja: |
    34 34 
         ninja -j ${NINJAJOBS} -C %{build-dir}

  • buildstream/plugins/elements/pip.yaml
    ... ... @@ -14,7 +14,7 @@ config:
    14 14 
       #
    15 15 
       install-commands:
    16 16 
       - |
    17 
    -    %{pip} install --no-deps --root=%{install-root} --prefix=%{prefix} .
    17 
    +    %{pip} install --no-deps --root=%{install-root} --prefix=%{prefix} %{conf-root}
    18 18
    19 19 
       # Commands for stripping debugging information out of
    20 20 
       # installed binaries

  • buildstream/plugins/elements/qmake.yaml
    ... ... @@ -2,7 +2,7 @@
    2 2
    3 3 
     variables:
    4 4
    5 
    -  qmake: qmake -makefile
    5 
    +  qmake: qmake -makefile %{conf-root}
    6 6 
       make: make
    7 7 
       make-install: make -j1 INSTALL_ROOT="%{install-root}" install
    8 8

  • buildstream/sandbox/_sandboxbwrap.py
    ... ... @@ -108,9 +108,6 @@ class SandboxBwrap(Sandbox):
    108 108 
                 bwrap_command += ['--unshare-uts', '--hostname', 'buildstream']
    109 109 
                 bwrap_command += ['--unshare-ipc']
    110 110
    111 
    -        if cwd is not None:
    112 
    -            bwrap_command += ['--chdir', cwd]
    113 
    -
    114 111 
             # Give it a proc and tmpfs
    115 112 
             bwrap_command += [
    116 113 
                 '--proc', '/proc',
    ... ... @@ -151,6 +148,10 @@ class SandboxBwrap(Sandbox):
    151 148 
             if flags & SandboxFlags.ROOT_READ_ONLY:
    152 149 
                 bwrap_command += ["--remount-ro", "/"]
    153 150
    151 
    +        if cwd is not None:
    152 
    +            bwrap_command += ['--dir', cwd]
    153 
    +            bwrap_command += ['--chdir', cwd]
    154 
    +
    154 155 
             # Set UID and GUI
    155 156 
             if self.user_ns_available:
    156 157 
                 bwrap_command += ['--unshare-user']
    ... ... @@ -179,11 +180,6 @@ class SandboxBwrap(Sandbox):
    179 180 
             with ExitStack() as stack:
    180 181 
                 stack.enter_context(mount_map.mounted(self))
    181 182
    182 
    -            # Ensure the cwd exists
    183 
    -            if cwd is not None:
    184 
    -                workdir = os.path.join(root_mount_source, cwd.lstrip(os.sep))
    185 
    -                os.makedirs(workdir, exist_ok=True)
    186 
    -
    187 183 
                 # If we're interactive, we want to inherit our stdin,
    188 184 
                 # otherwise redirect to /dev/null, ensuring process
    189 185 
                 # disconnected from terminal.

  • buildstream/sandbox/_sandboxchroot.py
    ... ... @@ -100,9 +100,8 @@ class SandboxChroot(Sandbox):
    100 100
    101 101 
                 # Ensure the cwd exists
    102 102 
                 if cwd is not None:
    103 
    -                workdir = os.path.join(root_mount_source, cwd.lstrip(os.sep))
    103 
    +                workdir = os.path.join(rootfs, cwd.lstrip(os.sep))
    104 104 
                     os.makedirs(workdir, exist_ok=True)
    105 
    -
    106 105 
                 status = self.chroot(rootfs, command, stdin, stdout,
    107 106 
                                      stderr, cwd, env, flags)
    108 107

  • buildstream/sandbox/sandbox.py
    ... ... @@ -223,7 +223,9 @@ class Sandbox():
    223 223 
             .. note::
    224 224
    225 225 
                The optional *cwd* argument will default to the value set with
    226 
    -           :func:`~buildstream.sandbox.Sandbox.set_work_directory`
    226 
    +           :func:`~buildstream.sandbox.Sandbox.set_work_directory` and this
    227 
    +           function must make sure the directory will be created if it does
    228 
    +           not exist yet, even if a workspace is being used.
    227 229 
             """
    228 230 
             raise ImplError("Sandbox of type '{}' does not implement run()"
    229 231 
                             .format(type(self).__name__))

  • buildstream/source.py
    ... ... @@ -20,6 +20,19 @@
    20 20 
     Source - Base source class
    21 21 
     ==========================
    22 22
    23 
    +Built-in functionality
    24 
    +----------------------
    25 
    +
    26 
    +The Source base class provides built in functionality that may be overridden
    27 
    +by individual plugins.
    28 
    +
    29 
    +* Directory
    30 
    +
    31 
    +  The ``directory`` variable can be set for all sources of a type in project.conf
    32 
    +  or per source within a element.
    33 
    +
    34 
    +  This sets the location within the build root that the content of the source
    35 
    +  will be loaded in to. If the location does not exist, it will be created.
    23 36
    24 37 
     .. _core_source_abstract_methods:
    25 38

  • doc/bst2html.py
    ... ... @@ -440,7 +440,7 @@ def run_session(description, tempdir, source_cache, palette, config_file, force)
    440 440 
     @click.option('--palette', '-p', default='tango',
    441 441 
                   type=click.Choice(['solarized', 'solarized-xterm', 'tango', 'xterm', 'console']),
    442 442 
                   help="Selects a palette for the output style")
    443 
    -@click.argument('description', click.Path(file_okay=True, dir_okay=False, readable=True))
    443 
    +@click.argument('description', type=click.Path(file_okay=True, dir_okay=False, readable=True))
    444 444 
     def run_bst(directory, force, source_cache, description, palette):
    445 445 
         """Run a bst command and capture stdout/stderr in html
    446 446

  • doc/examples/flatpak-autotools/elements/base/sdk.bst
    ... ... @@ -2,15 +2,15 @@ kind: import
    2 2 
     description: Import the base freedesktop SDK
    3 3 
     sources:
    4 4 
     - kind: ostree
    5 
    -  url: gnomesdk:repo/
    6 
    -  gpg-key: keys/gnome-sdk.gpg
    5 
    +  url: flathub:repo/
    6 
    +  gpg-key: keys/flathub.gpg
    7 7 
       (?):
    8 8 
       - arch == "x86_64":
    9 
    -      track: runtime/org.freedesktop.BaseSdk/x86_64/1.4
    10 
    -      ref: 0d9d255d56b08aeaaffb1c820eef85266eb730cb5667e50681185ccf5cd7c882
    9 
    +      track: runtime/org.freedesktop.Sdk/x86_64/18.08
    10 
    +      ref: 266c14050460f4747ffaac89b1050c027dfad51f4fc5742d221b4a5d12e3fb71
    11 11 
       - arch == "i386":
    12 
    -      track: runtime/org.freedesktop.BaseSdk/i386/1.4
    13 
    -      ref: 16036b747c1ec8e7fe291f5b1f667cb942f0267d08fcad962e9b7627d6cf1981
    12 
    +      track: runtime/org.freedesktop.Sdk/i386/18.08
    13 
    +      ref: 30230a5eba26b77c334e125493a4b798bd304ab6d5186574f150de5c8cdee350
    14 14 
     config:
    15 15 
       source: files
    16 16 
       target: usr

  • doc/examples/flatpak-autotools/keys/flathub.gpg
    No preview for this file type
  • doc/examples/flatpak-autotools/keys/gnome-sdk.gpg deleted
    No preview for this file type
  • doc/examples/flatpak-autotools/project.conf
    1 1 
     name: flatpak-autotools
    2 2
    3 3 
     aliases:
    4 
    -  gnomesdk: https://sdk.gnome.org/
    4 
    +  flathub: https://dl.flathub.org/
    5 5
    6 6 
     element-path: elements
    7 7

  • tests/format/variables.py
    ... ... @@ -19,10 +19,10 @@ DATA_DIR = os.path.join(
    19 19 
     @pytest.mark.parametrize("target,varname,expected", [
    20 20 
         ('autotools.bst', 'make-install', "make -j1 DESTDIR=\"/buildstream-install\" install"),
    21 21 
         ('cmake.bst', 'cmake',
    22 
    -     "cmake -B_builddir -H. -G\"Unix Makefiles\" -DCMAKE_INSTALL_PREFIX:PATH=\"/usr\" \\\n" +
    22 
    +     "cmake -B_builddir -H\".\" -G\"Unix Makefiles\" " + "-DCMAKE_INSTALL_PREFIX:PATH=\"/usr\" \\\n" +
    23 23 
          "-DCMAKE_INSTALL_LIBDIR=lib   "),
    24 24 
         ('distutils.bst', 'python-install',
    25 
    -     "python3 setup.py install --prefix \"/usr\" \\\n" +
    25 
    +     "python3 ./setup.py install --prefix \"/usr\" \\\n" +
    26 26 
          "--root \"/buildstream-install\""),
    27 27 
         ('makemaker.bst', 'configure', "perl Makefile.PL PREFIX=/buildstream-install/usr"),
    28 28 
         ('modulebuild.bst', 'configure', "perl Build.PL --prefix \"/buildstream-install/usr\""),
    ... ... @@ -45,10 +45,10 @@ def test_defaults(cli, datafiles, tmpdir, target, varname, expected):
    45 45 
     @pytest.mark.parametrize("target,varname,expected", [
    46 46 
         ('autotools.bst', 'make-install', "make -j1 DESTDIR=\"/custom/install/root\" install"),
    47 47 
         ('cmake.bst', 'cmake',
    48 
    -     "cmake -B_builddir -H. -G\"Ninja\" -DCMAKE_INSTALL_PREFIX:PATH=\"/opt\" \\\n" +
    48 
    +     "cmake -B_builddir -H\".\" -G\"Ninja\" " + "-DCMAKE_INSTALL_PREFIX:PATH=\"/opt\" \\\n" +
    49 49 
          "-DCMAKE_INSTALL_LIBDIR=lib   "),
    50 50 
         ('distutils.bst', 'python-install',
    51 
    -     "python3 setup.py install --prefix \"/opt\" \\\n" +
    51 
    +     "python3 ./setup.py install --prefix \"/opt\" \\\n" +
    52 52 
          "--root \"/custom/install/root\""),
    53 53 
         ('makemaker.bst', 'configure', "perl Makefile.PL PREFIX=/custom/install/root/opt"),
    54 54 
         ('modulebuild.bst', 'configure', "perl Build.PL --prefix \"/custom/install/root/opt\""),

  • tests/integration/autotools.py
    ... ... @@ -38,6 +38,30 @@ def test_autotools_build(cli, tmpdir, datafiles):
    38 38 
                                    '/usr/share/doc/amhello/README'])
    39 39
    40 40
    41 
    +# Test that an autotools build 'works' - we use the autotools sample
    42 
    +# amhello project for this.
    43 
    +@pytest.mark.integration
    44 
    +@pytest.mark.datafiles(DATA_DIR)
    45 
    +def test_autotools_confroot_build(cli, tmpdir, datafiles):
    46 
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    47 
    +    checkout = os.path.join(cli.directory, 'checkout')
    48 
    +    element_name = 'autotools/amhelloconfroot.bst'
    49 
    +
    50 
    +    result = cli.run(project=project, args=['build', element_name])
    51 
    +    assert result.exit_code == 0
    52 
    +
    53 
    +    result = cli.run(project=project, args=['checkout', element_name, checkout])
    54 
    +    assert result.exit_code == 0
    55 
    +
    56 
    +    assert_contains(checkout, ['/usr', '/usr/lib', '/usr/bin',
    57 
    +                               '/usr/share', '/usr/lib/debug',
    58 
    +                               '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
    59 
    +                               '/usr/lib/debug/usr/bin/hello',
    60 
    +                               '/usr/bin/hello', '/usr/share/doc',
    61 
    +                               '/usr/share/doc/amhello',
    62 
    +                               '/usr/share/doc/amhello/README'])
    63 
    +
    64 
    +
    41 65 
     # Test running an executable built with autotools
    42 66 
     @pytest.mark.datafiles(DATA_DIR)
    43 67 
     def test_autotools_run(cli, tmpdir, datafiles):

  • tests/integration/cmake.py
    ... ... @@ -32,6 +32,24 @@ def test_cmake_build(cli, tmpdir, datafiles):
    32 32 
                                    '/usr/lib/debug/usr/bin/hello'])
    33 33
    34 34
    35 
    +@pytest.mark.datafiles(DATA_DIR)
    36 
    +def test_cmake_confroot_build(cli, tmpdir, datafiles):
    37 
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    38 
    +    checkout = os.path.join(cli.directory, 'checkout')
    39 
    +    element_name = 'cmake/cmakeconfroothello.bst'
    40 
    +
    41 
    +    result = cli.run(project=project, args=['build', element_name])
    42 
    +    assert result.exit_code == 0
    43 
    +
    44 
    +    result = cli.run(project=project, args=['checkout', element_name, checkout])
    45 
    +    assert result.exit_code == 0
    46 
    +
    47 
    +    assert_contains(checkout, ['/usr', '/usr/bin', '/usr/bin/hello',
    48 
    +                               '/usr/lib/debug', '/usr/lib/debug/usr',
    49 
    +                               '/usr/lib/debug/usr/bin',
    50 
    +                               '/usr/lib/debug/usr/bin/hello'])
    51 
    +
    52 
    +
    35 53 
     @pytest.mark.datafiles(DATA_DIR)
    36 54 
     def test_cmake_run(cli, tmpdir, datafiles):
    37 55 
         project = os.path.join(datafiles.dirname, datafiles.basename)

  • tests/integration/project/elements/autotools/amhelloconfroot.bst
    1 
    +kind: autotools
    2 
    +description: Autotools test
    3 
    +
    4 
    +depends:
    5 
    +- base.bst
    6 
    +
    7 
    +sources:
    8 
    +- kind: tar
    9 
    +  url: project_dir:/files/amhello.tar.gz
    10 
    +  ref: 9ba123fa4e660929e9a0aa99f0c487b7eee59c5e7594f3284d015640b90f5590
    11 
    +  directory: SourceFile
    12 
    +
    13 
    +variables:
    14 
    +  conf-root: "%{build-root}/SourceFile"
    15 
    +  command-subdir: build

  • tests/integration/project/elements/cmake/cmakeconfroothello.bst
    1 
    +kind: cmake
    2 
    +description: Cmake test
    3 
    +
    4 
    +depends:
    5 
    +  - base.bst
    6 
    +
    7 
    +sources:
    8 
    +  - kind: tar
    9 
    +    directory: Source
    10 
    +    url: project_dir:/files/cmakehello.tar.gz
    11 
    +    ref: 508266f40dbc5875293bd24c4e50a9eb6b88cbacab742033f7b92f8c087b64e5
    12 
    +
    13 
    +variables:
    14 
    +  conf-root: "%{build-root}/Source"
    15 
    +  command-subdir: build

  • tests/integration/project/elements/workspace/workspace-commanddir.bst
    1 
    +kind: manual
    2 
    +description: Workspace mount test
    3 
    +
    4 
    +depends:
    5 
    +  - filename: base.bst
    6 
    +    type: build
    7 
    +
    8 
    +sources:
    9 
    +  - kind: local
    10 
    +    path: files/workspace-mount-src/
    11 
    +
    12 
    +variables:
    13 
    +  command-subdir: build
    14 
    +
    15 
    +config:
    16 
    +  build-commands:
    17 
    +    - cc -c ../hello.c

  • tests/integration/workspace.py
    ... ... @@ -32,6 +32,23 @@ def test_workspace_mount(cli, tmpdir, datafiles):
    32 32 
         assert os.path.exists(os.path.join(cli.directory, 'workspace'))
    33 33
    34 34
    35 
    +@pytest.mark.integration
    36 
    +@pytest.mark.datafiles(DATA_DIR)
    37 
    +def test_workspace_commanddir(cli, tmpdir, datafiles):
    38 
    +    project = os.path.join(datafiles.dirname, datafiles.basename)
    39 
    +    workspace = os.path.join(cli.directory, 'workspace')
    40 
    +    element_name = 'workspace/workspace-commanddir.bst'
    41 
    +
    42 
    +    res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
    43 
    +    assert res.exit_code == 0
    44 
    +
    45 
    +    res = cli.run(project=project, args=['build', element_name])
    46 
    +    assert res.exit_code == 0
    47 
    +
    48 
    +    assert os.path.exists(os.path.join(cli.directory, 'workspace'))
    49 
    +    assert os.path.exists(os.path.join(cli.directory, 'workspace', 'build'))
    50 
    +
    51 
    +
    35 52 
     @pytest.mark.integration
    36 53 
     @pytest.mark.datafiles(DATA_DIR)
    37 54 
     def test_workspace_updated_dependency(cli, tmpdir, datafiles):

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.

    [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]