[Notes] [Git][BuildStream/buildstream][raoul/775-execution-environment-r

Raoul Hidalgo Charman pushed to branch raoul/775-execution-environment-reqs at BuildStream / buildstream

Commits:

113bfa6f

by Raoul Hidalgo Charman at 2018-12-03T12:09:22Z

Sandbox: use linux32 for x86-32 builds on x86-64 machines

  o _platform/linux.py: Add flag to send to sandbox bwrap when the build arch
    is x86-32 and the machines arch is x86-64.
  o sandbox/_sandboxbwrap.py: Use flag passed to start bwrap command with
    linux32 if set.

12f87a1f

by Raoul Hidalgo Charman at 2018-12-03T12:09:27Z

format_project.rst: Update documentation for changes to OptionArch

89d52561

by Raoul Hidalgo Charman at 2018-12-03T12:09:27Z

format_declaring.rst: Update documentation for sandbox options

17982ee9
by Raoul Hidalgo Charman at 2018-12-03T12:09:27Z
```
format_project.rst: Add docs for OptionOS
```

Changes:

buildstream/_platform/linux.py

@@ -59,6 +59,9 @@ class Linux(Platform):
          else:
              self._user_ns_available = False
 +        # Set linux32 option
 +        self._linux32 = False
++
      def create_sandbox(self, *args, **kwargs):
          if not self._local_sandbox_available:
              return self._create_dummy_sandbox(*args, **kwargs)
@@ -78,11 +81,25 @@ class Linux(Platform):
              # will match the host UID/GID.
              return False
 -        # We can't do builds for another host or architecture
 -        if config.build_os != self.get_host_os():
 +        # We can't do builds for another host or architecture except x86-32 on
 +        # x86-64
 +        host_os = self.get_host_os()
 +        host_arch = self.get_host_arch()
 +        if config.build_os != host_os:
              raise PlatformError("Configured and host OS don't match.")
 -        elif config.build_arch != self.get_host_arch():
 -            raise PlatformError("Configured and host architecture don't match.")
 +        elif config.build_arch != host_arch:
 +            # We can use linux32 for building 32bit on 64bit machines
 +            if (host_os == "Linux" and
 +                    ((config.build_arch == "x86-32" and host_arch == "x86-64") or
 +                     (config.build_arch == "AArch32" and host_arch == "AArch64"))):
 +                # check linux32 is available
 +                try:
 +                    utils.get_host_tool('linux32')
 +                    self._linux32 = True
 +                except utils.ProgramNotFoundError:
 +                    pass
 +            else:
 +                raise PlatformError("Configured architecture and host architecture don't match.")
          return True
@@ -109,6 +126,7 @@ class Linux(Platform):
          kwargs['user_ns_available'] = self._user_ns_available
          kwargs['die_with_parent_available'] = self._die_with_parent_available
          kwargs['json_status_available'] = self._json_status_available
 +        kwargs['linux32'] = self._linux32
          return SandboxBwrap(*args, **kwargs)
      def _check_user_ns_available(self):

buildstream/sandbox/_sandboxbwrap.py

@@ -57,6 +57,7 @@ class SandboxBwrap(Sandbox):
          self.user_ns_available = kwargs['user_ns_available']
          self.die_with_parent_available = kwargs['die_with_parent_available']
          self.json_status_available = kwargs['json_status_available']
 +        self.linux32 = kwargs['linux32']
      def _run(self, command, flags, *, cwd, env):
          stdout, stderr = self._get_output()
@@ -74,8 +75,14 @@ class SandboxBwrap(Sandbox):
          mount_map = MountMap(self, flags & SandboxFlags.ROOT_READ_ONLY)
          root_mount_source = mount_map.get_mount_source('/')
 +        # start command with linux32 if needed
 +        if self.linux32:
 +            bwrap_command = [utils.get_host_tool('linux32')]
 +        else:
 +            bwrap_command = []
++
          # Grab the full path of the bwrap binary
 -        bwrap_command = [utils.get_host_tool('bwrap')]
 +        bwrap_command += [utils.get_host_tool('bwrap')]
          for k, v in env.items():
              bwrap_command += ['--setenv', k, v]

doc/source/format_declaring.rst

@@ -294,8 +294,10 @@ can be viewed in detail in the :ref:`builtin public data <public_builtin>` secti
  Sandbox
  ~~~~~~~
  Configuration for the build sandbox (other than :ref:`environment variables <format_environment>`)
 -can be placed in the ``sandbox`` configuration. At present, only the
 -UID and GID used by the user in the group can be specified.
 +can be placed in the ``sandbox`` configuration. The UID and GID used by the user
 +in the group can be specified, as well as the desired OS and machine
 +architecture. Possible machine architecture follow the same list as specified in
 +the :ref:`architecture option <project_options_arch>`.
  .. code:: yaml
@@ -311,6 +313,23 @@ you can supply a different uid or gid for the sandbox. Only
  bwrap-style sandboxes support custom user IDs at the moment, and hence
  this will only work on Linux host platforms.
 +.. code:: yaml
++
 +   # Specify build OS and architecture
 +   sandbox:
 +     build-os: AIX
 +     build-arch: Power-ISA-BE
++
 +When building locally, if these don't match the host machine then generally the
 +build will fail. The exception is when the OS is Linux and the architecture
 +specifies an ``x86-32`` build on an ``x86-64`` machine, in which case the
 +``linux32`` command is prepended to the bubblewrap command.
++
 +When building remotely, the OS and architecture are added to the ``Platform``
 +field in the ``Command`` uploaded. Whether this actually results in a building
 +the element for the desired OS and architecture is dependent on the server
 +having implemented these options the same as buildstream.
++
  .. note::
     The ``sandbox`` configuration is available since :ref:`format version 6 <project_format_version>`

doc/source/format_project.rst

@@ -552,9 +552,22 @@ exported as a comma separated list of selected value strings.
  Architecture
  ~~~~~~~~~~~~
 -The ``arch`` option type is special enumeration option which
 -defaults to the result of `uname -m`, and does not support
 -assigning any default in the project configuration.
 +The ``arch`` option type is a special enumeration option which defaults via
 +`uname -m` results to the following list.
++
 +* AArch32
 +* AArch64
 +* AArch64-BE
 +* Power-ISA-BE
 +* Power-ISA-LE
 +* Sparc-V9
 +* x86-32
 +* x86-64
++
 +The reason for this, opposed to using just `uname -m`, is that we want an
 +OS-independent list, as well as several results mapping to the same architecture
 +(e.g. i386, i486 etc. are all x86-32). It does not support assigning any default
 +in the project configuration.
  .. code:: yaml
@@ -563,16 +576,40 @@ assigning any default in the project configuration.
         type: arch
         description: The machine architecture
         values:
 -       - arm
 -       - aarch64
 -       - i386
 -       - x86_64
 +       - AArch32
 +       - AArch64
 +       - x86-32
 +       - x86-64
  Architecture options can be tested with the same expressions
  as other Enumeration options.
 +.. _project_options_os:
++
 +OS
 +~~
++
 +The ``os`` option type is a special enumeration option, which defaults to the
 +results of `uname -s`. It does not support assigning any default in the project
 +configuration.
++
 +.. code:: yaml
++
 +    options:
 +      machine_os:
 +        type: os
 +        description: The machine OS
 +        values:
 +        - Linux
 +        - SunOS
 +        - Darwin
 +        - FreeBSD
++
 +Os options can be tested with the same expressions as other Enumeration options.
++
++
  .. _project_options_element_mask:
  Element mask

[Notes] [Git][BuildStream/buildstream][raoul/775-execution-environment-reqs] 4 commits: Sandbox: use linux32 for x86-32 builds on x86-64 machines

Raoul Hidalgo Charman pushed to branch raoul/775-execution-environment-reqs at BuildStream / buildstream

Commits:

4 changed files:

Changes: