[Notes] [Git][BuildStream/buildstream][juerg/symlinks] 9 commits: .gitla

Jürg Billeter pushed to branch juerg/symlinks at BuildStream / buildstream

Commits:

da1560e4

by Javier Jardón at 2019-02-09T09:10:01Z

.gitlab-ci.yml: Remove testing on Fedora 27

Fedora 27 is EOL since 2018-11-30

See https://fedoraproject.org/wiki/End_of_life

79fbab61

by Jürg Billeter at 2019-02-09T10:30:20Z

Merge branch 'jjardon/fedora_27' into 'master'

.gitlab-ci.yml: Remove testing on Fedora 27

See merge request BuildStream/buildstream!1136

354c563d
by Jürg Billeter at 2019-02-10T20:03:46Z
```
import.py: Validate config node

Fixes #662.
```

5e1be71f

by Jürg Billeter at 2019-02-11T05:10:56Z

Merge branch 'juerg/import' into 'master'

import.py: Validate config node

Closes #662

See merge request BuildStream/buildstream!1141

99e1be45

by Jürg Billeter at 2019-02-11T05:12:25Z

local.py: Do not follow symlinks in local directories

isdir() follows symlinks on the host, resulting in potential host
contamination. This change reorders the file checks to avoid this issue.

f95e222e

by Jürg Billeter at 2019-02-11T05:12:25Z

sandbox/sandbox.py: Do not follow symlinks in _has_command()

This is required to ensure symlinks are not resolved on the host.

89973fb3

by Jürg Billeter at 2019-02-11T05:12:25Z

utils.py: Remove list_dirs parameter from list_relative_paths()

list_dirs was always True in the BuildStream code base. There was also a
bug in the list_dirs=False code path as it did not return symlinks in
`dirnames`.

This is an API break, however, there are no known external callers.

d1da3fb0

by Jürg Billeter at 2019-02-11T05:12:25Z

utils.py: Fix sorting of symlinks to directories

os.walk() resolves symlinks to check whether they point to a directory
even when followlinks is set to False. We already work around that
broken behavior by extracting symlinks from `dirnames`. However, the
sort order was still incorrect as we returned symlinks in dirnames
before files and other symlinks. This change fixes this, sorting all
files and symlinks in a single list.

7cf67ed3

by Jürg Billeter at 2019-02-11T05:12:25Z

_casbaseddirectory.py: Do not mimic os.walk() in list_relative_paths()

This matches the change in utils.list_relative_paths() that now sorts
all symlinks as files, instead of following the broken behavior of
os.walk().

6 changed files:

.gitlab-ci.yml
buildstream/plugins/elements/import.py
buildstream/plugins/sources/local.py
buildstream/sandbox/sandbox.py
buildstream/storage/_casbaseddirectory.py
buildstream/utils.py

Changes:

.gitlab-ci.yml

@@ -49,10 +49,6 @@ tests-debian-9:
    image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-debian:9-master-46405991
    <<: *tests
 -tests-fedora-27:
 -  image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:27-master-46405991
 -  <<: *tests
+-
  tests-fedora-28:
    image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:28-master-46405991
    <<: *tests
@@ -95,7 +91,7 @@ overnight-fedora-28-aarch64:
  tests-unix:
    # Use fedora here, to a) run a test on fedora and b) ensure that we
    # can get rid of ostree - this is not possible with debian-8
 -  image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:27-master-46405991
 +  image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:28-master-46405991
    <<: *tests
    variables:
      BST_FORCE_BACKEND: "unix"
@@ -291,7 +287,6 @@ coverage:
      - cp -a .coverage-reports/ ./coverage-report
    dependencies:
    - tests-debian-9
 -  - tests-fedora-27
    - tests-fedora-28
    - tests-fedora-missing-deps
    - tests-ubuntu-18.04

buildstream/plugins/elements/import.py

@@ -42,6 +42,10 @@ class ImportElement(Element):
      BST_VIRTUAL_DIRECTORY = True
      def configure(self, node):
 +        self.node_validate(node, [
 +            'source', 'target'
 +        ])
++
          self.source = self.node_subst_member(node, 'source')
          self.target = self.node_subst_member(node, 'target')

buildstream/plugins/sources/local.py

@@ -97,7 +97,7 @@ class LocalSource(Source):
          with self.timed_activity("Staging local files at {}".format(self.path)):
              if os.path.isdir(self.fullpath):
 -                files = list(utils.list_relative_paths(self.fullpath, list_dirs=True))
 +                files = list(utils.list_relative_paths(self.fullpath))
                  utils.copy_files(self.fullpath, directory, files=files)
              else:
                  destfile = os.path.join(directory, os.path.basename(self.path))
@@ -133,11 +133,11 @@ def unique_key(filename):
      # Return some hard coded things for files which
      # have no content to calculate a key for
 -    if os.path.isdir(filename):
 -        return "0"
 -    elif os.path.islink(filename):
 +    if os.path.islink(filename):
          # For a symbolic link, use the link target as its unique identifier
          return os.readlink(filename)
 +    elif os.path.isdir(filename):
 +        return "0"
      return utils.sha256sum(filename)

buildstream/sandbox/sandbox.py

@@ -525,11 +525,11 @@ class Sandbox():
      #         (bool): Whether a command exists inside the sandbox.
      def _has_command(self, command, env=None):
          if os.path.isabs(command):
 -            return os.path.exists(os.path.join(
 +            return os.path.lexists(os.path.join(
                  self._root, command.lstrip(os.sep)))
          for path in env.get('PATH').split(':'):
 -            if os.path.exists(os.path.join(
 +            if os.path.lexists(os.path.join(
                      self._root, path.lstrip(os.sep), command)):
                  return True

buildstream/storage/_casbaseddirectory.py

@@ -795,24 +795,11 @@ class CasBasedDirectory(Directory):
          Return value: List(str) - list of all paths
          """
 -        symlink_list = filter(lambda i: isinstance(i[1].pb_object, remote_execution_pb2.SymlinkNode),
 -                              self.index.items())
 -        file_list = list(filter(lambda i: isinstance(i[1].pb_object, remote_execution_pb2.FileNode),
 +        file_list = list(filter(lambda i: not isinstance(i[1].buildstream_object, CasBasedDirectory),
                                  self.index.items()))
          directory_list = filter(lambda i: isinstance(i[1].buildstream_object, CasBasedDirectory),
                                  self.index.items())
 -        # We need to mimic the behaviour of os.walk, in which symlinks
 -        # to directories count as directories and symlinks to file or
 -        # broken symlinks count as files. os.walk doesn't follow
 -        # symlinks, so we don't recurse.
 -        for (k, v) in sorted(symlink_list):
 -            target = self._resolve(k, absolute_symlinks_resolve=True)
 -            if isinstance(target, CasBasedDirectory):
 -                yield os.path.join(relpath, k)
 -            else:
 -                file_list.append((k, v))
+-
          if file_list == [] and relpath != "":
              yield relpath
          else:

buildstream/utils.py

@@ -111,7 +111,7 @@ class FileListResult():
          return ret
 -def list_relative_paths(directory, *, list_dirs=True):
 +def list_relative_paths(directory):
      """A generator for walking directory relative paths
      This generator is useful for checking the full manifest of
@@ -125,13 +125,26 @@ def list_relative_paths(directory, *, list_dirs=True):
      Args:
         directory (str): The directory to list files in
 -       list_dirs (bool): Whether to list directories
      Yields:
         Relative filenames in `directory`
      """
      for (dirpath, dirnames, filenames) in os.walk(directory):
 +        # os.walk does not decend into symlink directories, which
 +        # makes sense because otherwise we might have redundant
 +        # directories, or end up descending into directories outside
 +        # of the walk() directory.
 +        #
 +        # But symlinks to directories are still identified as
 +        # subdirectories in the walked `dirpath`, so we extract
 +        # these symlinks from `dirnames` and add them to `filenames`.
 +        #
 +        for d in dirnames:
 +            fullpath = os.path.join(dirpath, d)
 +            if os.path.islink(fullpath):
 +                filenames.append(d)
++
          # Modifying the dirnames directly ensures that the os.walk() generator
          # allows us to specify the order in which they will be iterated.
          dirnames.sort()
@@ -143,25 +156,10 @@ def list_relative_paths(directory, *, list_dirs=True):
          # `directory`, prefer to have no prefix in that case.
          basepath = relpath if relpath != '.' and dirpath != directory else ''
 -        # os.walk does not decend into symlink directories, which
 -        # makes sense because otherwise we might have redundant
 -        # directories, or end up descending into directories outside
 -        # of the walk() directory.
 -        #
 -        # But symlinks to directories are still identified as
 -        # subdirectories in the walked `dirpath`, so we extract
 -        # these symlinks from `dirnames`
 -        #
 -        if list_dirs:
 -            for d in dirnames:
 -                fullpath = os.path.join(dirpath, d)
 -                if os.path.islink(fullpath):
 -                    yield os.path.join(basepath, d)
+-
          # We've decended into an empty directory, in this case we
          # want to include the directory itself, but not in any other
          # case.
 -        if list_dirs and not filenames:
 +        if not filenames:
              yield relpath
          # List the filenames in the walked directory

[Notes] [Git][BuildStream/buildstream][juerg/symlinks] 9 commits: .gitlab-ci.yml: Remove testing on Fedora 27

Jürg Billeter pushed to branch juerg/symlinks at BuildStream / buildstream

Commits:

6 changed files:

Changes: