[Notes] [Git][BuildStream/buildstream][richardmaw/artifact-log] 71 commits: _scheduler/queues: Don't call update state outside of error handling harness

[richardmaw-codethink <gitlab mg gitlab com>]

Title: GitLab

richardmaw-codethink pushed to branch richardmaw/artifact-log at BuildStream / buildstream

Commits:

• 98c15463
  by Tristan Van Berkom at 2018-12-05T07:04:56Z

  _scheduler/queues: Don't call update state outside of error handling harness
  
  Commit 3fa79d8da, part of an initiative for caching of the failed builds,
  introduced a call to Element._update_state() after a job completes and before
  entering the error handling harness intended for handling plugin raised errors.
  
  Element._update_state() can result in triggering plugin code to
  run, so this is incorrect, and causes raised errors to crash BuildStream
  if they happen here.
  
  After analyzing the code, it appears that this additional call to
  Element._update_state() is unneeded, and was only added because the
  state needs to be updated for a failure as well as a success.
  
  Instead, we now have the BuildQueue call Element._assemble_done()
  unconditionally, regardless of whether the build was successful
  or not, which has the same effect and also reads better.
  
  In addition, added a FIXME comment that we are still conditionally
  updating the artifact cache size from BuildQueue.done() only
  if the build is successful, which is incorrect because failed builds
  also increase the local artifact cache size - to fix this we need
  to communicate the added artifact size through Element._assemble()
  regardless of whether the build succeeded or failed.
  

• 5d77b871
  by Tristan Van Berkom at 2018-12-05T07:37:54Z

  Merge branch 'tristan/refactor-queues-update-state' into 'master'
  
  _scheduler/queues/queue.py: Don't call update state outside of error handling harness
  
  See merge request BuildStream/buildstream!988

• 82513109
  by Valentin David at 2018-12-05T09:47:52Z

  git source plugin: Track git tags and save them to reproduce a minimum shallow repository
  
  Instead of tag information being fetched which can change with time,
  they are tracked and saved in the projects.refs/.bst. Then we re-tag
  automatically the closest tag so that `git describe` works and is
  reproducible.
  
  This new feature is opt-in with the new `track-tags` configuration,
  and must be used to fix modules which are broken by our new policy
  of omitting the `.git/` repository when staging git sources.
  
  This fixes issue #487
  

• ec7b62c3
  by Valentin David at 2018-12-05T09:50:15Z

  NEWS: Add news about broken 'git describe' by default.
  

• 761e7570
  by Tristan Van Berkom at 2018-12-05T10:23:49Z

  Merge branch 'valentindavid/git_describe_tracking' into 'master'
  
  Track of git tags and save them to reproduce minimum shallow repository
  
  Closes #487
  
  See merge request BuildStream/buildstream!906

• 2f5966bd
  by Raoul Hidalgo Charman at 2018-12-05T10:37:10Z

  platform.py: Add mapping of uname to OS-independent values
  

• 1ad35fcd
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  _config.py: Use os and architecture settings
  
  In element, platform asks for host os and architecture to default to when
  SandboxConfig is initialised.
  This changes element cache keys so those have been updated in the tests.
  

• 9f75c274
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  optionarch.py: update to use same arch names as SandboxConfig
  
  Also update tests to be consistent with this
  

• a825096c
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  _sandboxremote.py: Add os and arch to command uploaded
  
  These are contained within the platform field.
  

• 8e09a81c
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  optionos.py: Add new option for OS configuration
  

• defec112
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  optionos.py: Add tests for OptionOS
  
  Same format as OptionArch tests, but with OS's.
  

• e5d8c7d8
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  _platform: Add checks for os and architecture
  

• b0603fb0
  by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

  Sandbox: use linux32 for x86-32 builds on x86-64 machines
  
    o _platform/linux.py: Add linux32 flag to send to sandbox bwrap when the
      build arch is x86-32 and the machines arch is x86-64 or similarly with
      aarch32 and aarch64.
    o sandbox/_sandboxbwrap.py: Use flag passed to start bwrap command with
      linux32 if set.
  

• f4d76996
  by Raoul Hidalgo Charman at 2018-12-05T14:58:49Z

  format_project.rst: Update documentation for changes to OptionArch
  

• 81163a79
  by Raoul Hidalgo Charman at 2018-12-05T14:58:53Z

  format_declaring.rst: Update documentation for sandbox options
  

• 23c27580
  by Raoul Hidalgo Charman at 2018-12-05T14:58:53Z

  format_project.rst: Add docs for OptionOS
  

• eb1ed410
  by Jürg Billeter at 2018-12-05T15:28:52Z

  Merge branch 'raoul/775-execution-environment-reqs' into 'master'
  
  Execution environment reqs
  
  Closes #775
  
  See merge request BuildStream/buildstream!969

• 9ef1a8a2
  by Raoul Hidalgo Charman at 2018-12-05T15:32:22Z

  utils.py: Add message digest function
  
  Get cascache.py to use this
  

• 4a56bf2a
  by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

  _sandboxremote.py: Add checks to action cache before attempting to push.
  
  Stops unneccesary pushing of builds that have already been built, just checks
  the action cache to begin with.
  
  Fixes #628
  

• 3641b6f6
  by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

  _sandboxremote.py: Add action-cache-service endpoint
  
  The endpoint may not necessarily be the same as execution-service endpoint.
  
  Part of #628
  

• 9d77351f
  by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

  format_project.rst: Add docs for action-cache-service
  
  Part of #628
  

• a53d6d1f
  by Raoul Hidalgo Charman at 2018-12-05T16:36:00Z

  Merge branch 'raoul/628-RE-flow-optimisation' into 'master'
  
  Remote-execution client flow optimisation
  
  Closes #628
  
  See merge request BuildStream/buildstream!982

• 4688b2c9
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  tests/sources/git.py: Refactor ref-not-in-track test to use parameterization
  
  Instead of duplicating the whole test body, run it twice while
  checking for a warning or an error depending on the parameter.
  

• 03028c7f
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  types.py: Moving CoreWarnings to the types.py module
  
  This also ensures it is exposed via the main buildstream __init__.py
  file, fixes the imports of CoreWarnings, and adjusts the documentation
  links to point to the right place.
  

• 3b7f8df8
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  Source: Fix typo in documentation, and make it a link to `bst track`
  
  Now the commands have documentation linkability so we can link to
  them when referring to them anywhere in the docs. This reference
  was previously referring to a bogus non-existent `build-stream track`,
  maybe this dates way back to the days before we named the frontend `bst`.
  

• e5c0f067
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  git source plugin: Fixing documentation linking and typo
  
  Fixes the configurable warning part of this to properly link
  to the project.conf documentation describing what configurable
  warnings actually are.
  
  Also fix the plugin raise the actual configurable warning which
  it advertizes that it raises; which is 'inconsistent-submodule',
  not 'inconsistent-submodules'.
  

• 205c27d8
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  git source plugin: Rename symbol for consistency and readability
  
  Change 'INCONSISTENT_SUBMODULE' for 'WARN_INCONSISTENT_SUBMODULE' to
  make things a bit more readable and comprehensive.
  

• bea4d4f5
  by Tristan Van Berkom at 2018-12-06T09:37:50Z

  Merge branch 'tristan/misc-cleanup' into 'master'
  
  Misc cleanups
  
  See merge request BuildStream/buildstream!993

• b1d2f001
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  source.py: Add new delegate method validate_cache()
  
  This is guaranteed to be called only once for a given session once the
  sources are known to be Consistency.CACHED, if source tracking is enabled
  in the session for this source, then this will only be called if the
  sources become cached after tracking completes.
  

• ee7fc47f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  git source plugin: Implementing submodule warnings
  
   o Unlisted submodule warning
  
     Now the git plugin will issue a configurable warning if a submodule
     exists and is used (checking out the submodule is not disabled),
     but is not specified in the source configuration.
  
   o Invalid submodule warning
  
     Now the git source plugin will issue a warning if the configuration
     specified a submodule which does not exist in the underlying git repository.
  
  As a side effect, this patch also changes the flow control of the git
  plugin such that submodules which are explicitly set to not be
  checked out, are also not fetched but instead ignored completely.
  

• 5fc9a1da
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  git source plugin: Emmit the ref-not-in-track warning from validate_cache()
  
  Now that we have Source.validate_cache(), this is a better place to emmit
  the ref-not-in-track warning, since it will be emmitted at the earliest
  opportunity and not only at Source.stage() or Source.init_workspace().
  
  This also allows us to remove the `self.tracked` local state, and cleanup
  some convoluted calling paths, removing some unnecessary parameters from
  the usual codepaths and making the plugin overall more readable.
  

• 50cb2706
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/testutils/repo/git.py: Adding remove_path() helper
  
  A function for removing paths from a git repo, can also
  be used to remove submodules.
  

• f139ca8f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Testing the git:unlisted-submodule warning
  
   o Test that it is not triggered in show before fetch, because we
     don't know about the unlisted submodules yet
  
   o Test that it is triggered by a fetch command
  
   o Test that it is triggered by `show` after having completed a
     fetch command, since now we have the repository and know about
     the unlisted submodule
  
   o Test all of this under warning or error conditions (parameterized
     for fatal-warnings)
  

• afe1aa9f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Testing the git:invalid-submodule warning
  
     o Test that it is not triggered in show before fetch, because we
       don't know the submodules yet so we cannot know if they are
       valid or not.
  
     o Test that it is triggered by a fetch command
  
     o Test that it is triggered by `show` after having completed a
       fetch command, since now we have the repository and know which
       specified submodules are invalid
  
     o Test all of this under warning or error conditions (parameterized
       for fatal-warnings)
  

• 57300201
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Test unlisted submodules warning appearing after track
  

• a0297625
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Test invalid submodules warning appearing after track
  

• 60e96781
  by Tristan Van Berkom at 2018-12-06T15:14:37Z

  Merge branch 'tristan/submodule-warnings' into 'master'
  
  Implement submodule warnings
  
  See merge request BuildStream/buildstream!996

• 7fe55143
  by Jonathan Maw at 2018-12-06T15:48:12Z

  Fix bst source-checkout not working with open workspaces
  

• 2a6879a5
  by Jonathan Maw at 2018-12-06T17:46:33Z

  Merge branch 'jonathan/source-checkout-workspace' into 'master'
  
  Fix bst source-checkout not working with open workspaces
  
  Closes #807
  
  See merge request BuildStream/buildstream!997

• e61e6e50
  by Tristan Van Berkom at 2018-12-07T07:57:13Z

  _yaml.py: Added `allow_none` parameter to _yaml.node_get()
  
  This allows specifying whether or not the code believes a None
  value is acceptable if explicitly expressed to be None in the YAML.
  
  A previous patch 3ba544b80f9f268be8ffe62fc8589b30212ec4a2 changed
  the behavior to accept None for all values across the board, along
  with allowing explicitly setting the `default_value` to `None` which
  caused the code to be much more readable (this was the main motivation
  of the patch, but it had the side effect of allowing None for everything).
  
  In the majority of cases we load YAML however, it either has a value or
  it is not specified in the YAML, and None is hardly ever acceptable
  to be explicitly specified (it may be in the case you want to override
  something with nothing in the YAML, but it is the edge case).
  
  This had the side effect of causing unexpected crashes where the
  YAML specifies an empty dictionary for instance.
  
  Instead of forcing the caller to handle a possible None value, give
  the choice to the caller if they really want to accept a None value,
  and raise the LoadError otherwise.
  
  This fixes issue #803, which is a crash due to receival of an
  unexpected None value through _yaml.node_get()
  

• 504ecb3e
  by Tristan Van Berkom at 2018-12-07T07:57:56Z

  plugin.py: Added new `allow_none` parameter to Plugin.node_get_member(), defaulting to False.
  
  Allow plugins to conveniently leverage the new `allow_none` parameter,
  and have Plugin.node_get_member() raise a LoadError in the usual
  case where None is not an acceptable value to be explicitly set in
  the user provided YAML.
  

• 642ae4e8
  by Tristan Van Berkom at 2018-12-07T07:57:56Z

  tests/format/project.py: Added new regression test for empty dependency dictionaries
  
  This adds a regression test for issue #803
  

• 2a0676c3
  by Tristan Van Berkom at 2018-12-07T08:33:14Z

  Merge branch 'tristan/yaml-optionally-allow-none' into 'master'
  
  Only optionally allow None values in user provided YAML
  
  Closes #803
  
  See merge request BuildStream/buildstream!999

• 56c2c2cc
  by Javier Jardón at 2018-12-09T16:59:26Z

  buildstream/data/projectconfig.yaml: Remove default strip-binaries
  
  They are too specific to be included by default
  
  Recommendation is if you are building in Linux is to use the
  ones begin used in the freedesktop-sdk project, for example
  
  See #645
  

• dcb6b5e8
  by Javier Jardón at 2018-12-09T16:59:26Z

  NEWS: Update for removal of default strip-commands
  

• ae0afce5
  by Javier Jardón at 2018-12-09T17:08:16Z

  buildstream/buildelement.py: Document where to find strip commands for Linux
  

• 873b618c
  by Javier Jardón at 2018-12-10T09:49:52Z

  Merge branch 'jjardon/strip-binaries-removal' into 'master'
  
  Remove default strip-commands
  
  Closes #645
  
  See merge request BuildStream/buildstream!987

• f2fcc2f6
  by Valentin David at 2018-12-11T12:12:08Z

  Force updating tags when fetching git repository
  
  When using aliases there are multiple remotes used in the cache
  repository. When fetching, tags are not updated if the were previously
  fetched from a different remote. Commits that not in a branch and only
  tagged do not get fetched if the tag is not fetched.
  
  Fixes #812
  

• 717c10d1
  by Valentin David at 2018-12-11T12:44:56Z

  Merge branch 'valentindavid/git_force_fetch_tags' into 'master'
  
  Force updating tags when fetching git repository
  
  Closes #812
  
  See merge request BuildStream/buildstream!1000

• 7892287a
  by Jonathan Maw at 2018-12-11T12:56:32Z

  utils.py: Add a helper for searching upwards for files
  
  i.e. with a given directory and filename, check parent directories until
  either a directory with the filename is found, or you reach the root of
  the filesystem.
  
  This is a part of #222
  

• 67c7a58d
  by Jonathan Maw at 2018-12-11T12:56:32Z

  Create and store data inside projects when opening workspaces
  
  Changes to _context.py:
  * Context has been extended to contain a WorkspaceProjectCache, as there
    are times when we want to use it before a Workspaces can be initialised
    (looking up a WorkspaceProject to find the directory that the project is
    in)
  
  Changes to _stream.py:
  * Removed staging the elements from workspace_open() and workspace_reset()
  
  Changes in _workspaces.py:
  * A new WorkspaceProject contains all the information needed to refer back
    to a project from its workspace (currently this is the project path and
    the element used to create this workspace)
  * This is stored within a new WorkspaceProjectCache object, which keeps
    WorkspaceProjects around so they don't need to be loaded from disk
    repeatedly.
  * Workspaces has been extended to contain the WorkspaceProjectCache, and
    will use it when opening and closing workspaces.
  * Workspaces.create_workspace has been extended to handle the staging of
    the element into the workspace, in addition to creating the equivalent
    WorkspaceProject file.
  
  This is a part of #222
  

• 64836b18
  by Jonathan Maw at 2018-12-11T12:56:32Z

  _project.py: Rename _ensure_project_dir to _find_project_dir
  
  This is a part of #222
  

• 7deeb2c3
  by Jonathan Maw at 2018-12-11T12:56:32Z

  cli.py: Use utils' search upwards helper when searching for project.conf
  
  This is a part of #222
  

• 496f0ab7
  by Jonathan Maw at 2018-12-11T12:56:32Z

  _project.py: Find project from workspace if outside of a project
  
  This is a part of #222
  

• 921f2bcb
  by Jonathan Maw at 2018-12-11T12:56:32Z

  _workspaces.py: Do not include .bstproject.yaml in the cache key
  
  This is a part of #222
  

• f145a3e4
  by Jonathan Maw at 2018-12-11T12:56:32Z

  cli: Interactively warn if the user is trying to close the workspace they're using to load the project
  
  This involves changes in:
  * _stream.py:
    * Add the helper Stream.workspace_is_required()
  * userconfig.yaml:
    * Add a default value for prompt.really-workspace-close-project-inaccessible
  * _context.py:
    * Load the prompt 'really-workspace-close-project-inaccessible' from
      user config.
  * cli.py:
    * If buildstream is invoked interactively, prompt the user to confirm
      that they want to close the workspace they're using to load this
      project.
  
  This is a part of #222
  

• 494cb7c6
  by Jonathan Maw at 2018-12-11T14:12:54Z

  tests: Test bst commands from an external workspace
  
  This is a part of #222
  

• fd1c5c5a
  by Jonathan Maw at 2018-12-11T14:12:54Z

  NEWS: Add an entry for being able to run commands from a workspace
  
  This is a part of #222
  

• 4219a6b4
  by Jonathan Maw at 2018-12-11T16:00:55Z

  Merge branch 'jonathan/workspace-fragment-create' into 'master'
  
  Support invoking buildstream from a workspace outside a project
  
  See merge request BuildStream/buildstream!924

• e564251b
  by Jonathan Maw at 2018-12-11T16:15:19Z

  _stream.py: Fix ugly error when opening a workspace using a relative path
  

• 7cf83343
  by Jonathan Maw at 2018-12-11T16:15:19Z

  _context.py: Store the directory buildstream was invoked on
  

• a1dee91e
  by Jonathan Maw at 2018-12-11T16:15:19Z

  Make specifying elements optional in bst commands
  
  Known issues:
  * `bst shell` works, but `bst shell COMMANDS...` doesn't, because click
    has no way of separating optional args from variable-length args.
  * `bst checkout` and `bst source-checkout`'s usage strings mark LOCATION
    as an optional argument. Because click gets confused if there's an
    optional argument before a mandatory argument, I had to mark LOCATION
    as optional internally.
  * `bst workspace open` makes no sense with element being optional, so
    I skipped it.
  * `bst workspace close` will probably need to be revisited when multiple
    projects can own one workspace.
  * `bst workspace reset` will happily delete the directory you're
    currently in, requiring you to `cd $PWD` to see the contents of your
    directory.
    I could exclude the top-level directory of the workspace being
    deleted, but it is entirely valid to run workspace commands from deeper
    in the workspace.
  
  This is a part of #222
  

• 9b192015
  by Jonathan Maw at 2018-12-11T16:15:19Z

  tests: Add tests for guessing element names
  
  This is a part of #222
  

• e340371f
  by Jonathan Maw at 2018-12-11T16:15:19Z

  Add NEWS for guessing the element in commands
  
  This is a part of #222
  

• bbf81c3d
  by Jonathan Maw at 2018-12-11T16:15:19Z

  doc: Point out that element is optional when commands are run from the workspace
  
  This is a part of #222
  

• a5a53ddd
  by Jonathan Maw at 2018-12-12T13:44:09Z

  Merge branch 'jonathan/workspace-fragment-guess-element' into 'master'
  
  Guess the element when running commands in a workspace
  
  See merge request BuildStream/buildstream!954

• 49ad9f0f
  by Richard Maw at 2018-12-12T15:40:39Z

  cli: Add support for auto-completing artifact ref names
  

• a3dc79fb
  by Richard Maw at 2018-12-12T15:40:39Z

  cli: Add artifact command group
  

• 5e314cfc
  by Richard Maw at 2018-12-12T15:40:39Z

  cli: Add artifact log command
  

• f147afab
  by Richard Maw at 2018-12-12T15:40:39Z

  tests: Add tests for artifact group commands
  

• 20d855f4
  by Richard Maw at 2018-12-12T15:41:09Z

  NEWS: Mention `bst artifact log`
  

30 changed files:

• NEWS
• buildstream/__init__.py
• buildstream/_artifactcache/cascache.py
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_frontend/cli.py
• buildstream/_loader/loader.py
• buildstream/_options/optionarch.py
• + buildstream/_options/optionos.py
• buildstream/_options/optionpool.py
• buildstream/_platform/linux.py
• buildstream/_platform/platform.py
• buildstream/_platform/unix.py
• buildstream/_project.py
• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/queue.py
• buildstream/_stream.py
• buildstream/_versions.py
• buildstream/_workspaces.py
• buildstream/_yaml.py
• buildstream/buildelement.py
• buildstream/data/projectconfig.yaml
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/plugin.py
• buildstream/plugins/sources/git.py
• buildstream/sandbox/_config.py
• buildstream/sandbox/_sandboxbwrap.py
• buildstream/sandbox/_sandboxremote.py
• buildstream/source.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.