[Notes] [Git][BuildStream/buildstream][jonathan/workspace-fragment-multi-project] 52 commits: Add requirements files for install, test and plugin dependencies

[Jonathan Maw <gitlab mg gitlab com>]

Title: GitLab

Jonathan Maw pushed to branch jonathan/workspace-fragment-multi-project at BuildStream / buildstream

Commits:

• 1cbd9a73
  by Chandan Singh at 2019-01-02T19:39:58Z

  Add requirements files for install, test and plugin dependencies
  
  Add `.in` and `.txt` requirements files for BuildStream's pure python
  dependencies. For each pair, the `.in` file is supposed to capture the
  loose version requirements, and the corresponding `.txt` file is
  supposed to have frozen requirements. We have 3 such sets:
  
  * `requirements`: BuildStream's runtime dependencies
  * `dev-requirements`: Dependencies for running tests
  * `plugin-requirements`: Dependencies for core plugins
  
  Note that the frozen requirements files will only be used for testing
  purposes, and `setup.py` will continue to read loose requirements.
  

• 6b0cb5f3
  by Chandan Singh at 2019-01-02T19:41:21Z

  Add tox.ini to enable running tests using tox
  
  Add `tox.ini` file that will enable us to use
  [tox](https://tox.readthedocs.io/) as a frontend for running tests.
  Since we share the config via `setup.cfg` and requirements via
  requirements files, commands like `python3 setup.py test` will continue
  to work.
  

• 9d2d1d4f
  by Chandan Singh at 2019-01-03T03:31:52Z

  .gitlab-ci.yml: Run tests using tox
  
  Instead of invoking tests throung `setup.py`, use `tox` as a frontend in
  the CI pipelines.
  

• 8ae04283
  by Tristan Van Berkom at 2019-01-03T03:31:52Z

  CONTRIBUTING.rst: Updated to reflect running tests using tox.
  

• e7e2a5c1
  by Chandan Singh at 2019-01-03T03:31:52Z

  CONTRIBUTING.rst: Add steps for installing non-python build dependencies
  
  Since we don't allow use of site packages by default in our `tox`
  configuration, people will need to install non-python build dependencies
  of some of our dependencies that do not provide pre-built wheels. We
  have two such packages at the moment:
  
  * psuitl: requires python C headers, compiler
  * pygobject: requires python C headers, pkg-config, compiler, cairo
    headers, and gobject libraries
  

• afa0a369
  by Chandan Singh at 2019-01-03T03:31:52Z

  Move all requirements files into "tools" directory
  
  These new `.in` and `.txt` are making the repository look very
  cluttered. Move them to a separate `tools` directory to make it look a
  bit cleaner.
  

• 3fae3446
  by Chandan Singh at 2019-01-03T03:44:17Z

  Move sphinx build functionality to tox
  
  Currently the CI and the docs both have to duplicate the same inforation
  about how to gather dependencies etc, and have to use hacky ways to run
  them.
  
  Add a new `docs` environment to our tox setup so that building docs is
  as simple as running `tox -e docs`.
  

• 1f7bc655
  by Chandan Singh at 2019-01-03T03:44:17Z

  CONTRIBUTING.rst: Add instructions to recreate tox environment
  
  When we update our dependencies, developers will eventually need to
  recreate their `tox` environments to get correct results. This happens
  because `tox` isn't particularly good at recognizing changes in the
  requirements files.
  

• 669b55b0
  by Tristan Van Berkom at 2019-01-03T15:47:27Z

  Merge branch 'chandan/toxify' into 'master'
  
  Use tox as a frontend for running tests
  
  See merge request BuildStream/buildstream!1027

• ca547f19
  by Chandan Singh at 2019-01-03T19:34:49Z

  tests/frontend/buildcheckout.py: Fix bad filename issue for Windows
  
  In BuildStream/buildstream!1028, we added a test specifically to test
  that BuildStream correctly raises an warning when the name of an element
  contains characters that are invalid on Windows. Unfortunately, we
  didn't see it coming that it would make it impossible to checkout this
  branch on Windows.
  
  Fix it by generating this file, only if we are not running on Windows.
  
  * tests/testutils/site.py: Add `IS_WINDOWS` check
  * tests/frontend/buildcheckout.py: Generate file with invalid filename
    on the fly
  * Remove tests/frontend/project/elements/invalid-chars.
  
  Fixes #842.
  
  Note that this may still cause issues on WSL when running tests on a
  shared filesystem, but that seems to be a generic issue on WSL with
  `os.rename`.
  

• 01c3761c
  by Chandan Singh at 2019-01-03T20:03:32Z

  Merge branch 'chandan/fix-bad-filename-mini-disaster' into 'master'
  
  tests/frontend/buildcheckout.py: Fix bad filename issue for Windows
  
  Closes #842
  
  See merge request BuildStream/buildstream!1032

• 7850e4e2
  by Tristan Van Berkom at 2019-01-03T20:55:00Z

  setup.cfg: Specify the pycodestyle configurations separately from pytest configuration
  

• 3c9d592d
  by Tristan Van Berkom at 2019-01-03T20:55:00Z

  setup.cfg: Don't lint by default through setup.py test invocations
  

• feba48e7
  by Tristan Van Berkom at 2019-01-03T20:55:00Z

  tox.ini: Added new 'lint' environment to lint separately
  

• 9caa0ea0
  by Tristan Van Berkom at 2019-01-03T20:55:00Z

  .gitlab-ci.yml: Update the CI to run the linter separately through tox
  
  This should save us some cycles in CI, and also allow the
  developer to more conveniently lint separately from testing.
  

• 91c8b634
  by Tristan Van Berkom at 2019-01-03T21:04:49Z

  CONTRIBUTING.rst: Adding instructions to run the linter separately
  

• 892bf99e
  by Tristan Van Berkom at 2019-01-03T21:31:04Z

  tools/requirements.{in,txt}: Fix warning about duplicate jinja
  
  Apparently you need to spell Jinja2 with a capital J, otherwise
  we find it listed twice in the resulting requirements.txt files.
  

• 7c681c42
  by Tristan Van Berkom at 2019-01-03T21:32:40Z

  tools/dev-requirements.{in,txt}: Removed requirements on pylint/codestyle pytest plugins
  
  We no longer run the linters through pytest, so we don't need these
  plugins anymore.
  

• d6689ed2
  by Tristan Van Berkom at 2019-01-03T22:09:02Z

  Merge branch 'tristan/toxic-linting' into 'master'
  
  Run the linter separately from the tests
  
  See merge request BuildStream/buildstream!1033

• e15278d7
  by Thomas Coldrick at 2019-01-03T22:14:01Z

  Use collections.abc for Mapping, Iterable
  
  In _yaml.py and _frontend/complete.py we were getting pylint warnings
  for using collections.Mapping and collections.Iterable, which are
  abstract classes now provided from collections.abc. This patch just
  uses the classes from the right place.
  

• a85da591
  by Tristan Van Berkom at 2019-01-03T22:36:16Z

  Merge branch 'coldtom/collections' into 'master'
  
  Use collections.abc for Mapping, Iterable
  
  Closes #831
  
  See merge request BuildStream/buildstream!1020

• cf0e7d26
  by Jürg Billeter at 2019-01-04T07:07:59Z

  _frontend/cli.py: Reinstate support for guessing targets
  
  Moving fetch and track to the source command group accidentally dropped
  the support for guessing targets for these commands when invoked from a
  workspace directory. This brings it back.
  

• 0751fc8f
  by Jürg Billeter at 2019-01-04T11:27:58Z

  Merge branch 'juerg/source-guess-target' into 'master'
  
  _frontend/cli.py: Reinstate support for guessing targets
  
  See merge request BuildStream/buildstream!1036

• ebd965fe
  by Chandan Singh at 2019-01-04T15:21:35Z

  Rename "tools" directory to "requirements"
  
  This seems like a better name for the directory, as it more closely
  describes the purpose of its contents.
  

• bb712886
  by Chandan Singh at 2019-01-04T15:21:35Z

  Add Makefile to update requirements files
  

• b9792837
  by Chandan Singh at 2019-01-04T15:21:35Z

  CONTRIBUTING.rst: Add instructions to update requirements files
  
  Split the "The MANIFEST.in and setup.py" section in two:
  "Managing data files" and "Updating BuildStream's Python dependencies".
  
  Briefly explain the layout of `requirements` directory and add
  instructions to use the Makefile added in the last commit.
  

• 45692335
  by Tristan Van Berkom at 2019-01-04T15:50:14Z

  Merge branch 'chandan/update-requirements-one-liner' into 'master'
  
  Add Makefile to update requirements files
  
  See merge request BuildStream/buildstream!1035

• f42dcb54
  by Chandan Singh at 2019-01-04T17:16:43Z

  .gitlab-ci.yml: Remove prepare stage
  
  As we now run tests using `tox`, we don't need to worry about manually
  packing and unpacking BuildStream. So, we can remove the preapre stage
  entirely.
  
  Update `coverage` and nightly jobs to appropriately cope with this
  change. Both these jobs now install all runtime dependencies from
  requirements files.
  

• ee29f207
  by Chandan Singh at 2019-01-04T17:41:02Z

  Merge branch 'chandan/no-sdist' into 'master'
  
  .gitlab-ci.yml: Remove prepare stage
  
  See merge request BuildStream/buildstream!1037

• f122cfe8
  by Jürg Billeter at 2019-01-06T17:57:33Z

  _frontend/app.py: Use buildtree for interactive shell on build failure
  
  Fixes: e29aea36 ("Basic options for shell --build to use buildtrees")
  

• 7dfb85b3
  by Tristan Van Berkom at 2019-01-06T18:31:25Z

  Merge branch 'juerg/shell-buildtree' into 'master'
  
  _frontend/app.py: Use buildtree for interactive shell on build failure
  
  See merge request BuildStream/buildstream!1039

• 891fcb0e
  by Tristan Van Berkom at 2019-01-07T16:47:01Z

  Fix stack traces discovered with ^C forceful termination.
  
    * utils.py:_kill_process_tree(): Ignore NoSuchProcess errors
  
      These are caused because we issue SIGTERM, and if the process
      has not exited after a timeout, we kill it.
  
    * _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here
      redundantly, they are already ignored.
  
  It seems that we were ignoring it after sleeping when terminating
  tasks from the scheduler... but we were not ignoring it when performing
  the same pattern in the `Plugin.call()` -> `utils._call()` path, so
  we were still getting these exceptions at termination time from host
  tool processes launched by source plugins.
  

• 5de42d43
  by Tristan Van Berkom at 2019-01-07T18:00:37Z

  Merge branch 'tristan/keyboard-interrupt-stack-trace' into 'master'
  
  Fix stack traces discovered with ^C forceful termination.
  
  See merge request BuildStream/buildstream!1043

• 059035b9
  by Tristan Van Berkom at 2019-01-07T18:02:00Z

  _scheduler/scheduler.py: Make _schedule_jobs() private
  
  This is not used anywhere outside of the Scheduler, currently
  only the Scheduler itself is allowed to queue a job at this level.
  
  If the highlevel business logic for automatic queueing of auxiliary
  jobs moves to another location, we can make this public again.
  

• b83d1b1f
  by Tristan Van Berkom at 2019-01-07T18:02:00Z

  _scheduler/scheduler.py: Only run one cache size job at a time
  
  When queuing the special cache management related cleanup and
  cache size jobs, we now treat these jobs as special and do the
  following:
  
    * Avoid queueing a cleanup/cache_size job if one is already queued
  
      We just drop redundantly queued jobs here.
  
    * Ensure that jobs of this type only run one at a time
  
      This could have been done with the Resources mechanics,
      however as these special jobs have the same properties and
      are basically owned by the Scheduler, it seemed more straight
      forward to handle the behaviors of these special jobs together.
  
  This fixes issue #753
  

• 16a8816f
  by Tristan Van Berkom at 2019-01-07T18:02:00Z

  Scheduler: Introduced JobStatus instead of simple success boolean
  
  This changes the deepest callback from when a Job completes to
  propagate a JobStatus value instead of a simple boolean, and updates
  all of the effected code paths which used to receive a boolean
  to now handle the JobStatus values.
  
  This further improves the situation for issue #753, as now we avoid
  queueing cache size jobs for pull jobs which are skipped.
  

• c2fc2a5e
  by Tristan Van Berkom at 2019-01-07T18:02:00Z

  _scheduler/jobs/job.py: Removed 'skipped' property
  
  This is redundant now that we report it through the JobStatus.
  

• 3e3984ad
  by Tristan Van Berkom at 2019-01-07T18:50:23Z

  Merge branch 'tristan/one-cache-size-job' into 'master'
  
  Only queue one cache size job
  
  Closes #753
  
  See merge request BuildStream/buildstream!1040

• 512c726e
  by Tristan Van Berkom at 2019-01-08T03:38:11Z

  sandbox/sandbox.py: Fix regression of command logging
  
  Since we added batch commands, the batch commands print the
  text of the commands directly in the message text, but this is wrong.
  
  The detail string is the appropriate place for text of unknown lengths
  (the user can actually configure how many max lines of commands they
  want to see in their log), the message text itself should be controlled
  and brief enough to avoid text wrapping.
  

• 01171988
  by Tristan Van Berkom at 2019-01-08T04:20:14Z

  Merge branch 'tristan/fix-command-status-messages' into 'master'
  
  sandbox/sandbox.py: Fix regression of command logging
  
  See merge request BuildStream/buildstream!1044

• 6c1d06d6
  by Phil Dawson at 2019-01-08T10:24:32Z

  element.py: remove reference to source bundle command
  
  This command has been replacved by the bst source checkout command
  

• 914ecb72
  by Jürg Billeter at 2019-01-08T10:54:02Z

  Merge branch 'phil/remove-source-bundle-reference' into 'master'
  
  element.py: remove documentation reference to source bundle command
  
  See merge request BuildStream/buildstream!1041

• 81f7a964
  by Tristan Van Berkom at 2019-01-08T16:55:58Z

  sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.
  
  When code is faulty, in a plugin or in the core, we should
  get a stack trace and a BUG message.
  

• edcc43ed
  by Tristan Van Berkom at 2019-01-08T19:19:10Z

  Merge branch 'tristan/sandbox-programming-error' into 'master'
  
  sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.
  
  See merge request BuildStream/buildstream!1046

• 17c5ca2d
  by Valentin David at 2019-01-09T10:45:23Z

  Use relative path to project directory for remote execution certificates/keys
  

• 605f8d11
  by Valentin David at 2019-01-09T10:45:23Z

  Add support for user remote execution configuration
  
  Fixes #631.
  

• 65ea03ab
  by Valentin David at 2019-01-09T10:45:23Z

  Add support for https channel to remote execution and actions servers
  
  Fixes #780.
  

• ba05c4fa
  by Valentin David at 2019-01-09T12:58:59Z

  Merge branch 'valentindavid/remote_execution_configuration' into 'master'
  
  Remote execution configuration: HTTPS and user configuration
  
  Closes #780 and #631
  
  See merge request BuildStream/buildstream!1030

• f790b131
  by Jonathan Maw at 2019-01-09T13:04:39Z

  tests: Add a test for multiple projects opening one workspace
  
  This change is related to #222
  

• c25ce736
  by Jonathan Maw at 2019-01-09T13:04:39Z

  Implement basic handling of multiple projects to a single workspace
  
  This change is related to #222
  

• 0eb8ef81
  by Jonathan Maw at 2019-01-09T13:04:39Z

  WIP: Extend tests to cover repeat opening and closing of workspaces
  
  Opening fails, and doing it right is pretty damn complicated.
  

• 6f2066c8
  by Jonathan Maw at 2019-01-09T13:04:39Z

  test: start adding tests for multiple elements having the same workspace
  

30 changed files:

• .gitignore
• .gitlab-ci.yml
• CONTRIBUTING.rst
• MANIFEST.in
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_frontend/cli.py
• buildstream/_frontend/complete.py
• buildstream/_project.py
• buildstream/_scheduler/__init__.py
• buildstream/_scheduler/jobs/__init__.py
• buildstream/_scheduler/jobs/cachesizejob.py
• buildstream/_scheduler/jobs/cleanupjob.py
• buildstream/_scheduler/jobs/elementjob.py
• buildstream/_scheduler/jobs/job.py
• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/fetchqueue.py
• buildstream/_scheduler/queues/pullqueue.py
• buildstream/_scheduler/queues/queue.py
• buildstream/_scheduler/queues/trackqueue.py
• buildstream/_scheduler/scheduler.py
• buildstream/_stream.py
• buildstream/_workspaces.py
• buildstream/_yaml.py
• buildstream/element.py
• buildstream/sandbox/_sandboxremote.py
• buildstream/sandbox/sandbox.py
• buildstream/utils.py
• doc/source/format_project.rst
• doc/source/using_config.rst

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.