[Notes] [Git][BuildStream/buildstream][willsalmon/versionTagRegrex] 11 c

Will Salmon pushed to branch willsalmon/versionTagRegrex at BuildStream / buildstream

Commits:

3782c15f

by Ed Baunton at 2018-08-04T08:54:58Z

element.py (docs): dashes not underscores for build and install root

Fix small error in docs in relation to `get_variable` documentation.

f918205a

by Tristan Van Berkom at 2018-08-04T10:08:21Z

Merge branch 'edbaunton/doc-typo' into 'master'

element.py (docs): dashes not underscores for build and install root

See merge request BuildStream/buildstream!577

f69a0287

by Mathieu Bridon at 2018-08-04T21:02:10Z

doc: Build the docs without Buildstream installed

bc63f278

by Tristan Van Berkom at 2018-08-05T10:07:46Z

__main__.py: Adding private entry point so we can run BuildStream without installing

This is needed to build docs without installing BuildStream.

1618f405

by Tristan Van Berkom at 2018-08-05T10:07:46Z

doc: Fix junctions user guide to refer to doc/sessions, not doc/sessions-stored.

9e573eeb

by Tristan Van Berkom at 2018-08-05T10:07:46Z

doc: Some modifications to the bst2html program

  o Added feature to allow running shell commands

  o Removed unused codepaths to launch bst commands through the script,
    this program is anyway only used to parse and run the session files.

  o Fixed previous patch to run BuildStream uninstalled with
    the subprocess module, as running it in the same interpretor
    causes a bunch of problems.

19d2a072

by Tristan Van Berkom at 2018-08-05T10:07:46Z

doc/sessions/developing.run: Fix the session to actually run `patch` as a shell command

Since we added support for shell commands in bst2html, this can now be fixed.

This is a part of issue #553

9b937a97

by Tristan Van Berkom at 2018-08-05T10:07:46Z

HACKING.rst: It is not longer required to install BuildStream to build the docs.

63b33d07

by Tristan Van Berkom at 2018-08-05T10:57:03Z

Merge branch 'tristan/build-docs' into 'master'

Build documentation without installing

See merge request BuildStream/buildstream!605

e23555ab

by William Salmon at 2018-08-06T08:28:06Z

Catch Non Numeric versions

This patch just displays a better message than the default stack trace
but dose not try to fix the problem. A further patch will be created but
it effects versioneer so may take longer to land as it may need to go
via versioneer mainline.

636940fa

by William Salmon at 2018-08-06T08:28:06Z

Search for tags with the *.*.* patten for version

This has been done in a way that I hope might be able to be upstreamed
in to versioneer. This is not garanteed but it would be good if it
could be.

10 changed files:

.gitlab-ci.yml
HACKING.rst
+ buildstream/__main__.py
buildstream/_version.py
buildstream/element.py
buildstream/utils.py
doc/Makefile
doc/bst2html.py
doc/sessions/developing.run
doc/source/advanced-features/junction-elements.rst

Changes:

.gitlab-ci.yml

@@ -143,7 +143,6 @@ docs:
    - pip3 install sphinx-click
    - pip3 install sphinx_rtd_theme
    - cd dist && ./unpack.sh && cd buildstream
 -  - pip3 install .
    - make BST_FORCE_SESSION_REBUILD=1 -C doc
    - cd ../..
    - mv dist/buildstream/doc/build/html public

HACKING.rst

@@ -261,9 +261,6 @@ using pip or some other mechanism::
    # Additional optional dependencies required
    pip3 install --user arpy
 -Furthermore, the documentation build requires that BuildStream itself
 -be installed, as it will be used in the process of generating its docs.
+-
  To build the documentation, just run the following::
    make -C doc

buildstream/__main__.py

 +##################################################################
 +#                      Private Entry Point                       #
 +##################################################################
 +#
 +# This allows running the cli when BuildStream is uninstalled,
 +# as long as BuildStream repo is in PYTHONPATH, one can run it
 +# with:
 +#
 +#    python3 -m buildstream [program args]
 +#
 +# This is used when we need to run BuildStream before installing,
 +# like when we build documentation.
 +#
 +if __name__ == '__main__':
 +    # pylint: disable=no-value-for-parameter
 +    from ._frontend.cli import cli
 +    cli()

buildstream/_version.py

@@ -43,6 +43,7 @@ def get_config():
      cfg.VCS = "git"
      cfg.style = "pep440"
      cfg.tag_prefix = ""
 +    cfg.tag_regex = "*.*.*"
      cfg.parentdir_prefix = "BuildStream-"
      cfg.versionfile_source = "buildstream/_version.py"
      cfg.verbose = False
@@ -215,7 +216,7 @@ def git_versions_from_keywords(keywords, tag_prefix, verbose):
  @register_vcs_handler("git", "pieces_from_vcs")
 -def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command):
 +def git_pieces_from_vcs(tag_prefix, tag_regex, root, verbose, run_command=run_command):
      """Get version from 'git describe' in the root of the source tree.
      This only gets called if the git-archive 'subst' keywords were *not*
@@ -237,7 +238,7 @@ def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command):
      # if there isn't one, this yields HEX[-dirty] (no NUM)
      describe_out, rc = run_command(GITS, ["describe", "--tags", "--dirty",
                                            "--always", "--long",
 -                                          "--match", "%s*" % tag_prefix],
 +                                          "--match", "%s%s*" % (tag_prefix, tag_regex)],
                                     cwd=root)
      # --long was added in git-1.5.5
      if describe_out is None:
@@ -505,7 +506,7 @@ def get_versions():
                  "date": None}
      try:
 -        pieces = git_pieces_from_vcs(cfg.tag_prefix, root, verbose)
 +        pieces = git_pieces_from_vcs(cfg.tag_prefix, cfg.tag_regex, root, verbose)
          return render(pieces, cfg.style)
      except NotThisMethod:
          pass

buildstream/element.py

@@ -369,8 +369,8 @@ class Element(Plugin):
          generated script is run:
          - All element variables have been exported.
 -        - The cwd is `self.get_variable('build_root')/self.normal_name`.
 -        - $PREFIX is set to `self.get_variable('install_root')`.
 +        - The cwd is `self.get_variable('build-root')/self.normal_name`.
 +        - $PREFIX is set to `self.get_variable('install-root')`.
          - The directory indicated by $PREFIX is an empty directory.
          Files are expected to be installed to $PREFIX.

buildstream/utils.py

@@ -484,7 +484,13 @@ def get_bst_version():
          raise UtilError("Your git repository has no tags - BuildStream can't "
                          "determine its version. Please run `git fetch --tags`.")
 -    return (int(versions[0]), int(versions[1]))
 +    try:
 +        return (int(versions[0]), int(versions[1]))
 +    except Exception:
 +        # We should try a mitigation hear, like falling back to a previous version.
 +        raise UtilError("Cannot detect numeric Major and Minor version numbers\n"
 +                        "Version: {} not in Interger.Interger.whatever format"
 +                        .format(__version__))
  @contextmanager

doc/Makefile

@@ -31,6 +31,9 @@ ifneq ($(strip $(BST_FORCE_SESSION_REBUILD)),)
  BST2HTMLOPTS = --force
  endif
 +# Help Python find buildstream and its plugins
 +PYTHONPATH=$(CURDIR)/..:$(CURDIR)/../buildstream/plugins
++
  .PHONY: all clean templates templates-clean sessions sessions-prep sessions-clean html devhelp
@@ -65,7 +68,6 @@ define plugin-doc-skeleton
  endef
 -# We set PYTHONPATH here because source/conf.py sys.modules hacks dont seem to help sphinx-build import the plugins
  all: html devhelp
  clean: templates-clean sessions-clean
@@ -103,7 +105,7 @@ sessions-prep:
+ #
  sessions: sessions-prep
  	for file in $(wildcard sessions/*.run); do		\
 -	    $(BST2HTML) $(BST2HTMLOPTS) --description $$file;	\
 +	    PYTHONPATH=$(PYTHONPATH) $(BST2HTML) $(BST2HTMLOPTS) $$file;	\
  	done
  sessions-clean:
@@ -114,7 +116,7 @@ sessions-clean:
+ #
  html devhelp: templates sessions
  	@echo "Building $@..."
 -	PYTHONPATH=$(CURDIR)/../buildstream/plugins \
 +	PYTHONPATH=$(PYTHONPATH) \
  	    $(SPHINXBUILD) -b $@ $(ALLSPHINXOPTS) "$(BUILDDIR)/$@" \
  	    $(wildcard source/*.rst) \
  	    $(wildcard source/tutorial/*.rst) \

doc/bst2html.py

@@ -204,7 +204,7 @@ def workdir(source_cache=None):
          yield (tempdir, bst_config_file, source_cache)
 -# run_command()
 +# run_bst_command()
+ #
  # Runs a command
+ #
@@ -216,10 +216,30 @@ def workdir(source_cache=None):
  # Returns:
  #    (str): The colorized combined stdout/stderr of BuildStream
+ #
 -def run_command(config_file, directory, command):
 -    click.echo("Running command in directory '{}': bst {}".format(directory, command), err=True)
 +def run_bst_command(config_file, directory, command):
 +    click.echo("Running bst command in directory '{}': bst {}".format(directory, command), err=True)
 -    argv = ['bst', '--colors', '--config', config_file] + shlex.split(command)
 +    argv = ['python3', '-m', 'buildstream', '--colors', '--config', config_file] + shlex.split(command)
 +    p = subprocess.Popen(argv, cwd=directory, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
 +    out, _ = p.communicate()
 +    return out.decode('utf-8').strip()
++
++
 +# run_shell_command()
 +#
 +# Runs a command
 +#
 +# Args:
 +#    directory (str): The project directory
 +#    command (str): A shell command
 +#
 +# Returns:
 +#    (str): The combined stdout/stderr of the shell command
 +#
 +def run_shell_command(directory, command):
 +    click.echo("Running shell command in directory '{}': {}".format(directory, command), err=True)
++
 +    argv = shlex.split(command)
      p = subprocess.Popen(argv, cwd=directory, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
      out, _ = p.communicate()
      return out.decode('utf-8').strip()
@@ -373,12 +393,18 @@ def run_session(description, tempdir, source_cache, palette, config_file, force)
          # Get the command string
          command_str = _yaml.node_get(command, str, 'command')
 +        # Check whether this is a shell command and not a bst command
 +        is_shell = _yaml.node_get(command, bool, 'shell', default_value=False)
++
          # Check if there is fake output
          command_fake_output = _yaml.node_get(command, str, 'fake-output', default_value=None)
          # Run the command, or just use the fake output
          if command_fake_output is None:
 -            command_out = run_command(config_file, directory, command_str)
 +            if is_shell:
 +                command_out = run_shell_command(directory, command_str)
 +            else:
 +                command_out = run_bst_command(config_file, directory, command_str)
          else:
              command_out = command_fake_output
@@ -414,14 +440,8 @@ def run_session(description, tempdir, source_cache, palette, config_file, force)
  @click.option('--palette', '-p', default='tango',
                type=click.Choice(['solarized', 'solarized-xterm', 'tango', 'xterm', 'console']),
                help="Selects a palette for the output style")
 -@click.option('--output', '-o',
 -              type=click.Path(file_okay=True, dir_okay=False, writable=True),
 -              help="A file to store the output")
 -@click.option('--description', '-d',
 -              type=click.Path(file_okay=True, dir_okay=False, readable=True),
 -              help="A file describing what to do")
 -@click.argument('command', type=click.STRING, nargs=-1)
 -def run_bst(directory, force, source_cache, description, palette, output, command):
 +@click.argument('description', click.Path(file_okay=True, dir_okay=False, readable=True))
 +def run_bst(directory, force, source_cache, description, palette):
      """Run a bst command and capture stdout/stderr in html
      This command normally takes a description yaml file, see the HACKING
@@ -430,45 +450,8 @@ def run_bst(directory, force, source_cache, description, palette, output, comman
      if not source_cache and os.environ.get('BST_SOURCE_CACHE'):
          source_cache = os.environ['BST_SOURCE_CACHE']
 -    if output is not None and not check_needs_build(None, output, force=force):
 -        click.echo("No need to rebuild {}".format(output))
 -        return 0
+-
      with workdir(source_cache=source_cache) as (tempdir, config_file, source_cache):
+-
 -        if description:
 -            run_session(description, tempdir, source_cache, palette, config_file, force)
 -            return 0
+-
 -        # Run a command specified on the CLI
 -        #
 -        if not directory:
 -            directory = os.getcwd()
 -        else:
 -            directory = os.path.abspath(directory)
 -            directory = os.path.realpath(directory)
+-
 -        if not command:
 -            command = []
 -        command_str = " ".join(command)
+-
 -        # Run the command
 -        #
 -        command_out = run_command(config_file, directory, command_str)
+-
 -        # Generate a nice html div for this output
 -        #
 -        converted = generate_html(command_out, directory, config_file,
 -                                  source_cache, tempdir, palette,
 -                                  command_str)
+-
 -    if output is None:
 -        click.echo(converted)
 -    else:
 -        outdir = os.path.dirname(output)
 -        os.makedirs(outdir, exist_ok=True)
 -        with open(output, 'wb') as f:
 -            f.write(converted.encode('utf-8'))
 +        run_session(description, tempdir, source_cache, palette, config_file, force)
      return 0

doc/sessions/developing.run

@@ -16,7 +16,8 @@ commands:
  # Apply a patch in the workspace
  - directory: ../examples/developing/
 -  command: show hello.bst; patch workspace_hello/files/src/hello.c update.patch;
 +  shell: True
 +  command: patch workspace_hello/hello.c update.patch
  # Rebuild
  - directory: ../examples/developing/

doc/source/advanced-features/junction-elements.rst

@@ -41,7 +41,7 @@ This element consists of a script which calls hello.bst's hello command.
  Building callHello.bst,
  .. raw:: html
 -   :file: ../sessions-stored/junctions-build.html
 +   :file: ../sessions/junctions-build.html
  You can see that the hello.bst element and it's dependencies from the autotools
  project have been build as part of the pipeline for callHello.bst.
@@ -49,17 +49,18 @@ project have been build as part of the pipeline for callHello.bst.
  We can now invoke `bst shell`
  .. raw:: html
 -   :file: ../sessions-stored/junctions-shell.html
 +   :file: ../sessions/junctions-shell.html
  This runs the script files/callHello.sh which will makes use of the hello command from the hello.bst element in the autotools project.
++
  Cross-junction workspaces
  -------------------------
  You can open workspaces for elements in the project refered to by the junction
  using the syntax `bst open ${junction-name}:{element-name}`. In this example,
  .. raw:: html
 -   :file: ../sessions-stored/junctions-workspace-open.html
 +   :file: ../sessions/junctions-workspace-open.html
  This has opened a workspace for the hello.bst element from the autotools project.
  This workspace can now be used as normal.

[Notes] [Git][BuildStream/buildstream][willsalmon/versionTagRegrex] 11 commits: element.py (docs): dashes not underscores for build and install root

Will Salmon pushed to branch willsalmon/versionTagRegrex at BuildStream / buildstream

Commits:

10 changed files:

Changes: