GNOME.org
 

[Notes] [Git][BuildStream/buildstream][chandan/fix-bst-docker-import-logging] 10 commits: CONTRIBUTING.rst: Updating CONTRIBUTING guidelines

Title: GitLab

Chandan Singh pushed to branch chandan/fix-bst-docker-import-logging at BuildStream / buildstream

Commits:

2 changed files:

Changes:

  • CONTRIBUTING.rst
    ... ... @@ -3,72 +3,137 @@ Contributing
    3 3 
     Some tips and guidelines for developers hacking on BuildStream
    4 4
    5 5
    6 
    -Feature additions
    7 
    ------------------
    8 
    -Major feature additions should be proposed on the
    9 
    -`mailing list <https://mail.gnome.org/mailman/listinfo/buildstream-list>`_
    10 
    -before being considered for inclusion, we strongly recommend proposing
    11 
    -in advance of commencing work.
    6 
    +.. _contributing_filing_issues:
    7 
    +
    8 
    +Filing issues
    9 
    +-------------
    10 
    +If you are experiencing an issue with BuildStream, or would like to submit a patch
    11 
    +to fix an issue, then you should first search the list of `open issues <https://gitlab.com/BuildStream/buildstream/issues>`_
    12 
    +to see if the issue is already filed, and `open an issue <https://gitlab.com/BuildStream/buildstream/issues/new>`_
    13 
    +if no issue already exists.
    14 
    +
    15 
    +For policies on how to submit an issue and how to use our project labels,
    16 
    +we recommend that you read the `policies guide
    17 
    +<https://gitlab.com/BuildStream/nosoftware/alignment/blob/master/BuildStream_policies.md>`_
    18 
    +
    19 
    +
    20 
    +.. _contributing_fixing_bugs:
    21 
    +
    22 
    +Fixing bugs
    23 
    +-----------
    24 
    +Before fixing a bug, it is preferred that an :ref:`issue be filed <contributing_filing_issues>`
    25 
    +first in order to better document the defect, however this need not be followed to the
    26 
    +letter for minor fixes.
    27 
    +
    28 
    +Patches which fix bugs should always come with a regression test.
    29 
    +
    12 30
    13 
    -If you are experiencing an issue with BuildStream or would like to submit a small patch/feature, then
    14 
    -you can `oepn an issue <https://gitlab.com/BuildStream/buildstream/issues/new?issue%5Bassignee_id%5D=&issue%5Bmilestone_id%5D=>`_
    31 
    +.. _contributing_adding_features:
    15 32
    16 
    -For policies on how to submit and issue and how to use our project labels, we recommend that you read the `policies guide <https://gitlab.com/BuildStream/nosoftware/alignment/blob/master/BuildStream_policies.md>`_
    33 
    +Adding new features
    34 
    +-------------------
    35 
    +Feature additions should be proposed on the `mailing list
    36 
    +<https://mail.gnome.org/mailman/listinfo/buildstream-list>`_
    37 
    +before being considered for inclusion. To save time and avoid any frustration,
    38 
    +we strongly recommend proposing your new feature in advance of commencing work.
    39 
    +
    40 
    +Once consensus has been reached on the mailing list, then the proposing
    41 
    +party should :ref:`file an issue <contributing_filing_issues>` to track the
    42 
    +work. Please use the *bst_task* template for issues which represent
    43 
    +feature additions.
    17 44
    18 
    -New features must be well documented and tested either in our main
    19 
    -test suite if possible, or otherwise in the integration tests.
    45 
    +New features must be well documented and tested in our test suite.
    20 46
    21 47 
     It is expected that the individual submitting the work take ownership
    22 48 
     of their feature within BuildStream for a reasonable timeframe of at least
    23 49 
     one release cycle after their work has landed on the master branch. This is
    24 
    -to say that the submitter is expected to address and fix any side effects and
    25 
    -bugs which may have fell through the cracks in the review process, giving us
    26 
    -a reasonable timeframe for identifying these.
    50 
    +to say that the submitter is expected to address and fix any side effects,
    51 
    +bugs or regressions which may have fell through the cracks in the review
    52 
    +process, giving us a reasonable timeframe for identifying these.
    27 53
    28 54
    29 
    -Patch submissions
    30 
    ------------------
    31 
    -If you want to submit a patch, do ask for developer permissions on our
    32 
    -IRC channel first (GitLab's button also works, but you may need to
    33 
    -shout about it - we often overlook this) - for CI reasons, it's much
    34 
    -easier if patches are in branches of the main repository.
    55 
    +.. _contributing_submitting_patches:
    56 
    +
    57 
    +Submitting patches
    58 
    +------------------
    35 59
    36 
    -Branches must be submitted as merge requests in gitlab. If the branch
    37 
    -fixes an issue or is related to any issues, these issues must be mentioned
    38 
    -in the merge request or preferably the commit messages themselves.
    39 60
    61 
    +Ask for developer access
    62 
    +~~~~~~~~~~~~~~~~~~~~~~~~
    63 
    +If you want to submit a patch, do ask for developer permissions, either
    64 
    +by asking us directly on our public IRC channel (irc://irc.gnome.org/#buildstream)
    65 
    +or by visiting our `project page on GitLab <https://gitlab.com/BuildStream/buildstream>`_
    66 
    +and using the GitLab UI to ask for permission.
    67 
    +
    68 
    +This will make your contribution experience smoother, as you will not
    69 
    +need to setup any complicated CI settings, and rebasing your branch
    70 
    +against the upstream master branch will be more painless.
    71 
    +
    72 
    +
    73 
    +Branch names
    74 
    +~~~~~~~~~~~~
    40 75 
     Branch names for merge requests should be prefixed with the submitter's
    41 
    -name or nickname, e.g. ``username/implement-flying-ponies``.
    76 
    +name or nickname, followed by a forward slash, and then a descriptive
    77 
    +name. e.g.::
    42 78
    43 
    -You may open merge requests for the branches you create before you
    44 
    -are ready to have them reviewed upstream, as long as your merge request
    45 
    -is not yet ready for review then it must be prefixed with the ``WIP:``
    46 
    -identifier.
    79 
    +  username/fix-that-bug
    47 80
    81 
    +This allows us to more easily identify which branch does what and
    82 
    +belongs to whom, especially so that we can effectively cleanup stale
    83 
    +branches in the upstream repository over time.
    84 
    +
    85 
    +
    86 
    +Merge requests
    87 
    +~~~~~~~~~~~~~~
    88 
    +Once you have created a local branch, you can push it to the upstream
    89 
    +BuildStream repository using the command line::
    90 
    +
    91 
    +  git push origin username/fix-that-bug:username/fix-that-bug
    92 
    +
    93 
    +GitLab will respond to this with a message and a link to allow you to create
    94 
    +a new merge request. You can also `create a merge request for an existing branch
    95 
    +<https://gitlab.com/BuildStream/buildstream/merge_requests/new>`_.
    96 
    +
    97 
    +You may open merge requests for the branches you create before you are ready
    98 
    +to have them reviewed and considered for inclusion if you like. Until your merge
    99 
    +request is ready for review, the merge request title must be prefixed with the
    100 
    +``WIP:`` identifier.
    101 
    +
    102 
    +
    103 
    +Organized commits
    104 
    +~~~~~~~~~~~~~~~~~
    48 105 
     Submitted branches must not contain a history of the work done in the
    49 
    -feature branch. Please use git's interactive rebase feature in order to
    50 
    -compose a clean patch series suitable for submission.
    106 
    +feature branch. For example, if you had to change your approach, or
    107 
    +have a later commit which fixes something in a previous commit on your
    108 
    +branch, we do not want to include the history of how you came up with
    109 
    +your patch in the upstream master branch.
    110 
    +
    111 
    +Please use git's interactive rebase feature in order to compose a clean
    112 
    +patch series suitable for submission upstream.
    113 
    +
    114 
    +Every commit in series should pass the test suite, this is very important
    115 
    +for tracking down regressions and performing git bisections in the future.
    51 116
    52 117 
     We prefer that documentation changes be submitted in separate commits from
    53 
    -the code changes which they document, and new test cases are also preferred
    54 
    -in separate commits.
    118 
    +the code changes which they document, and newly added test cases are also
    119 
    +preferred in separate commits.
    55 120
    56 121 
     If a commit in your branch modifies behavior such that a test must also
    57 122 
     be changed to match the new behavior, then the tests should be updated
    58 
    -with the same commit. Ideally every commit in the history of master passes
    59 
    -its test cases, this makes bisections more easy to perform, but is not
    60 
    -always practical with more complex branches.
    123 
    +with the same commit, so that every commit passes it's own tests.
    61 124
    62 125
    63 126 
     Commit messages
    64 127 
     ~~~~~~~~~~~~~~~
    65 
    -Commit messages must be formatted with a brief summary line, optionally
    66 
    -followed by an empty line and then a free form detailed description of
    67 
    -the change.
    128 
    +Commit messages must be formatted with a brief summary line, followed by
    129 
    +an empty line and then a free form detailed description of the change.
    68 130
    69 131 
     The summary line must start with what changed, followed by a colon and
    70 132 
     a very brief description of the change.
    71 133
    134 
    +If the commit fixes an issue, or is related to an issue; then the issue
    135 
    +number must be referenced in the commit message.
    136 
    +
    72 137 
     **Example**::
    73 138
    74 139 
       element.py: Added the frobnicator so that foos are properly frobbed.
    ... ... @@ -77,33 +142,459 @@ a very brief description of the change.
    77 142 
       the element. Elements that are not properly frobnicated raise
    78 143 
       an error to inform the user of invalid frobnication rules.
    79 144
    145 
    +  Fixes #123
    146 
    +
    147 
    +In the case that you have a commit which necessarily modifies multiple
    148 
    +components, then the summary line should still mention generally what
    149 
    +changed (if possible), followed by a colon and a brief summary.
    150 
    +
    151 
    +In this case the free form detailed description of the change should
    152 
    +contain a bullet list describing what was changed in each component
    153 
    +separately.
    154 
    +
    155 
    +**Example**::
    156 
    +
    157 
    +  artifact cache: Fixed automatic expiry in the local cache
    80 158
    81 
    -Coding style
    82 
    -------------
    83 
    -Coding style details for BuildStream
    159 
    +    o _artifactcache/artifactcache.py: Updated the API contract
    160 
    +      of ArtifactCache.remove() so that something detailed is
    161 
    +      explained here.
    84 162
    163 
    +    o _artifactcache/cascache.py: Adhere to the new API contract
    164 
    +      dictated by the abstract ArtifactCache class.
    85 165
    86 
    -Style guide
    87 
    -~~~~~~~~~~~
    88 
    -Python coding style for BuildStream is pep8, which is documented here: https://www.python.org/dev/peps/pep-0008/
    166 
    +    o tests/artifactcache/expiry.py: Modified test expectations to
    167 
    +      match the new behavior.
    168 
    +
    169 
    +  This is a part of #123
    170 
    +
    171 
    +
    172 
    +Coding guidelines
    173 
    +-----------------
    174 
    +This section discusses coding style and other guidelines for hacking
    175 
    +on BuildStream. This is important to read through for writing any non-trivial
    176 
    +patches and especially outlines what people should watch out for when
    177 
    +reviewing patches.
    178 
    +
    179 
    +Much of the rationale behind what is layed out in this section considers
    180 
    +good traceability of lines of code with *git blame*, overall sensible
    181 
    +modular structure, consistency in how we write code, and long term maintenance
    182 
    +in mind.
    183 
    +
    184 
    +
    185 
    +Approximate PEP-8 Style
    186 
    +~~~~~~~~~~~~~~~~~~~~~~~
    187 
    +Python coding style for BuildStream is approximately `pep8 <https://www.python.org/dev/peps/pep-0008/>`_.
    89 188
    90 189 
     We have a couple of minor exceptions to this standard, we dont want to compromise
    91 190 
     code readability by being overly restrictive on line length for instance.
    92 191
    93 
    -The pep8 linter will run automatically when running the test suite.
    192 
    +The pep8 linter will run automatically when :ref:`running the test suite <contributing_testing>`.
    193 
    +
    194 
    +
    195 
    +Line lengths
    196 
    +''''''''''''
    197 
    +Regarding laxness on the line length in our linter settings, it should be clarified
    198 
    +that the line length limit is a hard limit which causes the linter to bail out
    199 
    +and reject commits which break the high limit - not an invitation to write exceedingly
    200 
    +long lines of code, comments, or API documenting docstrings.
    201 
    +
    202 
    +Code, comments and docstrings should strive to remain written for approximately 80
    203 
    +or 90 character lines, where exceptions can be made when code would be less readable
    204 
    +when exceeding 80 or 90 characters (often this happens in conditional statements
    205 
    +when raising an exception, for example). Or, when comments contain a long link that
    206 
    +causes the given line to to exceed 80 or 90 characters, we don't want this to cause
    207 
    +the linter to refuse the commit.
    208 
    +
    209 
    +
    210 
    +.. _contributing_documenting_symbols:
    211 
    +
    212 
    +Documenting symbols
    213 
    +~~~~~~~~~~~~~~~~~~~
    214 
    +In BuildStream, we maintain what we call a *"Public API Surface"* that
    215 
    +is guaranteed to be stable and unchanging across stable releases. The
    216 
    +symbols which fall into this special class are documented using Python's
    217 
    +standard *docstrings*, while all other internals of BuildStream are documented
    218 
    +with comments above the related symbol.
    219 
    +
    220 
    +When documenting the public API surface which is rendered in the reference
    221 
    +manual, we always mention the major version in which the API was introduced,
    222 
    +as shown in the examples below. If a public API exists without the *Since*
    223 
    +annotation, this is taken to mean that it was available since the first stable
    224 
    +release 1.0.
    225 
    +
    226 
    +Here are some examples to get the hang of the format of API documenting
    227 
    +comments and docstrings.
    228 
    +
    229 
    +**Public API Surface method**::
    230 
    +
    231 
    +  def frobnicate(self, source, *, frobilicious=False):
    232 
    +      """Frobnicates this element with the specified source
    233 
    +
    234 
    +      Args:
    235 
    +         source (Source): The Source to frobnicate with
    236 
    +         frobilicious (bool): Optionally specify that frobnication should be
    237 
    +                              performed fribiliciously
    238 
    +
    239 
    +      Returns:
    240 
    +         (Element): The frobnicated version of this Element.
    241 
    +
    242 
    +      *Since: 1.2*
    243 
    +      """
    244 
    +      ...
    245 
    +
    246 
    +**Internal method**::
    247 
    +
    248 
    +  # frobnicate():
    249 
    +  #
    250 
    +  # Frobnicates this element with the specified source
    251 
    +  #
    252 
    +  # Args:
    253 
    +  #    source (Source): The Source to frobnicate with
    254 
    +  #    frobilicious (bool): Optionally specify that frobnication should be
    255 
    +  #                         performed fribiliciously
    256 
    +  #
    257 
    +  # Returns:
    258 
    +  #    (Element): The frobnicated version of this Element.
    259 
    +  #
    260 
    +  def frobnicate(self, source, *, frobilicious=False):
    261 
    +      ...
    262 
    +
    263 
    +**Public API Surface instance variable**::
    264 
    +
    265 
    +  def __init__(self, context, element):
    266 
    +
    267 
    +    self.name = self._compute_name(context, element)
    268 
    +    """The name of this foo
    269 
    +
    270 
    +    *Since: 1.2*
    271 
    +    """
    272 
    +
    273 
    +**Internal instance variable**::
    274 
    +
    275 
    +  def __init__(self, context, element):
    276 
    +
    277 
    +    self.name = self._compute_name(context, element) # The name of this foo
    278 
    +
    279 
    +**Internal instance variable (long)**::
    280 
    +
    281 
    +  def __init__(self, context, element):
    282 
    +
    283 
    +    # This instance variable required a longer explanation, so
    284 
    +    # it is on a line above the instance variable declaration.
    285 
    +    self.name = self._compute_name(context, element)
    286 
    +
    287 
    +
    288 
    +**Public API Surface class**::
    289 
    +
    290 
    +  class Foo(Bar):
    291 
    +      """The main Foo object in the data model
    292 
    +
    293 
    +      Explanation about Foo. Note that we always document
    294 
    +      the constructor arguments here, and not beside the __init__
    295 
    +      method.
    296 
    +
    297 
    +      Args:
    298 
    +         context (Context): The invocation Context
    299 
    +         count (int): The number to count
    300 
    +
    301 
    +      *Since: 1.2*
    302 
    +      """
    303 
    +      ...
    304 
    +
    305 
    +**Internal class**::
    306 
    +
    307 
    +  # Foo()
    308 
    +  #
    309 
    +  # The main Foo object in the data model
    310 
    +  #
    311 
    +  # Args:
    312 
    +  #    context (Context): The invocation Context
    313 
    +  #    count (int): The number to count
    314 
    +  #
    315 
    +  class Foo(Bar):
    316 
    +      ...
    317 
    +
    318 
    +
    319 
    +.. _contributing_class_order:
    320 
    +
    321 
    +Class structure and ordering
    322 
    +~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    323 
    +When creating or modifying an object class in BuildStream, it is
    324 
    +important to keep in mind the order in which symbols should appear
    325 
    +and keep this consistent.
    326 
    +
    327 
    +Here is an example to illustrate the expected ordering of symbols
    328 
    +on a Python class in BuildStream::
    329 
    +
    330 
    +  class Foo(Bar):
    331 
    +
    332 
    +      # Public class-wide variables come first, if any.
    333 
    +
    334 
    +      # Private class-wide variables, if any
    335 
    +
    336 
    +      # Now we have the dunder/magic methods, always starting
    337 
    +      # with the __init__() method.
    338 
    +
    339 
    +      def __init__(self, name):
    340 
    +
    341 
    +          super().__init__()
    342 
    +
    343 
    +          # NOTE: In the instance initializer we declare any instance variables,
    344 
    +          #       always declare the public instance variables (if any) before
    345 
    +          #       the private ones.
    346 
    +          #
    347 
    +          #       It is preferred to avoid any public instance variables, and
    348 
    +          #       always expose an accessor method for it instead.
    349 
    +
    350 
    +          #
    351 
    +          # Public instance variables
    352 
    +          #
    353 
    +          self.name = name  # The name of this foo
    354 
    +
    355 
    +          #
    356 
    +          # Private instance variables
    357 
    +          #
    358 
    +          self._count = 0   # The count of this foo
    359 
    +
    360 
    +      ################################################
    361 
    +      #               Abstract Methods               #
    362 
    +      ################################################
    363 
    +
    364 
    +      # NOTE: Abstract methods in BuildStream are allowed to have
    365 
    +      #       default methods.
    366 
    +      #
    367 
    +      #       Subclasses must NEVER override any method which was
    368 
    +      #       not advertized as an abstract method by the parent class.
    369 
    +
    370 
    +      # frob()
    371 
    +      #
    372 
    +      # Implementors should implement this to frob this foo
    373 
    +      # count times if possible.
    374 
    +      #
    375 
    +      # Args:
    376 
    +      #    count (int): The number of times to frob this foo
    377 
    +      #
    378 
    +      # Returns:
    379 
    +      #    (int): The number of times this foo was frobbed.
    380 
    +      #
    381 
    +      # Raises:
    382 
    +      #    (FooError): Implementors are expected to raise this error
    383 
    +      #
    384 
    +      def frob(self, count):
    385 
    +
    386 
    +          #
    387 
    +          # An abstract method in BuildStream is allowed to have
    388 
    +          # a default implementation.
    389 
    +          #
    390 
    +          self._count = self._do_frobbing(count)
    391 
    +
    392 
    +          return self._count
    393 
    +
    394 
    +      ################################################
    395 
    +      #     Implementation of abstract methods       #
    396 
    +      ################################################
    397 
    +
    398 
    +      # NOTE: Implementations of abstract methods defined by
    399 
    +      #       the parent class should NEVER document the API
    400 
    +      #       here redundantly.
    401 
    +
    402 
    +      def frobbish(self):
    403 
    +         #
    404 
    +         # Implementation of the "frobbish" abstract method
    405 
    +         # defined by the parent Bar class.
    406 
    +         #
    407 
    +	 return True
    408 
    +
    409 
    +      ################################################
    410 
    +      #                 Public Methods               #
    411 
    +      ################################################
    412 
    +
    413 
    +      # NOTE: Public methods here are the ones which are expected
    414 
    +      #       to be called from outside of this class.
    415 
    +      #
    416 
    +      #       These, along with any abstract methods, usually
    417 
    +      #       constitute the API surface of this class.
    418 
    +
    419 
    +      # frobnicate()
    420 
    +      #
    421 
    +      # Perform the frobnication process on this Foo
    422 
    +      #
    423 
    +      # Raises:
    424 
    +      #    (FrobError): In the case that a frobnication error was
    425 
    +      #                 encountered
    426 
    +      #
    427 
    +      def frobnicate(self):
    428 
    +          frobnicator.frobnicate(self)
    429 
    +
    430 
    +      # set_count()
    431 
    +      #
    432 
    +      # Sets the count of this foo
    433 
    +      #
    434 
    +      # Args:
    435 
    +      #    count (int): The new count to set
    436 
    +      #
    437 
    +      def set_count(self, count):
    438 
    +
    439 
    +          self._count = count
    440 
    +
    441 
    +      # get_count()
    442 
    +      #
    443 
    +      # Accessor for the count value of this foo.
    444 
    +      #
    445 
    +      # Returns:
    446 
    +      #    (int): The count of this foo
    447 
    +      #
    448 
    +      def set_count(self, count):
    449 
    +
    450 
    +          return self._count
    451 
    +
    452 
    +      ################################################
    453 
    +      #                 Private Methods              #
    454 
    +      ################################################
    455 
    +
    456 
    +      # NOTE: Private methods are the ones which are internal
    457 
    +      #       implementation details of this class.
    458 
    +      #
    459 
    +      #       Even though these are private implementation
    460 
    +      #       details, they still MUST have API documenting
    461 
    +      #       comments on them.
    462 
    +
    463 
    +      # _do_frobbing()
    464 
    +      #
    465 
    +      # Does the actual frobbing
    466 
    +      #
    467 
    +      # Args:
    468 
    +      #    count (int): The number of times to frob this foo
    469 
    +      #
    470 
    +      # Returns:
    471 
    +      #    (int): The number of times this foo was frobbed.
    472 
    +      #
    473 
    +      def self._do_frobbing(self, count):
    474 
    +          return count
    475 
    +
    476 
    +
    477 
    +.. _contributing_public_and_private:
    478 
    +
    479 
    +Public and private symbols
    480 
    +~~~~~~~~~~~~~~~~~~~~~~~~~~
    481 
    +BuildStream mostly follows the PEP-8 for defining *public* and *private* symbols
    482 
    +for any given class, with some deviations. Please read the `section on inheritance
    483 
    +<https://www.python.org/dev/peps/pep-0008/#designing-for-inheritance>`_ for
    484 
    +reference on how the PEP-8 defines public and non-public.
    485 
    +
    486 
    +* A *public* symbol is any symbol which you expect to be used by clients
    487 
    +  of your class or module within BuildStream.
    488 
    +
    489 
    +  Public symbols are written without any leading underscores.
    490 
    +
    491 
    +* A *private* symbol is any symbol which is entirely internal to your class
    492 
    +  or module within BuildStream. These symbols cannot ever be accessed by
    493 
    +  external clients or modules.
    494 
    +
    495 
    +  A private symbol must be denoted by a leading underscore.
    496 
    +
    497 
    +* When a class can have subclasses (for example, the ``Sandbox`` or ``Platform``
    498 
    +  classes which have various implementations, or the ``Element`` and ``Source``
    499 
    +  classes which plugins derive from), then private symbols should be denoted
    500 
    +  by two leading underscores.
    501 
    +
    502 
    +  The double leading underscore naming convention invokes Python's name
    503 
    +  mangling algorithm which helps prevent namespace collisions in the case
    504 
    +  that subclasses might have a private symbol with the same name.
    505 
    +
    506 
    +In BuildStream, we have what we call a *"Public API Surface"*, as previously
    507 
    +mentioned in :ref:`contributing_documenting_symbols`. In the :ref:`next section
    508 
    +<contributing_public_api_surface>` we will discuss the *"Public API Surface"* and
    509 
    +outline the exceptions to the rules discussed here.
    510 
    +
    511 
    +
    512 
    +.. _contributing_public_api_surface:
    513 
    +
    514 
    +Public API surface
    515 
    +~~~~~~~~~~~~~~~~~~
    516 
    +BuildStream exposes what we call a *"Public API Surface"* which is stable
    517 
    +and unchanging. This is for the sake of stability of the interfaces which
    518 
    +plugins use, so it can also be referred to as the *"Plugin facing API"*.
    519 
    +
    520 
    +Any symbols which are a part of the *"Public API Surface*" are never allowed
    521 
    +to change once they have landed in a stable release version of BuildStream. As
    522 
    +such, we aim to keep the *"Public API Surface"* as small as possible at all
    523 
    +times, and never expose any internal details to plugins inadvertently.
    524 
    +
    525 
    +One problem which arises from this is that we end up having symbols
    526 
    +which are *public* according to the :ref:`rules discussed in the previous section
    527 
    +<contributing_public_and_private>`, but must be hidden away from the
    528 
    +*"Public API Surface"*. For example, BuildStream internal classes need
    529 
    +to invoke methods on the ``Element`` and ``Source`` classes, wheras these
    530 
    +methods need to be hidden from the *"Public API Surface"*.
    531 
    +
    532 
    +This is where BuildStream deviates from the PEP-8 standard for public
    533 
    +and private symbol naming.
    534 
    +
    535 
    +In order to disambiguate between:
    536 
    +
    537 
    +* Symbols which are publicly accessible details of the ``Element`` class, can
    538 
    +  be accessed by BuildStream internals, but must remain hidden from the
    539 
    +  *"Public API Surface"*
    540 
    +
    541 
    +* Symbols which are private to the ``Element`` class, and cannot be accessed
    542 
    +  from outside of the ``Element`` class at all.
    543 
    +
    544 
    +We denote the former category of symbols with only a single underscore, and the latter
    545 
    +category of symbols with a double underscore. We often refer to this distinction
    546 
    +as *"API Private"* (the former category) and *"Local Private"* (the latter category).
    547 
    +
    548 
    +Classes which are a part of the *"Public API Surface"* and require this disambiguation
    549 
    +were not discussed in :ref:`the class ordering section <contributing_class_order>`, for
    550 
    +these classes, the *"API Private"* symbols always come **before** the *"Local Private"*
    551 
    +symbols in the class declaration.
    552 
    +
    553 
    +Modules which are not a part of the *"Public API Surface"* have their Python files
    554 
    +prefixed with a single underscore, and are not imported in BuildStream's the master
    555 
    +``__init__.py`` which is used by plugins.
    556 
    +
    557 
    +.. note::
    558 
    +
    559 
    +   The ``utils.py`` module is public and exposes a handful of utility functions,
    560 
    +   however many of the functions it provides are *"API Private"*.
    561 
    +
    562 
    +   In this case, the *"API Private"* functions are prefixed with a single underscore.
    563 
    +
    564 
    +Any objects which are a part of the *"Public API Surface"* should be exposed via the
    565 
    +toplevel ``__init__.py`` of the ``buildstream`` package.
    566 
    +
    567 
    +
    568 
    +File naming convention
    569 
    +~~~~~~~~~~~~~~~~~~~~~~
    570 
    +With the exception of a few helper objects and data structures, we structure
    571 
    +the code in BuildStream such that every filename is named after the object it
    572 
    +implements. E.g. The ``Project`` object is implemented in ``_project.py``, the
    573 
    +``Context`` object in ``_context.py``, the base ``Element`` class in ``element.py``,
    574 
    +etc.
    575 
    +
    576 
    +As mentioned in the previous section, objects which are not a part of the
    577 
    +:ref:`public, plugin facing API surface <contributing_public_api_surface>` have their
    578 
    +filenames prefixed with a leading underscore (like ``_context.py`` and ``_project.py``
    579 
    +in the examples above).
    580 
    +
    581 
    +When an object name has multiple words in it, e.g. ``ArtifactCache``, then the
    582 
    +resulting file is named all in lower case without any underscore to separate
    583 
    +words. In the case of ``ArtifactCache``, the filename implementing this object
    584 
    +is found at ``_artifactcache/artifactcache.py``.
    94 585
    95 586
    96 587 
     Imports
    97 588 
     ~~~~~~~
    98 589 
     Module imports inside BuildStream are done with relative ``.`` notation
    99 590
    100 
    -Good::
    591 
    +**Good**::
    101 592
    102 
    -  from .context import Context
    593 
    +  from ._context import Context
    103 594
    104 
    -Bad::
    595 
    +**Bad**::
    105 596
    106 
    -  from buildstream.context import Context
    597 
    +  from buildstream._context import Context
    107 598
    108 599 
     The exception to the above rule is when authoring plugins,
    109 600 
     plugins do not reside in the same namespace so they must
    ... ... @@ -122,128 +613,586 @@ This makes things clear when reading code that said functions
    122 613 
     are not defined in the same file but come from utils.py for example.
    123 614
    124 615
    125 
    -Policy for private symbols
    126 
    -~~~~~~~~~~~~~~~~~~~~~~~~~~
    127 
    -Private symbols are expressed via a leading ``_`` single underscore, or
    128 
    -in some special circumstances with a leading ``__`` double underscore.
    616 
    +.. _contributing_instance_variables:
    617 
    +
    618 
    +Instance variables
    619 
    +~~~~~~~~~~~~~~~~~~
    620 
    +It is preferred that all instance state variables be declared as :ref:`private symbols
    621 
    +<contributing_public_and_private>`, however in some cases, especially when the state
    622 
    +is immutable for the object's life time (like an ``Element`` name for example), it
    623 
    +is acceptable to save some typing by using a publicly accessible instance variable.
    624 
    +
    625 
    +It is never acceptable to modify the value of an instance variable from outside
    626 
    +of the declaring class, even if the variable is *public*. In other words, the class
    627 
    +which exposes an instance variable is the only one in control of the value of this
    628 
    +variable.
    629 
    +
    630 
    +* If an instance variable is public and must be modified; then it must be
    631 
    +  modified using a :ref:`mutator <contributing_accessor_mutator>`
    632 
    +
    633 
    +* Ideally for better encapsulation, all object state is declared as
    634 
    +  :ref:`private instance variables <contributing_public_and_private>` and can
    635 
    +  only be accessed by external classes via public :ref:`accessors and mutators
    636 
    +  <contributing_accessor_mutator>`
    637 
    +
    638 
    +.. note::
    639 
    +
    640 
    +   In some cases, we may use small data structures declared as objects for the sake
    641 
    +   of better readability, where the object class itself has no real supporting code.
    642 
    +
    643 
    +   In these exceptions, it can be acceptable to modify the instance variables
    644 
    +   of these objects directly, unless they are otherwise documented to be immutable.
    645 
    +
    646 
    +
    647 
    +.. _contributing_accessor_mutator:
    648 
    +
    649 
    +Accessors and mutators
    650 
    +~~~~~~~~~~~~~~~~~~~~~~
    651 
    +An accessor and mutator, are methods defined on the object class to access (get)
    652 
    +or mutate (set) a value owned by the declaring class, respectively.
    653 
    +
    654 
    +An accessor might derive the returned value from one or more of its components,
    655 
    +and a mutator might have side effects, or delegate the mutation to a component.
    656 
    +
    657 
    +Accessors and mutators are always :ref:`public <contributing_public_and_private>`
    658 
    +(even if they might have a single leading underscore and are considered
    659 
    +:ref:`API Private <contributing_public_api_surface>`), as their purpose is to
    660 
    +enforce encapsulation with regards to any accesses to the state which is owned
    661 
    +by the declaring class.
    662 
    +
    663 
    +Accessors and mutators are functions prefixed with ``get_`` and ``set_``
    664 
    +respectively, e.g.::
    665 
    +
    666 
    +  class Foo():
    667 
    +
    668 
    +      def __init__(self):
    669 
    +
    670 
    +          # Declare some internal state
    671 
    +          self._count = 0
    672 
    +
    673 
    +      # get_count()
    674 
    +      #
    675 
    +      # Gets the count of this Foo.
    676 
    +      #
    677 
    +      # Returns:
    678 
    +      #    (int): The current count of this Foo
    679 
    +      #
    680 
    +      def get_foo(self):
    681 
    +          return self._count
    682 
    +
    683 
    +      # set_count()
    684 
    +      #
    685 
    +      # Sets the count of this Foo.
    686 
    +      #
    687 
    +      # Args:
    688 
    +      #    count (int): The new count for this Foo
    689 
    +      #
    690 
    +      def set_foo(self, count):
    691 
    +          self._count = count
    692 
    +
    693 
    +.. attention::
    129 694
    130 
    -Before understanding the naming policy, it is first important to understand
    131 
    -that in BuildStream, there are two levels of privateness which need to be
    132 
    -considered.
    695 
    +   We are aware that Python offers a facility for accessors and
    696 
    +   mutators using the ``@property`` decorator instead. Do not use
    697 
    +   the ``@property`` decorator.
    133 698
    134 
    -These are treated subtly differently and thus need to be understood:
    699 
    +   The decision to use explicitly defined functions instead of the
    700 
    +   ``@property`` decorator is rather arbitrary, there is not much
    701 
    +   technical merit to preferring one technique over the other.
    702 
    +   However as :ref:`discussed below <contributing_always_consistent>`,
    703 
    +   it is of the utmost importance that we do not mix both techniques
    704 
    +   in the same codebase.
    135 705
    136 
    -* API Private
    137 706
    138 
    -  A symbol is considered to be *API private* if it is not exposed in the *public API*.
    707 
    +.. _contributing_abstract_methods:
    139 708
    140 
    -  Even if a symbol does not have any leading underscore, it may still be *API private*
    141 
    -  if the containing *class* or *module* is named with a leading underscore.
    709 
    +Abstract methods
    710 
    +~~~~~~~~~~~~~~~~
    711 
    +In BuildStream, an *"Abstract Method"* is a bit of a misnomer and does
    712 
    +not match up to how Python defines abstract methods, we need to seek out
    713 
    +a new nomanclature to refer to these methods.
    142 714
    143 
    -* Local private
    715 
    +In Python, an *"Abstract Method"* is a method which **must** be
    716 
    +implemented by a subclass, whereas all methods in Python can be
    717 
    +overridden.
    144 718
    145 
    -  A symbol is considered to be *local private* if it is not intended for access
    146 
    -  outside of the defining *scope*.
    719 
    +In BuildStream, we use the term *"Abstract Method"*, to refer to
    720 
    +a method which **can** be overridden by a subclass, whereas it
    721 
    +is **illegal** to override any other method.
    147 722
    148 
    -  If a symbol has a leading underscore, it might not be *local private* if it is
    149 
    -  declared on a publicly visible class, but needs to be accessed internally by
    150 
    -  other modules in the BuildStream core.
    723 
    +* Abstract methods are allowed to have default implementations
    151 724
    725 
    +* Subclasses are not allowed to redefine the calling signature
    726 
    +  of an abstract method, or redefine the API contract in any way
    152 727
    153 
    -Ordering
    154 
    -''''''''
    155 
    -For better readability and consistency, we try to keep private symbols below
    156 
    -public symbols. In the case of public modules where we may have a mix of
    157 
    -*API private* and *local private* symbols, *API private* symbols should come
    158 
    -before *local private* symbols.
    728 
    +* Subclasses are not allowed to override any other methods.
    159 729
    730 
    +The key here is that in BuildStream, we consider it unacceptable
    731 
    +that a subclass overrides a method of it's parent class unless
    732 
    +the said parent class has explicitly given permission to subclasses
    733 
    +to do so, and outlined the API contract for this purpose. No surprises
    734 
    +are allowed.
    160 735
    161 
    -Symbol naming
    162 
    -'''''''''''''
    163 
    -Any private symbol must start with a single leading underscore for two reasons:
    164 736
    165 
    -* So that it does not bleed into documentation and *public API*.
    737 
    +Error handling
    738 
    +~~~~~~~~~~~~~~
    739 
    +In BuildStream, all non recoverable errors are expressed via
    740 
    +subclasses of the ``BstError`` exception.
    166 741
    167 
    -* So that it is clear to developers which symbols are not used outside of the declaring *scope*
    742 
    +This exception is handled deep in the core in a few places, and
    743 
    +it is rarely necessary to handle a ``BstError``.
    168 744
    169 
    -Remember that with python, the modules (python files) are also symbols
    170 
    -within their containing *package*, as such; modules which are entirely
    171 
    -private to BuildStream are named as such, e.g. ``_thismodule.py``.
    172 745
    746 
    +Raising exceptions
    747 
    +''''''''''''''''''
    748 
    +When writing code in the BuildStream core, ensure that all system
    749 
    +calls and third party library calls are wrapped in a ``try:`` block,
    750 
    +and raise a descriptive ``BstError`` of the appropriate class explaining
    751 
    +what exactly failed.
    173 752
    174 
    -Cases for double underscores
    175 
    -''''''''''''''''''''''''''''
    176 
    -The double underscore in python has a special function. When declaring
    177 
    -a symbol in class scope which has a leading underscore, it can only be
    178 
    -accessed within the class scope using the same name. Outside of class
    179 
    -scope, it can only be accessed with a *cheat*.
    753 
    +Ensure that the original system call error is formatted into your new
    754 
    +exception, and that you use the Python ``from`` semantic to retain the
    755 
    +original call trace, example::
    180 756
    181 
    -We use the double underscore in cases where the type of privateness can be
    182 
    -ambiguous.
    757 
    +  try:
    758 
    +      os.utime(self._refpath(ref))
    759 
    +  except FileNotFoundError as e:
    760 
    +      raise ArtifactError("Attempt to access unavailable artifact: {}".format(e)) from e
    183 761
    184 
    -* For private modules and classes
    185 762
    186 
    -  We never need to disambiguate with a double underscore
    763 
    +Enhancing exceptions
    764 
    +''''''''''''''''''''
    765 
    +Sometimes the ``BstError`` originates from a lower level component,
    766 
    +and the code segment which raised the exception did not have enough context
    767 
    +to create a complete, informative summary of the error for the user.
    187 768
    188 
    -* For private symbols declared in a public *scope*
    769 
    +In these cases it is necessary to handle the error and raise a new
    770 
    +one, e.g.::
    189 771
    190 
    -  In the case that we declare a private method on a public object, it
    191 
    -  becomes ambiguous whether:
    772 
    +  try:
    773 
    +      extracted_artifact = self._artifacts.extract(self, cache_key)
    774 
    +  except ArtifactError as e:
    775 
    +      raise ElementError("Failed to extract {} while checking out {}: {}"
    776 
    +                         .format(cache_key, self.name, e)) from e
    192 777
    193 
    -  * The symbol is *local private*, and only used within the given scope
    194 778
    195 
    -  * The symbol is *API private*, and will be used internally by BuildStream
    196 
    -    from other parts of the codebase.
    779 
    +Programming errors
    780 
    +''''''''''''''''''
    781 
    +Sometimes you are writing code and have detected an unexpected condition,
    782 
    +or a broken invariant for which the code cannot be prepared to handle
    783 
    +gracefully.
    197 784
    198 
    -  In this case, we use a single underscore for *API private* methods which
    199 
    -  are not *local private*, and we use a double underscore for *local private*
    200 
    -  methods declared in public scope.
    785 
    +In these cases, do **not** raise any of the ``BstError`` class exceptions.
    201 786
    787 
    +Instead, use the ``assert`` statement, e.g.::
    202 788
    203 
    -Documenting private symbols
    204 
    -'''''''''''''''''''''''''''
    205 
    -Any symbol which is *API Private* (regardless of whether it is also
    206 
    -*local private*), should have some documentation for developers to
    207 
    -better understand the codebase.
    789 
    +  assert utils._is_main_process(), \
    790 
    +      "Attempted to save workspace configuration from child process"
    208 791
    209 
    -Contrary to many other python projects, we do not use docstrings to
    210 
    -document private symbols, but prefer to keep *API Private* symbols
    211 
    -documented in code comments placed *above* the symbol (or *beside* the
    212 
    -symbol in some cases, such as variable declarations in a class where
    213 
    -a shorter comment is more desirable), rather than docstrings placed *below*
    214 
    -the symbols being documented.
    792 
    +This will result in a ``BUG`` message with the stack trace included being
    793 
    +logged and reported in the frontend.
    215 794
    216 
    -Other than this detail, follow the same guidelines for documenting
    217 
    -symbols as described below.
    218 795
    796 
    +BstError parameters
    797 
    +'''''''''''''''''''
    798 
    +When raising ``BstError`` class exceptions, there are some common properties
    799 
    +which can be useful to know about:
    219 800
    220 
    -Documenting BuildStream
    221 
    ------------------------
    801 
    +* **message:** The brief human readable error, will be formatted on one line in the frontend
    802 
    +
    803 
    +* **detail:** An optional detailed human readable message to accompany the **message** summary
    804 
    +  of the error. This is often used to recommend the user some course of action, or to provide
    805 
    +  additional context about the error.
    806 
    +
    807 
    +* **temporary:** Some errors are allowed to be *temporary*, this attribute is only
    808 
    +  observed from child processes which fail in a temporary way. This distinction
    809 
    +  is used to determine whether the task should be *retried* or not. An error is usually
    810 
    +  only a *temporary* error if the cause of the error was a network timeout.
    811 
    +
    812 
    +* **reason:** A machine readable identifier for the error. This is used for the purpose
    813 
    +  of regression testing, such that we check that BuildStream has errored out for the
    814 
    +  expected reason in a given failure mode.
    815 
    +
    816 
    +
    817 
    +Documenting Exceptions
    818 
    +''''''''''''''''''''''
    819 
    +We have already seen :ref:`some examples <contributing_class_order>` of how
    820 
    +exceptions are documented in API documenting comments, but this is worth some
    821 
    +additional disambiguation.
    822 
    +
    823 
    +* Only document the exceptions which are raised directly by the function in question.
    824 
    +  It is otherwise nearly impossible to keep track of what exceptions *might* be raised
    825 
    +  indirectly by calling the given function.
    826 
    +
    827 
    +* For a regular public or private method, your audience is a caller of the function;
    828 
    +  document the exception in terms of what exception might be raised as a result of
    829 
    +  calling this method.
    830 
    +
    831 
    +* For an :ref:`abstract method <contributing_abstract_methods>`, your audience is the
    832 
    +  implementor of the method in a subclass; document the exception in terms of what
    833 
    +  exception is prescribed for the implementing class to raise.
    834 
    +
    835 
    +
    836 
    +.. _contributing_always_consistent:
    837 
    +
    838 
    +Always be consistent
    839 
    +~~~~~~~~~~~~~~~~~~~~
    840 
    +There are various ways to define functions and classes in Python,
    841 
    +which has evolved with various features over time.
    842 
    +
    843 
    +In BuildStream, we may not have leveraged all of the nice features
    844 
    +we could have, that is okay, and where it does not break API, we
    845 
    +can consider changing it.
    846 
    +
    847 
    +Even if you know there is a *better* way to do a given thing in
    848 
    +Python when compared to the way we do it in BuildStream, *do not do it*.
    849 
    +
    850 
    +Consistency of how we do things in the codebase is more important
    851 
    +than the actual way in which things are done, always.
    852 
    +
    853 
    +Instead, if you like a certain Python feature and think the BuildStream
    854 
    +codebase should use it, then propose your change on the `mailing list
    855 
    +<https://mail.gnome.org/mailman/listinfo/buildstream-list>`_. Chances
    856 
    +are that we will reach agreement to use your preferred approach, and
    857 
    +in that case, it will be important to apply the change unilaterally
    858 
    +across the entire codebase, such that we continue to have a consistent
    859 
    +codebase.
    860 
    +
    861 
    +
    862 
    +Avoid tail calling
    863 
    +~~~~~~~~~~~~~~~~~~
    864 
    +With the exception of tail calling with simple functions from
    865 
    +the standard Python library, such as splitting and joining lines
    866 
    +of text and encoding/decoding text; always avoid tail calling.
    867 
    +
    868 
    +**Good**::
    869 
    +
    870 
    +  # Variables that we will need declared up top
    871 
    +  context = self._get_context()
    872 
    +  workspaces = context.get_workspaces()
    873 
    +
    874 
    +  ...
    875 
    +
    876 
    +  # Saving the workspace configuration
    877 
    +  workspaces.save_config()
    878 
    +
    879 
    +**Bad**::
    880 
    +
    881 
    +  # Saving the workspace configuration
    882 
    +  self._get_context().get_workspaces().save_config()
    883 
    +
    884 
    +**Acceptable**::
    885 
    +
    886 
    +  # Decode the raw text loaded from a log file for display,
    887 
    +  # join them into a single utf-8 string and strip away any
    888 
    +  # trailing whitespace.
    889 
    +  return '\n'.join([line.decode('utf-8') for line in lines]).rstrip()
    890 
    +
    891 
    +When you need to obtain a delegate object via an accessor function,
    892 
    +either do it at the beginning of the function, or at the beginning
    893 
    +of a code block within the function that will use that object.
    894 
    +
    895 
    +There are several reasons for this convention:
    896 
    +
    897 
    +* When observing a stack trace, it is always faster and easier to
    898 
    +  determine what went wrong when all statements are on separate lines.
    899 
    +
    900 
    +* We always want individual lines to trace back to their origin as
    901 
    +  much as possible for the purpose of tracing the history of code
    902 
    +  with *git blame*.
    903 
    +
    904 
    +  One day, you might need the ``Context`` or ``Workspaces`` object
    905 
    +  in the same function for another reason, at which point it will
    906 
    +  be unacceptable to leave the existing line as written, because it
    907 
    +  will introduce a redundant accessor to the same object, so the
    908 
    +  line written as::
    909 
    +
    910 
    +    self._get_context().get_workspaces().save_config()
    911 
    +
    912 
    +  Will have to change at that point, meaning we lose the valuable
    913 
    +  information of which commit originally introduced this call
    914 
    +  when running *git blame*.
    915 
    +
    916 
    +* For similar reasons, we prefer delegate objects be accessed near
    917 
    +  the beginning of a function or code block so that there is less
    918 
    +  chance that this statement will have to move in the future, if
    919 
    +  the same function or code block needs the delegate object for any
    920 
    +  other reason.
    921 
    +
    922 
    +  Asides from this, code is generally more legible and uniform when
    923 
    +  variables are declared at the beginning of function blocks.
    924 
    +
    925 
    +
    926 
    +Vertical stacking of modules
    927 
    +~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    928 
    +For the sake of overall comprehensiveness of the BuildStream
    929 
    +architecture, it is important that we retain vertical stacking
    930 
    +order of the dependencies and knowledge of modules as much as
    931 
    +possible, and avoid any cyclic relationships in modules.
    932 
    +
    933 
    +For instance, the ``Source`` objects are owned by ``Element``
    934 
    +objects in the BuildStream data model, and as such the ``Element``
    935 
    +will delegate some activities to the ``Source`` objects in it's
    936 
    +possesion. The ``Source`` objects should however never call functions
    937 
    +on the ``Element`` object, nor should the ``Source`` object itself
    938 
    +have any understanding of what an ``Element`` is.
    939 
    +
    940 
    +If you are implementing a low level utility layer, for example
    941 
    +as a part of the ``YAML`` loading code layers, it can be tempting
    942 
    +to derive context from the higher levels of the codebase which use
    943 
    +these low level utilities, instead of defining properly stand alone
    944 
    +APIs for these utilities to work: Never do this.
    945 
    +
    946 
    +Unfortunately, unlike other languages where include files play
    947 
    +a big part in ensuring that it is difficult to make a mess; Python,
    948 
    +allows you to just call methods on arbitrary objects passed through
    949 
    +a function call without having to import the module which defines
    950 
    +those methods - this leads to cyclic dependencies of modules quickly
    951 
    +if the developer does not take special care of ensuring this does not
    952 
    +happen.
    953 
    +
    954 
    +
    955 
    +Minimize arguments in methods
    956 
    +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    957 
    +When creating an object, or adding a new API method to an existing
    958 
    +object, always strive to keep as much context as possible on the
    959 
    +object itself rather than expecting callers of the methods to provide
    960 
    +everything the method needs every time.
    961 
    +
    962 
    +If the value or object that is needed in a function call is a constant
    963 
    +for the lifetime of the object which exposes the given method, then
    964 
    +that value or object should be passed in the constructor instead of
    965 
    +via a method call.
    966 
    +
    967 
    +
    968 
    +Minimize API surfaces
    969 
    +~~~~~~~~~~~~~~~~~~~~~
    970 
    +When creating an object, or adding new functionality in any way,
    971 
    +try to keep the number of :ref:`public, outward facing <contributing_public_and_private>`
    972 
    +symbols to a minimum, this is important for both
    973 
    +:ref:`internal and public, plugin facing API surfaces <contributing_public_api_surface>`.
    974 
    +
    975 
    +When anyone visits a file, there are two levels of comprehension:
    976 
    +
    977 
    +* What do I need to know in order to *use* this object
    978 
    +
    979 
    +* What do I need to know in order to *modify* this object
    980 
    +
    981 
    +For the former, we want the reader to understand with as little effort
    982 
    +as possible, what the public API contract is for a given object and consequently,
    983 
    +how it is expected to be used. This is also why we
    984 
    +:ref:`order the symbols of a class <contributing_class_order>` in such a way
    985 
    +as to keep all outward facing public API surfaces at the top of the file, so that the
    986 
    +reader never needs to dig deep into the bottom of the file to find something they
    987 
    +might need to use.
    988 
    +
    989 
    +For the latter, when it comes to having to modify the file or add functionality,
    990 
    +you want to retain as much freedom as possible to modify internals, while
    991 
    +being sure that nothing external will be affected by internal modifications.
    992 
    +Less client facing API means that you have less surrounding code to modify
    993 
    +when your API changes. Further, ensuring that there is minimal outward facing
    994 
    +API for any module minimizes the complexity for the developer working on
    995 
    +that module, by limiting the considerations needed regarding external side
    996 
    +effects of their modifications to the module.
    997 
    +
    998 
    +When modifying a file, one should not have to understand or think too
    999 
    +much about external side effects, when the API surface of the file is
    1000 
    +well documented and minimal.
    1001 
    +
    1002 
    +When adding new API to a given object for a new purpose, consider whether
    1003 
    +the new API is in any way redundant with other API (should this value now
    1004 
    +go into the constructor, since we use it more than once ? could this
    1005 
    +value be passed along with another function, and the other function renamed,
    1006 
    +to better suit the new purposes of this module/object ?) and repurpose
    1007 
    +the outward facing API of an object as a whole every time.
    1008 
    +
    1009 
    +
    1010 
    +Avoid transient state on instances
    1011 
    +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    1012 
    +At times, it can be tempting to store transient state that is
    1013 
    +the result of one operation on an instance, only to be retrieved
    1014 
    +later via an accessor function elsewhere.
    1015 
    +
    1016 
    +As a basic rule of thumb, if the value is transient and just the
    1017 
    +result of one operation, which needs to be observed directly after
    1018 
    +by another code segment, then never store it on the instance.
    1019 
    +
    1020 
    +BuildStream is complicated in the sense that it is multi processed
    1021 
    +and it is not always obvious how to pass the transient state around
    1022 
    +as a return value or a function parameter. Do not fall prey to this
    1023 
    +obstacle and pollute object instances with transient state.
    1024 
    +
    1025 
    +Instead, always refactor the surrounding code so that the value
    1026 
    +is propagated to the desired end point via a well defined API, either
    1027 
    +by adding new code paths or changing the design such that the
    1028 
    +architecture continues to make sense.
    1029 
    +
    1030 
    +
    1031 
    +Refactor the codebase as needed
    1032 
    +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    1033 
    +Especially when implementing features, always move the BuildStream
    1034 
    +codebase forward as a whole.
    1035 
    +
    1036 
    +Taking a short cut is alright when prototyping, but circumventing
    1037 
    +existing architecture and design to get a feature implemented without
    1038 
    +re-designing the surrounding architecture to accommodate the new
    1039 
    +feature instead, is never acceptable upstream.
    1040 
    +
    1041 
    +For example, let's say that you have to implement a feature and you've
    1042 
    +successfully prototyped it, but it launches a ``Job`` directly from a
    1043 
    +``Queue`` implementation to get the feature to work, while the ``Scheduler``
    1044 
    +is normally responsible for dispatching ``Jobs`` for the elements on
    1045 
    +a ``Queue``. This means that you've proven that your feature can work,
    1046 
    +and now it is time to start working on a patch for upstream.
    1047 
    +
    1048 
    +Consider what the scenario is and why you are circumventing the design,
    1049 
    +and then redesign the ``Scheduler`` and ``Queue`` objects to accommodate for
    1050 
    +the new feature and condition under which you need to dispatch a ``Job``,
    1051 
    +or how you can give the ``Queue`` implementation the additional context it
    1052 
    +needs.
    1053 
    +
    1054 
    +
    1055 
    +Adding core plugins
    1056 
    +-------------------
    1057 
    +This is a checklist of things which need to be done when adding a new
    1058 
    +core plugin to BuildStream proper.
    1059 
    +
    1060 
    +
    1061 
    +Update documentation index
    1062 
    +~~~~~~~~~~~~~~~~~~~~~~~~~~
    1063 
    +The documentation generating scripts will automatically pick up your
    1064 
    +newly added plugin and generate HTML, but will not add a link to the
    1065 
    +documentation of your plugin automatically.
    1066 
    +
    1067 
    +Whenever adding a new plugin, you must add an entry for it in ``doc/source/core_plugins.rst``.
    1068 
    +
    1069 
    +
    1070 
    +Bump format version
    1071 
    +~~~~~~~~~~~~~~~~~~~
    1072 
    +In order for projects to assert that they have a new enough version
    1073 
    +of BuildStream to use the new plugin, the ``BST_FORMAT_VERSION`` must
    1074 
    +be incremented in the ``_versions.py`` file.
    1075 
    +
    1076 
    +Remember to include in your plugin's main documentation, the format
    1077 
    +version in which the plugin was introduced, using the standard annotation
    1078 
    +which we use throughout the documentation, e.g.::
    1079 
    +
    1080 
    +  .. note::
    1081 
    +
    1082 
    +     The ``foo`` plugin is available since :ref:`format version 16 <project_format_version>`
    1083 
    +
    1084 
    +
    1085 
    +Add tests
    1086 
    +~~~~~~~~~
    1087 
    +Needless to say, all new feature additions need to be tested. For ``Element``
    1088 
    +plugins, these usually need to be added to the integration tests. For ``Source``
    1089 
    +plugins, the tests are added in two ways:
    1090 
    +
    1091 
    +* For most normal ``Source`` plugins, it is important to add a new ``Repo``
    1092 
    +  implementation for your plugin in the ``tests/testutils/repo/`` directory
    1093 
    +  and update ``ALL_REPO_KINDS`` in ``tests/testutils/repo/__init__.py``. This
    1094 
    +  will include your new ``Source`` implementation in a series of already existing
    1095 
    +  tests, ensuring it works well under normal operating conditions.
    1096 
    +
    1097 
    +* For other source plugins, or in order to test edge cases, such as failure modes,
    1098 
    +  which are not tested under the normal test battery, add new tests in ``tests/sources``.
    1099 
    +
    1100 
    +
    1101 
    +Extend the cachekey test
    1102 
    +~~~~~~~~~~~~~~~~~~~~~~~~
    1103 
    +For any newly added plugins, it is important to add some new simple elements
    1104 
    +in ``tests/cachekey/project/elements`` or ``tests/cachekey/project/sources``,
    1105 
    +and ensure that the newly added elements are depended on by ``tests/cachekey/project/target.bst``.
    1106 
    +
    1107 
    +One new element should be added to the cache key test for every configuration
    1108 
    +value which your plugin understands which can possibly affect the result of
    1109 
    +your plugin's ``Plugin.get_unique_key()`` implementation.
    1110 
    +
    1111 
    +This test ensures that cache keys do not unexpectedly change or become incompatible
    1112 
    +due to code changes. As such, the cache key test should have full coverage of every
    1113 
    +YAML configuration which can possibly affect cache key outcome at all times.
    1114 
    +
    1115 
    +See the ``tests/cachekey/update.py`` file for instructions on running the updater,
    1116 
    +you need to run the updater to generate the ``.expected`` files and add the new
    1117 
    +``.expected`` files in the same commit which extends the cache key test.
    1118 
    +
    1119 
    +
    1120 
    +Protocol buffers
    1121 
    +----------------
    1122 
    +BuildStream uses protobuf and gRPC for serialization and communication with
    1123 
    +artifact cache servers.  This requires ``.proto`` files and Python code
    1124 
    +generated from the ``.proto`` files using protoc.  All these files live in the
    1125 
    +``buildstream/_protos`` directory.  The generated files are included in the
    1126 
    +git repository to avoid depending on grpcio-tools for user installations.
    1127 
    +
    1128 
    +
    1129 
    +Regenerating code
    1130 
    +~~~~~~~~~~~~~~~~~
    1131 
    +When ``.proto`` files are modified, the corresponding Python code needs to
    1132 
    +be regenerated.  As a prerequisite for code generation you need to install
    1133 
    +``grpcio-tools`` using pip or some other mechanism::
    1134 
    +
    1135 
    +  pip3 install --user grpcio-tools
    1136 
    +
    1137 
    +To actually regenerate the code::
    1138 
    +
    1139 
    +  ./setup.py build_grpc
    1140 
    +
    1141 
    +
    1142 
    +Documenting
    1143 
    +-----------
    222 1144 
     BuildStream starts out as a documented project from day one and uses
    223 1145 
     sphinx to document itself.
    224 1146
    1147 
    +This section discusses formatting policies for editing files in the
    1148 
    +``doc/source`` directory, and describes the details of how the docs are
    1149 
    +generated so that you can easily generate and view the docs yourself before
    1150 
    +submitting patches to the documentation.
    1151 
    +
    1152 
    +For details on how API documenting comments and docstrings are formatted,
    1153 
    +refer to the :ref:`documenting section of the coding guidelines
    1154 
    +<contributing_documenting_symbols>`.
    1155 
    +
    225 1156
    226 1157 
     Documentation formatting policy
    227 1158 
     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    228 1159 
     The BuildStream documentation style is as follows:
    229 1160
    230 
    -* Titles and headings require two leading empty lines above them. Only the first word should be capitalized.
    1161 
    +* Titles and headings require two leading empty lines above them.
    1162 
    +  Only the first word in a title should be capitalized.
    231 1163
    232 
    -  * If there is an ``.. _internal_link`` anchor, there should be two empty lines above the anchor, followed by one leading empty line.
    1164 
    +  * If there is an ``.. _internal_link:`` anchor, there should be two empty lines
    1165 
    +    above the anchor, followed by one leading empty line.
    233 1166
    234 1167 
     * Within a section, paragraphs should be separated by one empty line.
    235 1168
    236 
    -* Notes are defined using: ``.. note::`` blocks, followed by an empty line and then indented (3 spaces) text.
    1169 
    +* Notes are defined using: ``.. note::`` blocks, followed by an empty line
    1170 
    +  and then indented (3 spaces) text.
    1171 
    +
    1172 
    +  * Other kinds of notes can be used throughout the documentation and will
    1173 
    +    be decorated in different ways, these work in the same way as ``.. note::`` does.
    237 1174
    238 
    -* Code blocks are defined using: ``.. code:: LANGUAGE`` blocks, followed by an empty line and then indented (3 spaces) text. Note that the default language is `python`.
    1175 
    +    Feel free to also use ``.. attention::`` or ``.. important::`` to call special
    1176 
    +    attention to a paragraph, ``.. tip::`` to give the reader a special tip on how
    1177 
    +    to use an advanced feature or ``.. warning::`` to warn the user about a potential
    1178 
    +    misuse of the API and explain it's consequences.
    1179 
    +
    1180 
    +* Code blocks are defined using: ``.. code:: LANGUAGE`` blocks, followed by an empty
    1181 
    +  line and then indented (3 spaces) text. Note that the default language is ``python``.
    239 1182
    240 1183 
     * Cross references should be of the form ``:role:`target```.
    241 1184
    242 
    -  * To cross reference arbitrary locations with, for example, the anchor ``_anchor_name``, you must give the link an explicit title: ``:ref:`Link text <anchor_name>```. Note that the "_" prefix is not required.
    1185 
    +  * Explicit anchors can be declared as ``.. _anchor_name:`` on a line by itself.
    1186 
    +
    1187 
    +  * To cross reference arbitrary locations with, for example, the anchor ``anchor_name``,
    1188 
    +    always provide some explicit text in the link instead of deriving the text from
    1189 
    +    the target, e.g.: ``:ref:`Link text <anchor_name>```.
    1190 
    +    Note that the "_" prefix is not used when referring to the target.
    243 1191
    244 1192 
     Useful links:
    245 1193
    246 
    -For further information, please see the `Sphinx Documentation <http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_.
    1194 
    +For further information, please see the `Sphinx Documentation
    1195 
    +<http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_.
    247 1196
    248 1197
    249 1198 
     Building Docs
    ... ... @@ -312,32 +1261,6 @@ the ``man/`` subdirectory, which will be automatically included
    312 1261 
     in the buildstream distribution.
    313 1262
    314 1263
    315 
    -Documenting conventions
    316 
    -~~~~~~~~~~~~~~~~~~~~~~~
    317 
    -We use the sphinx.ext.napoleon extension for the purpose of having
    318 
    -a bit nicer docstrings than the default sphinx docstrings.
    319 
    -
    320 
    -A docstring for a method, class or function should have the following
    321 
    -format::
    322 
    -
    323 
    -  """Brief description of entity
    324 
    -
    325 
    -  Args:
    326 
    -     argument1 (type): Description of arg
    327 
    -     argument2 (type): Description of arg
    328 
    -
    329 
    -  Returns:
    330 
    -     (type): Description of returned thing of the specified type
    331 
    -
    332 
    -  Raises:
    333 
    -     (SomeError): When some error occurs
    334 
    -     (SomeOtherError): When some other error occurs
    335 
    -
    336 
    -  A detailed description can go here if one is needed, only
    337 
    -  after the above part documents the calling conventions.
    338 
    -  """
    339 
    -
    340 
    -
    341 1264 
     Documentation Examples
    342 1265 
     ~~~~~~~~~~~~~~~~~~~~~~
    343 1266 
     The examples section of the documentation contains a series of standalone
    ... ... @@ -419,30 +1342,10 @@ regenerate them locally in order to build the docs.
    419 1342 
          command: build hello.bst
    420 1343
    421 1344
    422 
    -Protocol Buffers
    423 
    -----------------
    424 
    -BuildStream uses protobuf and gRPC for serialization and communication with
    425 
    -artifact cache servers.  This requires ``.proto`` files and Python code
    426 
    -generated from the ``.proto`` files using protoc.  All these files live in the
    427 
    -``buildstream/_protos`` directory.  The generated files are included in the
    428 
    -git repository to avoid depending on grpcio-tools for user installations.
    1345 
    +.. _contributing_testing:
    429 1346
    430 
    -
    431 
    -Regenerating code
    432 
    -~~~~~~~~~~~~~~~~~
    433 
    -When ``.proto`` files are modified, the corresponding Python code needs to
    434 
    -be regenerated.  As a prerequisite for code generation you need to install
    435 
    -``grpcio-tools`` using pip or some other mechanism::
    436 
    -
    437 
    -  pip3 install --user grpcio-tools
    438 
    -
    439 
    -To actually regenerate the code::
    440 
    -
    441 
    -  ./setup.py build_grpc
    442 
    -
    443 
    -
    444 
    -Testing BuildStream
    445 
    --------------------
    1347 
    +Testing
    1348 
    +-------
    446 1349 
     BuildStream uses pytest for regression tests and testing out
    447 1350 
     the behavior of newly added components.
    448 1351
    ... ... @@ -495,6 +1398,7 @@ with::
    495 1398 
     Alternatively, any IDE plugin that uses pytest should automatically
    496 1399 
     detect the ``.pylintrc`` in the project's root directory.
    497 1400
    1401 
    +
    498 1402 
     Adding tests
    499 1403 
     ~~~~~~~~~~~~
    500 1404 
     Tests are found in the tests subdirectory, inside of which
    ... ... @@ -521,8 +1425,9 @@ Tests that run a sandbox should be decorated with::
    521 1425
    522 1426 
     and use the integration cli helper.
    523 1427
    524 
    -Measuring BuildStream performance
    525 
    ----------------------------------
    1428 
    +
    1429 
    +Measuring performance
    1430 
    +---------------------
    526 1431
    527 1432
    528 1433 
     Benchmarking framework

  • contrib/bst-docker-import
    ... ... @@ -93,10 +93,10 @@ echo "INFO: Checking out $element ..." >&2
    93 93 
     $bst_cmd checkout --tar "$element" "$checkout_tar" || die "Failed to checkout $element"
    94 94 
     echo "INFO: Successfully checked out $element" >&2
    95 95
    96 
    -echo "INFO: Importing Docker image ..."
    96 
    +echo "INFO: Importing Docker image ..." >&2
    97 97 
     "${docker_import_cmd[@]}" "$checkout_tar" "$docker_image_tag" || die "Failed to import Docker image from tarball"
    98 
    -echo "INFO: Successfully import Docker image $docker_image_tag"
    98 
    +echo "INFO: Successfully import Docker image $docker_image_tag" >&2
    99 99
    100 
    -echo "INFO: Cleaning up ..."
    100 
    +echo "INFO: Cleaning up ..." >&2
    101 101 
     rm "$checkout_tar" || die "Failed to remove $checkout_tar"
    102 
    -echo "INFO: Clean up finished"
    102 
    +echo "INFO: Clean up finished" >&2

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.

    [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]