[Notes] [Git][BuildStream/buildstream][master] 2 commits: contributing: mandate end-to-end tests, allow others



Title: GitLab

Angelos Evripiotis pushed to branch master at BuildStream / buildstream

Commits:

1 changed file:

Changes:

  • CONTRIBUTING.rst
    ... ... @@ -1547,23 +1547,50 @@ Tests that run a sandbox should be decorated with::
    1547 1547
     
    
    1548 1548
     and use the integration cli helper.
    
    1549 1549
     
    
    1550
    -You should first aim to write tests that exercise your changes from the cli.
    
    1551
    -This is so that the testing is end-to-end, and the changes are guaranteed to
    
    1552
    -work for the end-user. The cli is considered stable, and so tests written in
    
    1553
    -terms of it are unlikely to require updating as the internals of the software
    
    1554
    -change over time.
    
    1555
    -
    
    1556
    -It may be impractical to sufficiently examine some changes this way. For
    
    1557
    -example, the number of cases to test and the running time of each test may be
    
    1558
    -too high. It may also be difficult to contrive circumstances to cover every
    
    1559
    -line of the change. If this is the case, next you can consider also writing
    
    1560
    -unit tests that work more directly on the changes.
    
    1561
    -
    
    1562
    -It is important to write unit tests in such a way that they do not break due to
    
    1563
    -changes unrelated to what they are meant to test. For example, if the test
    
    1564
    -relies on a lot of BuildStream internals, a large refactoring will likely
    
    1565
    -require the test to be rewritten. Pure functions that only rely on the Python
    
    1566
    -Standard Library are excellent candidates for unit testing.
    
    1550
    +You must test your changes in an end-to-end fashion. Consider the first end to
    
    1551
    +be the appropriate user interface, and the other end to be the change you have
    
    1552
    +made.
    
    1553
    +
    
    1554
    +The aim for our tests is to make assertions about how you impact and define the
    
    1555
    +outward user experience. You should be able to exercise all code paths via the
    
    1556
    +user interface, just as one can test the strength of rivets by sailing dozens
    
    1557
    +of ocean liners. Keep in mind that your ocean liners could be sailing properly
    
    1558
    +*because* of a malfunctioning rivet. End-to-end testing will warn you that
    
    1559
    +fixing the rivet will sink the ships.
    
    1560
    +
    
    1561
    +The primary user interface is the cli, so that should be the first target 'end'
    
    1562
    +for testing. Most of the value of BuildStream comes from what you can achieve
    
    1563
    +with the cli.
    
    1564
    +
    
    1565
    +We also have what we call a *"Public API Surface"*, as previously mentioned in
    
    1566
    +:ref:`contributing_documenting_symbols`. You should consider this a secondary
    
    1567
    +target. This is mainly for advanced users to implement their plugins against.
    
    1568
    +
    
    1569
    +Note that both of these targets for testing are guaranteed to continue working
    
    1570
    +in the same way across versions. This means that tests written in terms of them
    
    1571
    +will be robust to large changes to the code. This important property means that
    
    1572
    +BuildStream developers can make large refactorings without needing to rewrite
    
    1573
    +fragile tests.
    
    1574
    +
    
    1575
    +Another user to consider is the BuildStream developer, therefore internal API
    
    1576
    +surfaces are also targets for testing. For example the YAML loading code, and
    
    1577
    +the CasCache. Remember that these surfaces are still just a means to the end of
    
    1578
    +providing value through the cli and the *"Public API Surface"*.
    
    1579
    +
    
    1580
    +It may be impractical to sufficiently examine some changes in an end-to-end
    
    1581
    +fashion. The number of cases to test, and the running time of each test, may be
    
    1582
    +too high. Such typically low-level things, e.g. parsers, may also be tested
    
    1583
    +with unit tests; alongside the mandatory end-to-end tests.
    
    1584
    +
    
    1585
    +It is important to write unit tests that are not fragile, i.e. in such a way
    
    1586
    +that they do not break due to changes unrelated to what they are meant to test.
    
    1587
    +For example, if the test relies on a lot of BuildStream internals, a large
    
    1588
    +refactoring will likely require the test to be rewritten. Pure functions that
    
    1589
    +only rely on the Python Standard Library are excellent candidates for unit
    
    1590
    +testing.
    
    1591
    +
    
    1592
    +Unit tests only make it easier to implement things correctly, end-to-end tests
    
    1593
    +make it easier to implement the right thing.
    
    1567 1594
     
    
    1568 1595
     
    
    1569 1596
     Measuring performance
    



  • [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]