| ... |
... |
@@ -1547,50 +1547,40 @@ Tests that run a sandbox should be decorated with:: |
|
1547
|
1547
|
|
|
1548
|
1548
|
and use the integration cli helper.
|
|
1549
|
1549
|
|
|
1550
|
|
-- End-to-end testing is the most valuable.
|
|
1551
|
|
-- tests which observe how these behaviors impact and define the outward user
|
|
1552
|
|
- experience.
|
|
1553
|
|
-
|
|
1554
|
1550
|
You must test your changes in an end-to-end fashion. Consider the first end to
|
|
1555
|
|
-be the appropriate user interface, and the other the change you have made. The
|
|
1556
|
|
-aim for testing is to make assertions about how you impact and define the
|
|
|
1551
|
+be the appropriate user interface, and the other end to be the change you have
|
|
|
1552
|
+made.
|
|
|
1553
|
+
|
|
|
1554
|
+The aim for our tests is to make assertions about how you impact and define the
|
|
1557
|
1555
|
outward user experience. You should be able to exercise all code paths via the
|
|
1558
|
1556
|
user interface, just as one can test the strength of rivets by sailing dozens
|
|
1559
|
|
-of ocean liners.
|
|
|
1557
|
+of ocean liners. Keep in mind that your ocean liners could be sailing properly
|
|
|
1558
|
+*because* of a malfunctioning rivet. End-to-end testing will warn you that
|
|
|
1559
|
+fixing the rivet will sink the ships.
|
|
1560
|
1560
|
|
|
1561
|
1561
|
The primary user interface is the cli, so that should be the first candidate
|
|
1562
|
1562
|
'end' for testing. Most of the value of BuildStream comes from what you can
|
|
1563
|
1563
|
achieve with the cli.
|
|
1564
|
1564
|
|
|
1565
|
|
-We also have a *"Public API Surface"*, which you should consider the next
|
|
1566
|
|
-target. This is mainly for advanced users to implement their plugins against.
|
|
1567
|
|
-
|
|
1568
|
|
-- Useful for Internal API surfaces, e.g. YAML and CasCache.
|
|
1569
|
|
-- Test what the function does, rather than what it should do, need to think in thelarger context.
|
|
1570
|
|
-- Don't fool yourself that because a function has a test that passes, that it
|
|
1571
|
|
- is doing the right thing.
|
|
1572
|
|
-- If something is not reachable in end-to-end testing, perhaps it is dead
|
|
1573
|
|
- code and if so should be removed, instead of tested.
|
|
1574
|
|
-- Unit tests can provide direct indication of what is broken.
|
|
1575
|
|
-- You should be able to exercise all code paths via the user interface, just
|
|
1576
|
|
- as one can test the strength of rivets by sailing dozens of ocean liners,
|
|
1577
|
|
-- providing coverage isn't all of testing.
|
|
1578
|
|
-- it could turn out that your ocean liners are sailing properly *because* of
|
|
1579
|
|
- a malfunctioning rivet, this is also a very undesirable situation.
|
|
1580
|
|
-- End-to-end testing and unit-testing are not mutually exclusive, you must
|
|
1581
|
|
- implement end-to-end tests.
|
|
1582
|
|
-
|
|
1583
|
|
-You should first aim to write tests that exercise your changes from the cli.
|
|
1584
|
|
-This is so that the testing is end-to-end, and the changes are guaranteed to
|
|
1585
|
|
-work for the end-user. The cli is considered stable, and so tests written in
|
|
1586
|
|
-terms of it are unlikely to require updating as the internals of the software
|
|
1587
|
|
-change over time.
|
|
1588
|
|
-
|
|
1589
|
|
-It may be impractical to sufficiently examine some changes this way. For
|
|
1590
|
|
-example, the number of cases to test and the running time of each test may be
|
|
1591
|
|
-too high. It may also be difficult to contrive circumstances to cover every
|
|
1592
|
|
-line of the change. If this is the case, next you can consider also writing
|
|
1593
|
|
-unit tests that work more directly on the changes.
|
|
|
1565
|
+We also have a :ref:`*"Public API Surface"*<contributing_public_api_surface>`,
|
|
|
1566
|
+which you should consider a secondary target. This is mainly for advanced users
|
|
|
1567
|
+to implement their plugins against.
|
|
|
1568
|
+
|
|
|
1569
|
+Note that both of these targets for testing are guaranteed to continue working
|
|
|
1570
|
+in the same way across versions. This means that tests written in terms of them
|
|
|
1571
|
+will be robust to large changes to the code. This important property means that
|
|
|
1572
|
+BuildStream developers can make large refactorings without needing to rewrite
|
|
|
1573
|
+fragile tests.
|
|
|
1574
|
+
|
|
|
1575
|
+Another user to consider is the BuildStream developer, therefore internal API
|
|
|
1576
|
+surfaces are also targets for testing. For example, the YAML loading code and
|
|
|
1577
|
+the CasCache. Remember that these surfaces are still just a means to the end of
|
|
|
1578
|
+providing value through the cli and the Public API Surface.
|
|
|
1579
|
+
|
|
|
1580
|
+It may be impractical to sufficiently examine some changes in an end-to-end
|
|
|
1581
|
+fashion. For example, the number of cases to test and the running time of each
|
|
|
1582
|
+test may be too high. Such typically low-level things, e.g. parsers, may also
|
|
|
1583
|
+be tested with unit tests - alongside the mandatory end-to-end tests.
|
|
1594
|
1584
|
|
|
1595
|
1585
|
It is important to write unit tests in such a way that they do not break due to
|
|
1596
|
1586
|
changes unrelated to what they are meant to test. For example, if the test
|
| ... |
... |
@@ -1598,6 +1588,9 @@ relies on a lot of BuildStream internals, a large refactoring will likely |
|
1598
|
1588
|
require the test to be rewritten. Pure functions that only rely on the Python
|
|
1599
|
1589
|
Standard Library are excellent candidates for unit testing.
|
|
1600
|
1590
|
|
|
|
1591
|
+Unit tests only make it easier to implement things correctly, end-to-end tests
|
|
|
1592
|
+make it easier to implement the right thing.
|
|
|
1593
|
+
|
|
1601
|
1594
|
|
|
1602
|
1595
|
Measuring performance
|
|
1603
|
1596
|
---------------------
|
