... |
... |
@@ -1478,48 +1478,76 @@ Don't get lost in the docs if you don't need to, follow existing examples instea |
1478
|
1478
|
|
1479
|
1479
|
Running tests
|
1480
|
1480
|
~~~~~~~~~~~~~
|
1481
|
|
-To run the tests, just type::
|
|
1481
|
+We use `tox <https://tox.readthedocs.org/>`_ as a frontend run the tests which
|
|
1482
|
+are implemented using `pytest <https://pytest.org/>`_. To run the tests, simply
|
|
1483
|
+navigate to the toplevel directory of your buildstream checkout and run::
|
1482
|
1484
|
|
1483
|
|
- ./setup.py test
|
|
1485
|
+ tox
|
1484
|
1486
|
|
1485
|
|
-At the toplevel.
|
|
1487
|
+By default, the test suite will be run against every supported python version
|
|
1488
|
+found on your host. If you have multiple python versions installed, you may
|
|
1489
|
+want to run tests against only one version and you can do that using the ``-e``
|
|
1490
|
+option when running tox::
|
1486
|
1491
|
|
1487
|
|
-When debugging a test, it can be desirable to see the stdout
|
1488
|
|
-and stderr generated by a test, to do this use the ``--addopts``
|
1489
|
|
-function to feed arguments to pytest as such::
|
|
1492
|
+ tox -e py37
|
1490
|
1493
|
|
1491
|
|
- ./setup.py test --addopts -s
|
|
1494
|
+The output of all failing tests will always be printed in the summary, but
|
|
1495
|
+if you want to observe the stdout and stderr generated by a passing test,
|
|
1496
|
+you can pass the ``-s`` option to pytest as such::
|
|
1497
|
+
|
|
1498
|
+ tox -- -s
|
|
1499
|
+
|
|
1500
|
+.. tip::
|
|
1501
|
+
|
|
1502
|
+ The ``-s`` option is `a pytest option <https://docs.pytest.org/latest/usage.html>`_.
|
|
1503
|
+
|
|
1504
|
+ Any options specified before the ``--`` separator are consumed by ``tox``,
|
|
1505
|
+ and any options after the ``--`` separator will be passed along to pytest.
|
1492
|
1506
|
|
1493
|
1507
|
You can always abort on the first failure by running::
|
1494
|
1508
|
|
1495
|
|
- ./setup.py test --addopts -x
|
|
1509
|
+ tox -- -x
|
1496
|
1510
|
|
1497
|
1511
|
If you want to run a specific test or a group of tests, you
|
1498
|
1512
|
can specify a prefix to match. E.g. if you want to run all of
|
1499
|
1513
|
the frontend tests you can do::
|
1500
|
1514
|
|
1501
|
|
- ./setup.py test --addopts 'tests/frontend/'
|
|
1515
|
+ tox -- tests/frontend/
|
1502
|
1516
|
|
1503
|
1517
|
Specific tests can be chosen by using the :: delimeter after the test module.
|
1504
|
1518
|
If you wanted to run the test_build_track test within frontend/buildtrack.py you could do::
|
1505
|
1519
|
|
1506
|
|
- ./setup.py test --addopts 'tests/frontend/buildtrack.py::test_build_track'
|
|
1520
|
+ tox -- tests/frontend/buildtrack.py::test_build_track
|
1507
|
1521
|
|
1508
|
1522
|
We also have a set of slow integration tests that are disabled by
|
1509
|
1523
|
default - you will notice most of them marked with SKIP in the pytest
|
1510
|
1524
|
output. To run them, you can use::
|
1511
|
1525
|
|
1512
|
|
- ./setup.py test --addopts '--integration'
|
|
1526
|
+ tox -- --integration
|
1513
|
1527
|
|
1514
|
1528
|
By default, buildstream also runs pylint on all files. Should you want
|
1515
|
1529
|
to run just pylint (these checks are a lot faster), you can do so
|
1516
|
1530
|
with::
|
1517
|
1531
|
|
1518
|
|
- ./setup.py test --addopts '-m pylint'
|
|
1532
|
+ tox -- -m pylint
|
1519
|
1533
|
|
1520
|
1534
|
Alternatively, any IDE plugin that uses pytest should automatically
|
1521
|
1535
|
detect the ``.pylintrc`` in the project's root directory.
|
1522
|
1536
|
|
|
1537
|
+.. note::
|
|
1538
|
+
|
|
1539
|
+ While using ``tox`` is practical for developers running tests in
|
|
1540
|
+ more predictable execution environments, it is still possible to
|
|
1541
|
+ execute the test suite against a specific installation environment
|
|
1542
|
+ using pytest directly::
|
|
1543
|
+
|
|
1544
|
+ ./setup.py test
|
|
1545
|
+
|
|
1546
|
+ Specific options can be passed to ``pytest`` using the ``--addopts``
|
|
1547
|
+ option::
|
|
1548
|
+
|
|
1549
|
+ ./setup.py test --addopts 'tests/frontend/buildtrack.py::test_build_track'
|
|
1550
|
+
|
1523
|
1551
|
|
1524
|
1552
|
Adding tests
|
1525
|
1553
|
~~~~~~~~~~~~
|