[gnome-build-meta.wiki] Update OpenQA for GNOME developers - tidy up 'when tests fail'
- From: Sam Thursfield <sthursfield src gnome org>
- To: commits-list gnome org
- Cc:
- Subject: [gnome-build-meta.wiki] Update OpenQA for GNOME developers - tidy up 'when tests fail'
- Date: Fri, 20 Aug 2021 15:07:09 +0000 (UTC)
commit b382f89f5f08eea082e1736970d345d8d472d427
Author: Sam Thursfield <sam afuera me uk>
Date: Fri Aug 20 15:07:09 2021 +0000
Update OpenQA for GNOME developers - tidy up 'when tests fail'
openqa/OpenQA-for-GNOME-developers.md | 15 ++++-----------
1 file changed, 4 insertions(+), 11 deletions(-)
---
diff --git a/openqa/OpenQA-for-GNOME-developers.md b/openqa/OpenQA-for-GNOME-developers.md
index 5cba65e..05094db 100644
--- a/openqa/OpenQA-for-GNOME-developers.md
+++ b/openqa/OpenQA-for-GNOME-developers.md
@@ -30,24 +30,17 @@ We would like to run the tests for every MR, which requires us to first [speed u
## When tests fail
-The test failure will be shown in the Web UI at https://openqa.gnome.org/.
+Tests can fail for several reasons. You will see the failure in the Web UI at https://openqa.gnome.org/, and
the corresponding Gitlab CI job will also fail.
-If the machine fails to install or boot this is likely a regression somewhere
-in GNOME or Freedesktop SDK.
+First check the "Logs & Assets" tab. Most useful is `autoinst-log.txt`, containing the log of installing the
ISO and running the testcases - this should log what went wrong. The `video.ogv` file can also be useful to
see what went wrong.
-If the test fails because GNOME's UI has changed, we might need to update the
-test in question.
+If you see "no candidate needle matched" or "`assert_screen` failure", this indicates a graphical change
meaning the screenshot no longer matches. If the UI change is intentional then we will need to update the
needle. See below.
-In either case, the first step is to
-[open a gnome-build-meta issue](https://gitlab.gnome.org/GNOME/gnome-build-meta/-/issues).
+If in doubt about a test failure, [open a gnome-build-meta
issue](https://gitlab.gnome.org/GNOME/gnome-build-meta/-/issues).
## How to update a test
-A well-written test is robust against minor changes, but major changes to GNOME's
-UI will result in a test failure at the `assert_screen` step. In this case the
-[needle](http://open.qa/docs/#_needles) needs updating for a UI change.
-
OpenQA's [developer mode](http://open.qa/docs/#_developer_mode) allows you to
update the needle from the web UI.
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]