[gnome-build-meta.wiki] Update OpenQA for GNOME developers - tidy up 'when tests fail'



commit b382f89f5f08eea082e1736970d345d8d472d427
Author: Sam Thursfield <sam afuera me uk>
Date:   Fri Aug 20 15:07:09 2021 +0000

    Update OpenQA for GNOME developers - tidy up 'when tests fail'

 openqa/OpenQA-for-GNOME-developers.md | 15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)
---
diff --git a/openqa/OpenQA-for-GNOME-developers.md b/openqa/OpenQA-for-GNOME-developers.md
index 5cba65e..05094db 100644
--- a/openqa/OpenQA-for-GNOME-developers.md
+++ b/openqa/OpenQA-for-GNOME-developers.md
@@ -30,24 +30,17 @@ We would like to run the tests for every MR, which requires us to first [speed u
 
 ## When tests fail
 
-The test failure will be shown in the Web UI at https://openqa.gnome.org/.
+Tests can fail for several reasons. You will see the failure in the Web UI at https://openqa.gnome.org/, and 
the corresponding Gitlab CI job will also fail.
 
-If the machine fails to install or boot this is likely a regression somewhere
-in GNOME or Freedesktop SDK.
+First check the "Logs & Assets" tab. Most useful is `autoinst-log.txt`, containing the log of installing the 
ISO and running the testcases - this should log what went wrong. The `video.ogv` file can also be useful to 
see what went wrong.
 
-If the test fails because GNOME's UI has changed, we might need to update the
-test in question.
+If you see "no candidate needle matched" or "`assert_screen` failure", this indicates a graphical change 
meaning the screenshot no longer matches. If the UI change is intentional then we will need to update the 
needle. See below.
 
-In either case, the first step is to
-[open a gnome-build-meta issue](https://gitlab.gnome.org/GNOME/gnome-build-meta/-/issues).
+If in doubt about a test failure, [open a gnome-build-meta 
issue](https://gitlab.gnome.org/GNOME/gnome-build-meta/-/issues).
 
 
 ## How to update a test
 
-A well-written test is robust against minor changes, but major changes to GNOME's
-UI will result in a test failure at the `assert_screen` step. In this case the
-[needle](http://open.qa/docs/#_needles) needs updating for a UI change.
-
 OpenQA's [developer mode](http://open.qa/docs/#_developer_mode) allows you to
 update the needle from the web UI.
 


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]