[librsvg/librsvg-2.40] rsvg-test: allow skipping directories that start with "ignore" as well as files

[Federico Mena Quintero <federico src gnome org>]

commit 69911f6bf46f61b012052ba585512a83776a5375
Author: Federico Mena Quintero <federico gnome org>
Date:   Wed Dec 14 10:20:08 2016 -0600

    rsvg-test: allow skipping directories that start with "ignore" as well as files
    
    It turns out that it's more convenient to move the whole set of
    problematic SVGs into an ignore/ subdirectory for now.

 tests/README.md                                    |  229 ++++++++++++++++++++
 .../reftests/svg1.1/ignore-pservers-grad-17-b.svg  |   92 --------
 tests/rsvg-test.c                                  |   28 ++-
 3 files changed, 249 insertions(+), 100 deletions(-)
---
diff --git a/tests/README.md b/tests/README.md
new file mode 100644
index 0000000..57bde80
--- /dev/null
+++ b/tests/README.md
@@ -0,0 +1,229 @@
+ibrsvg test suite
+==================
+
+Librsvg's test harness is built upon Glib's GTest utility functions.
+These let you define tests in the C language.
+
+There are three different styles of tests:
+
+* crash - Ensures that loading and parsing (but not rendering) a
+  particular SVG doesn't yield a GError.
+
+* styles - Loads and parses an SVG, and checks that a particular SVG
+  object has a certain style.
+
+* rsvg-test - The main test suite; loads, parses, and renders SVGs and
+  compares the results against reference PNG images.
+
+These are all "black box tests": they run the library with its public
+API, and test the results.  They do not test the library's internals;
+just the output.
+
+Additionally, the library's source code has smaller unit tests for
+particular sections of the code.
+
+
+Running the test suite
+----------------------
+
+The easiest way to run all the tests is to go to librsvg's toplevel
+directory and run `make check`.  This will run both the small unit
+tests and the black box tests in this `librsvg/tests` directory.
+
+If you want to run just the black box tests, go into this
+`librsvg/tests` directory and run `make check`.
+
+Those commands will yield exit code 0 if all the tests pass, or
+nonzero if some tests fail.
+
+Running `make check` will produce a `test/test-suite.log` file.  You can
+see this file for the details of failed tests.
+
+Additionally, all three sets of black box tests (crash, styles,
+rsvg-test) will produce a test report in a text file.  In the tests
+directory, you can see `crash.log`, `styles.log`, `rsvg-test.log`,
+respectively.
+
+
+# Tests and test fixtures
+
+## Image-based reference tests for rsvg-test.c
+
+These will load, parse, and render an SVG, and compare the results
+against a reference PNG image.
+
+Each image-based reference test uses two files: `foo.svg` and
+`foo-ref.png`.  The test harness will render `foo.svg` and compare the
+results to `foo-ref.png`.  Currently we assume a pixel-perfect match.
+If there are differences in the output, the test will fail; see
+"Examining failed reference tests" below.
+
+These files can go anywhere under the `fixtures/reftests`
+directory; the `rsvg-test` program will recursively look inside
+`fixtures/reftests` for all SVG files, render them, and compare them to
+the `-ref.png` reference images.
+
+SVG test files or entire subdirectories in fixtures/reftests whose
+names begin with "ignore" will be skipped from the tests.  That is,
+anything that matches "fixtures/reftests/ignore*" will not be included
+in the tests.  You can use this to skip a few problematic files
+temporarily.
+
+As of 2016/Nov/03 we have an informal organization of these files:
+
+* `fixtures/reftests/*.svg` - Tests for special situations
+  that arose during development.
+
+* `fixtures/reftests/bugs/*.svg` - Tests for particular bug numbers.
+  Please use the bug number from Bugzilla, like 123456.svg, and the
+  corresponding 123456-ref.png for the known-good reference image.
+
+* `fixtures/reftests/svg1.1` - Tests from the W3C's SVG test suite.
+  These are supposed to test all of SVG's features; we'll add them one
+  by one as librsvg starts implementing the features.
+
+### Examining failed reference tests
+
+Let's say you run `make check` and see that one of the tests fails.
+For example, `rsvg-test.log` may have lines that look like
+
+```
+# Storing test result image at /tmp/paths-data-18-f-out.png
+# 6798 pixels differ (with maximum difference of 255) from reference image
+
+# Storing test result image at /tmp/paths-data-18-f-diff.png
+not ok 29 /rsvg-test/reftests/svg1.1/paths-data-18-f.svg
+FAIL: rsvg-test 29 /rsvg-test/reftests/svg1.1/paths-data-18-f.svg
+```
+
+This means that the test named
+`/rsvg-test/reftests/svg1.1/paths-data-18-f.svg` failed:  those are
+autogenerated test names from GTest.  In this case, it means that the
+test file `fixtures/reftests/svg1.1/paths-data-18-f.svg` got rendered,
+and produced incorrect output when compared to
+`fixtures/reftests/svg1.1/paths-data-18-f-out.png`.
+
+When a test fails, rsvg-test creates two images in `/tmp`:
+
+```
+/tmp/foo-out.png
+/tmp/foo-diff.png
+```
+
+In this case, `foo-out.png` is the actual rendered output, which is presumed to
+be incorrect, since it differs from the `foo-ref.png` reference image.
+The `foo-diff.png` is a "visual diff" that you can see in an image
+viewer; pixels that differ are highlighted.
+
+It is up to you to decide what to do next:
+
+* If the `foo-out.png` image looks correct, and the only difference
+  with respect to the `foo-ref.png` reference image is that
+  antialiased edges look different, or font rendering is slightly
+  different due to the font-rendering machinery in your system, you
+  can just regenerate the test image.  See "Regenerating reference
+  images" below.
+
+* If the `foo-out.png` image is obviously wrong when compared to the
+  `foo-ref.png` reference, you can file a bug in
+  https://bugzilla.gnome.org.  You can wait until someone fixes it, or
+  try to fix the bug yourself!
+
+* Any other situation of course deserves attention.  Feel free to ask
+  the maintainers about it; even if you figure out the problem
+  yourself, a failed test almost always indicates a problem that is
+  not just on your end.  See the `librsvg/MAINTAINERS` file for where
+  to reach them.
+
+
+### Regenerating reference images
+
+Let's continue with the example above, where the test
+`/rsvg-test/reftests/svg1.1/paths-data-18-f.svg` failed.  Let's say you
+fix the bug, or determine that the output image is in fact correct,
+and just differs from the reference image due to antialiasing
+artifacts.  In this case, you'll want to regenerate the reference
+image so the test passes again.
+
+If you want to regenerate the reference image
+`fixtures/reftests/foo/bar-ref.png` from the corresponding `bar.svg`, you can run
+
+```
+rsvg-convert -o fixtures/reftests/foo/bar-ref.png fixtures/reftests/foo/bar.svg
+```
+
+This is just the normal rsvg-convert program doing its job; it will
+just render the SVG image and output it to the specified PNG file.
+
+You can then run `make check` again and ensure that the tests pass.
+
+
+### Issues with the official SVG test suite
+
+Our SVG files in tests/fixtures/reftests/svg1.1 come from the "SVG 1.1
+Second Edition test suite" archive linked here:
+
+https://www.w3.org/Graphics/SVG/WG/wiki/Test_Suite_Overview
+
+We don't know how the reference PNG files in that archive are
+generated.  However, they are done in such a way that objects tend not
+to be pixel-aligned.  For example, many tests have a rectangular frame
+around the whole viewport, defined like this:
+
+  <rect id="test-frame" x="1" y="1" width="478" height="358" fill="none" stroke="#000000"/>
+
+This specifies no stroke with, so it uses 1 by default.  The desired
+effect is "stroke this rectangle with a 1-pixel wide line".
+
+However, notice that the (x, y) coordinates of the rect are (1, 1).
+This means that the actual bounds of the stroked outline are from
+(0.5, 0.5) to (479.5, 359.5).  The result is a fuzzy outline: it
+occupies two pixels of width, with each pixel having half-black
+coverage.
+
+Some elements in the reference PNG images from the official SVG test
+suite are pixel-aligned, and some are not, like the example test-frame
+above.  It looks like their renderer uses a (0.5, 0.5) offset just for
+strokes, but not for fills, which sounds hackish.
+
+Our test suite doesn't use special offsets so that SVG images not from
+the official test suite are rendered "normally".  This means that the
+reference images from the official test suite will always fail
+initially, since stroke outlines will be fuzzy in librsvg, but not in
+the test suite (and conversely, SVGs *not* from the test suite would
+be crips in librsvg but probably not in the test suite's renderer
+renderer).
+
+Also, the original reference PNGs either use fonts that are different
+from those usually on free software systems, or they use SVG fonts
+which librsvg currently doesn't support (with glyph shapes referenced
+from a secondary SVG).
+
+In any case, look at the results by hand, and compare them by eye to
+the official reference image.  If the thing being tested looks
+correct, and just the outlines are fuzzy --- and also it is just the
+actual font shapes that are different --- then the test is probably
+correct.  Follow the procedure as in "Regenerating reference images"
+listed above in order to have a reference image suitable for librsvg.
+
+
+## Crash tests for crash.c
+
+These load and parse an SVG, and ensure that there are no errors in
+the process.  Note that this does *not* render the images.
+
+The SVG images live in the `fixtures/crash` directory.  The `crash`
+program will look recursively look inside `fixtures/crash` for all SVG
+files, load them, and ensure that no GError was produced while loading
+each file.
+
+
+## Style tests for styles.c
+
+These load and parse an SVG, ask librsvg to fetch an SVG object by its
+id, and ensure that the object has certain style properties with
+particular values.
+
+The SVG images live in the `fixtures/styles` directory.  The
+corresponding object IDs and values to be tested for are in the
+`styles.c` source code.
diff --git a/tests/rsvg-test.c b/tests/rsvg-test.c
index 2386ddd..0969333 100644
--- a/tests/rsvg-test.c
+++ b/tests/rsvg-test.c
@@ -173,17 +173,29 @@ save_image (cairo_surface_t *surface,
 static gboolean
 is_svg_or_subdir (GFile *file)
 {
-  char *basename;
-  gboolean result;
+    char *basename;
+    gboolean ignore;
+    gboolean result;
 
-  if (g_file_query_file_type (file, 0, NULL) == G_FILE_TYPE_DIRECTORY)
-    return TRUE;
+    result = FALSE;
 
-  basename = g_file_get_basename (file);
-  result = !g_str_has_prefix (basename, "ignore-") && g_str_has_suffix (basename, ".svg");
-  g_free (basename);
+    basename = g_file_get_basename (file);
+    ignore = g_str_has_prefix (basename, "ignore");
 
-  return result;
+    if (ignore)
+       goto out;
+
+    if (g_file_query_file_type (file, 0, NULL) == G_FILE_TYPE_DIRECTORY) {
+       result = TRUE;
+       goto out;
+    }
+
+    result = g_str_has_suffix (basename, ".svg");
+
+out:
+    g_free (basename);
+
+    return result;
 }
 
 static cairo_status_t