On 02/28/2017 04:05 PM, Michael Catanzaro wrote:
> Regarding installed tests. The benefits of installed tests versus make
> check tests are not very clear to me. I don't think it should be
> necessary to install the tests in order to be able to run and test
> applications. That indicates a failure somewhere in the test
> infrastructure.
In-tree tests are beneficial to application developers.
Installed tests are beneficial to distributions, systems implementer,
automated QA, etc.
As an example use case, I've made the tests in GJS able to run both uninstalled and installed. (The latter is optional via a configure flag. That flag also controls which set of tests is run by "make check". I didn't know about gnome-desktop-testing-runner when I wrote that part, so I should probably remove the flag, and always install tests, and always have "make check" run the uninstalled ones.)
There are also some extra GJS tests that can't sensibly be run installed at all, which test internal code that isn't exposed via an API. These are always run uninstalled. I personally didn't interpret the goal as advocating replacing all uninstalled tests with installed tests, as that wouldn't make sense for this reason.
I find both sets of tests useful for the reasons Christian mentioned. However, we have a Travis CI configuration for GJS and it jumps through quite a lot of hoops in order to be able to run the installed tests properly. Therefore I'd love to see the installed tests run automatically in Continuous or something like that, which is more the intended environment for running them in.
I think automated installed tests can be really valuable to signal problems with the GNOME installation as a whole at an early stage. Especially if we can clearly articulate to developers what their purpose is and what kind of test makes a good installed test; I certainly didn't understand that fully when I started maintaining GJS.
Regards,
Philip C
