Hi, On Mon, 2016-09-19 at 12:03 +0200, Hanno Böck wrote:
I recently reported a couple of bugs to GNOME-components that were easily discovered with a feature called Address Sanitizer (asan): https://bugzilla.gnome.org/show_bug.cgi?id=762417 (glib) https://bugzilla.gnome.org/show_bug.cgi?id=762493 (glib) https://bugzilla.gnome.org/show_bug.cgi?id=768441 (gnome-session) The first two are out of bounds reads in glib that get triggered by running the test suite with "make check". The third is a buffer overflow in gnome-session that gets triggered on every startup.
Nice work!
Things that could be done, by increasing complexity: * Run the existing test suites with address sanitizer This could be mostly automated, it's usually as simple as doing something like: ./configure CFLAGS="-fsanitize=address -g -fno-common -U_FORTIFY_SOURCE" CXXFLAGS="-fsanitize=address -g -fno-common -U_FORTIFY_SOURCE" LDFLAGS="-fsanitize=address -g -fno-common -U_FORTIFY_SOURCE" make ASAN_OPTIONS="log_path=/var/log/asan/asan-error" make check
It’s something which would be good to have running as part of CI, along with the other -fsanitize options (address, thread, leak, undefined) and things like -D_FORTIFY_SOURCE=1. Unfortunately, these flags are mostly mutually exclusive, which means that we’d need 5 new CI pipelines. I guess the right approach is to set up a few more parallel instances of gnome-continuous, each running with different compiler options. I’ll have a play with that tomorrow. I guess eventually it could be integrated into gnome-continuous as a new set of tree suffixes.
* Run GNOME itself and its applications with asan enabled. Test suites are fine, but they don't replace real testing, i.e. actually running the software. Ideally one would built the full software stack with address sanitizer enabled to find bugs both in the applications themselve and in libraries.
Automated tests complement manual testing, and require a lot less effort over time. (While it would be lovely in theory, in practice, very few maintainers are going to recompile each of their projects with 5 different sets of compiler options and manually test every feature in the UI before a release — that’s a huge amount of work.) Enabling asan in the jhbuild configuration by default would be one option, with the idea that people would then develop using it by default. However, asan interferes with Valgrind, and I think that Valgrind is probably the more useful tool to have working in a development environment like jhbuild. Additionally, this means consciously prioritising asan over any other -fsanitize or hardening flags (which is fine, but it is a tradeoff).
* Use Fuzzing. This is especially interesting for everything that is parsing complex data formats. There has been a lot of improvement in fuzzing techniques lately and advanced fuzzing tools like american fuzzy lop or libfuzzer can greatly help in improving and securing code. I have some introductions to these things on the fuzzing project webpage: https://fuzzing-project.org/tutorials.html
Not really related to asan, but good advice all the same. afl is pretty amazing.
I discussed this already with Tobias Mueller recently and he already started finding a couple of issues with asan: https://bugzilla.gnome.org/show_bug.cgi?id=771607 https://bugzilla.gnome.org/show_bug.cgi?id=771608 https://bugzilla.gnome.org/show_bug.cgi?id=771609 https://bugzilla.gnome.org/show_bug.cgi?id=771610 https://bugzilla.gnome.org/show_bug.cgi?id=771612 https://bugzilla.gnome.org/show_bug.cgi?id=771298 https://bugzilla.gnome.org/show_bug.cgi?id=771289
I should reiterate that I think doing testing like this manually is a trap. For this to be useful, it needs to be automated — that will rely on each module having good test coverage, but that’s a good goal to aim for anyway. Philip
Attachment:
signature.asc
Description: This is a digitally signed message part