Hello gvfs hackers, gvfs currently does not have any automatic test at all, which bothers me. First, some functionality tends to not get manually tested very often; we did have cases where e. g. SMB mounts were broken for several months, as most developers tend to work on Linux only, so we wait for users to tell us. Also, it would be nice to have a place where ad-hoc test scripts for race conditions like in https://bugzilla.gnome.org/show_bug.cgi?id=677648 could be put into, so that we build up regression tests over time. To get a feeling for the task, I have done some experiments and wrote a prototypic integration testsuite which currently covers archive:// for tar.gz and .zip, sftp://, and some tools (gvfs-ls, etc.). This is by far not complete, but already exposes some questions which I'd like to discuss and get feedback on. The current prototype consists of two parts: - http://people.canonical.com/~pitti/tmp/gvfs-test That's the actual testsuite. Currently written in python as that's what's quickest for me to try different ideas, but NB it's a prototype. - http://people.canonical.com/~pitti/tmp/gvfs-testbed Wrapper which builds a "sandbox" in the sense of using a temporary /tmp, /etc/, and /home, and creating a temporary "testuser" user in the sandbox. It uses unshare(1) with tmpfs bind mounts, so it's both very quick and safe - all the bind mounts will be cleaned up automatically as soon as the program terminates, and cannot be accessed from outside, so they do not interfere with the system (well, some stuff like useradd does log to /var/log/auth.log, but that can be covered by bind mounts as well eventually) If you just run "./gvfs-test" it will run the archive:// and tool tests, as they do not need any changes to the system, but skip the sftp:// one. I'd love to have tests which only need user privileges and do not have to change anything in the system, but unfortunately that won't get us too far: e. g. sshd cannot be coerced into using a temporary $HOME for a user, and as soon as we get to using the scsi_debug kernel module to simulate an USB stick for testing the udisks2 (or gdu) backends, we are firmly in the realm of "needs root". So you need to run "sudo ./gvfs-testbed ./gvfs-test" for running the whole test suite. So, my questions: - First, are you interested in eventually adding integration tests to the upstream git at all? If not, I can put it into the Debian/Ubuntu packaging where they will be run as part of https://jenkins.qa.ubuntu.com/view/Quantal/view/AutoPkg%20Test/ but that would make them a lot less useful for other developers. - I found it really hard to run gvfs right out of the build tree, due to all the .service etc. indirection, and some hardcoded paths. I also tried installing into a temp dir and running it out of there with LD_LIBRARY_PATH and some sedery in the .service/.mount files, but that didn't get far either. Do you have a trick for that? How do you test gvfs before release? - For the reason above, because we need root for some things, and we want to be able to test an actual install (daily ISO testing etc.) we'll need a system integration suite. However, some parts of the test should also work without root privs (see above); should these get integrated into "make check", and we search for a way to run gvfs out of the build tree? - What language do you prefer for the test suite? Is Python ok, or perhaps Vala? (please don't say C -- not having exception handling or "finally:" etc. will make the tests unnecessarily hard to write and maintain, and a lot more brittle) - Do you prefer if the test suite uses the command line tools (the current prototype does that), or the API (we can use Gio through introspection from Python)? Presumably we should test them both? FYI, my current TODO list for this is to add tests for cifs (with local samba server), ftp (python has builtin FTP server API), cdrom (with scsi-debug), and USB stick (scsi-debug with a small patch I've been trying to get upstream for 3 months now), as well as completing the archive:// ones and adding more checks to sftp://. I also want to create some reproducers for existing bugzilla reports (you can exfail tests), to see what we need to do to provide reproducers in the tests. Thanks in advance for your thoughts! Martin -- Martin Pitt | http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
Attachment:
signature.asc
Description: Digital signature