Re: Writing tests for gvfs

[Alexander Larsson <alexl redhat com>]

On Thu, 2012-08-23 at 17:13 +0200, Martin Pitt wrote:
> Hello gvfs hackers,
> 
> gvfs currently does not have any automatic test at all, which bothers
> me. First, some functionality tends to not get manually tested very
> often; we did have cases where e. g. SMB mounts were broken for
> several months, as most developers tend to work on Linux only, so we
> wait for users to tell us. Also, it would be nice to have a place
> where ad-hoc test scripts for race conditions like in
> https://bugzilla.gnome.org/show_bug.cgi?id=677648 could be put into,
> so that we build up regression tests over time.

There are actually some tests, but they are not really integrated with
gvfs nor run regulary unfortunately. The live-g-file test in
glib/gio/tests is meant to be able to run against non-local backends and
tests a lot of the corner-cases in the GIO api that GVfs backends have
to support.

But yeah, this is clearly a bad situation and we would love to have a
better testsuite that we ran regularly.

> To get a feeling for the task, I have done some experiments and wrote
> a prototypic integration testsuite which currently covers archive://
> for tar.gz and .zip, sftp://, and some tools (gvfs-ls, etc.). This is
> by far not complete, but already exposes some questions which I'd
> like to discuss and get feedback on.
> 
> The current prototype consists of two parts:
> 
>  - http://people.canonical.com/~pitti/tmp/gvfs-test
> 
>    That's the actual testsuite. Currently written in python as that's
>    what's quickest for me to try different ideas, but NB it's a
>    prototype.
>
>  - http://people.canonical.com/~pitti/tmp/gvfs-testbed
> 
>    Wrapper which builds a "sandbox" in the sense of using a temporary
>    /tmp, /etc/, and /home, and creating a temporary "testuser" user in
>    the sandbox. It uses unshare(1) with tmpfs bind mounts, so it's
>    both very quick and safe - all the bind mounts will be cleaned up
>    automatically as soon as the program terminates, and cannot be
>    accessed from outside, so they do not interfere with the system
>    (well, some stuff like useradd does log to /var/log/auth.log, but
>    that can be covered by bind mounts as well eventually)

Another approach would be to use a real VM as testbed. We could even
ship a pre-built one with whatever dependencies is needed. That would be
arch-specific and somewhat larger/cumbersome, but it would let us do
isolated testing without depending on the services installed on the
system (i.e. sshd, twistd, etc). It also avoids having to run as root.
Whats your opinion on this?

I also think we need to ship a bunch of testcases in the form of
directories and files we want to explode onto the various backends so we
can try to access them via gvfs.

> So, my questions:
> 
>  - First, are you interested in eventually adding integration tests to
>    the upstream git at all? If not, I can put it into the
>    Debian/Ubuntu packaging where they will be run as part of
>    https://jenkins.qa.ubuntu.com/view/Quantal/view/AutoPkg%20Test/ but
>    that would make them a lot less useful for other developers.

Yes, we're very interested in this. Its sadly lacking in the 

>  - I found it really hard to run gvfs right out of the build tree, due
>    to all the .service etc. indirection, and some hardcoded paths. I
>    also tried installing into a temp dir and running it out of there
>    with LD_LIBRARY_PATH and some sedery in the .service/.mount files,
>    but that didn't get far either. Do you have a trick for that? How
>    do you test gvfs before release?

I don't do any specific testing in the release, but I always run a gnome
git desktop where I can test things. I realize this is not a great way
though, and any improvements here would be great. ostree & co seems like
a way forward here, especially as gvfs tests can run headless with a
small subset of the gnome stack built.

>  - For the reason above, because we need root for some things, and we
>    want to be able to test an actual install (daily ISO testing etc.)
>    we'll need a system integration suite. However, some parts of the
>    test should also work without root privs (see above); should these
>    get integrated into "make check", and we search for a way to run
>    gvfs out of the build tree?

Having *some* kind of sanity check at buildtime seems like a good idea
to me, even if its just a subset. 

All of gvfs is grounded in the session dbus instance, so if you run a
separate instance of dbus-daemon --session with the right config file it
*should* be able to run from the build tree, although you need some
changes to the daemon/mount.c to make MOUNTABLE_DIR selectable at
runtime. Should all be doable.

>  - What language do you prefer for the test suite? Is Python ok, or
>    perhaps Vala? (please don't say C -- not having exception handling
>    or "finally:" etc. will make the tests unnecessarily hard to write
>    and maintain, and a lot more brittle)

Thats up to whoever does the work. I think python makes a great deal of
sense. Although in some particular tricky tests we might need to write a
helper binary in some lower level language like C which we can call out
to from the tests.

>  - Do you prefer if the test suite uses the command line tools (the
>    current prototype does that), or the API (we can use Gio through
>    introspection from Python)? Presumably we should test them both?

It should use the Gio api imho. The command line tools are not competent
enought to test anything but the most rudimentary details.

> FYI, my current TODO list for this is to add tests for cifs (with local
> samba server), ftp (python has builtin FTP server API), cdrom (with
> scsi-debug), and USB stick (scsi-debug with a small patch I've been
> trying to get upstream for 3 months now), as well as completing the
> archive:// ones and adding more checks to sftp://. I also want to
> create some reproducers for existing bugzilla reports (you can exfail
> tests), to see what we need to do to provide reproducers in the tests.

It would be good to test dav too, and various different types of ftp
servers too, as these behave rather differently at times.