GNOME.org
 

Re: [Vala] Testing framework

I have started working on such project,

first effects are here: https://github.com/mspanc/vspec

m.

2016-02-02 3:25 GMT+01:00 Chris Daley <chebizarro gmail com>:


Hi Evan,
Thanks for the offer, I've sent you a pull request on Github. I've got some
new Vala posts in the can that aren't related to Valadate that should be up
later this week as well.
Cheers
Chris D

2016-02-01 16:12 GMT-08:00 Evan Nemerson <evan nemerson com>:


Since you're actively blogging about Vala, how would you feel about
being added to Planet Vala (planet.vala-project.org)?  Nobody on is
very active right now, so it would be good to get some new blood…


On Mon, 2016-02-01 at 14:37 -0800, Chris Daley wrote:

Hi Steven,

It's still very much active, I've just had to take some time out to
do some
work on my night job (managing a Rock band). I'm drafting a blog post
this
afternoon with an update on progress and some new ways to contribute,
so
stay tuned.

Cheers
Chris D



2016-02-01 14:22 GMT-08:00 Steven Oliver <oliver steven gmail com>:


Hey,
I just wanted to drop in and see if any progress had been made on
this?

I for one am very excited about the possibility.

Thank you,
Steven N. Oliver



On Fri, Jan 8, 2016 at 2:55 PM -0800, "Chris Daley" <chebizarro gma
il.com>
wrote:

Hi Al,


Thanks for the input, this is very much appreciated and
incredibly useful!
I had in fact just written a post on my blog asking for exactly
this sort
of feedback - you can read it here -http://chrisdaley.biz/test-dr
iven-development-in-vala-pt-1.html

I've got some time to work on this over the next few weeks and
hope to have
a usable release out for testing not long after, so now's the
time for
anyone else interested in a testing framework to hit me up with
your ideas!

Have a good weekend,
Cheers
Chris D


2016-01-08 14:29 GMT-08:00 Al Thomas :


From: Chris Daley





Sent: Tuesday, 5 January 2016, 1:16
Subject: Re: [Vala] Testing framework

I've done some thinking about this over the last couple of
months and

used

the holiday period to finish off a few things, namely a port
of Gherkin

to

Vala. You can grab it from here if you want to check it out:
https://github.com/chebizarro/gherkin-vala

I'm going to be sketching out what I think is a reasonable
roadmap over the next week or so with a view to an alpha
release around

the

end of February. If anyone is interested in contributing, or
has any
specific ideas about what sort of features they would find
the most

useful,


please get in touch



Hi,

Great work on porting Gherkin 3 to Vala.

I wanted to put forward a few ideas that I have been slowly
researching over the past year or so in the hope they are

insightful to any developments of testing tools for Vala.

Gherkin
-------
Gherkin is a language for structuring human language in a
way that allows business analysts, developers and testers to
define
features of an application.

Some computer languages have tools available to developers to
convert

Gherkin to an outline of a computer program. The following
meaningless

example shows the structure. This uses PHP's Behat to convert
Gherkin

to PHP:

Feature: test

Background: given this is a test # features/test.feature:3

Scenario: testing of app # features/test.feature:5
When i run something
Then it passes

1 scenario (1 undefined)
2 steps (2 undefined)
0m0.17s (9.41Mb)

--- FeatureContext has missing steps. Define them with these
snippets:

/**
* @When i run something
*/
public function iRunSomething()
{
throw new PendingException();
}

/**
* @Then it passes
*/
public function itPasses()
{
throw new PendingException();
}

It is the feature context that forms the basis of generating
automated acceptance tests from the features specified in

Gherkin. The developer then fills in the gaps with code that
drives the tests. In PHP web development this is a tool like
Mink that can drive various headless web browsers.

Automatic code generation for Vala
----------------------------------
I can think of two approaches to generation code. I have
tried neither.

The first is to use libvala to generate a Vala AST then output
the AST. Potentially libvala could be modified to generate a

Genie version of the AST. This appeals to me.

The other approach would be similar to Valadoc:

https://git.gnome.org/browse/valadoc/tree/src/libvaladoc/api/fo
rmalparameter.vala#n131

Code generation could also be useful for anyone wanting to

develop a tool similar to RSpec. So a common approach using
libvala
may be helpful. I think Anjuta CTags also uses libvala for

autosuggestion of function names etc. See:

https://github.com/GNOME/anjuta/blob/master/plugins/symbol-db/a
njuta-tags/ctags-visitor.vala

Acceptance Testing Drivers
--------------------------
This is probably the hardest part given the wide range of
interfaces available.

Gherkin is from Cucumber written in Ruby with web application
development in mind. So I think most tools there use web

interfaces. In the Vala world this could be done with libsoup
for a text analysis of the web interface, but also embedding
Webkit or Gecko which also allows Javascript to be tested.

Vala is often used for desktop GUI development. So Linux
Desktop Testing Project ( http://ldtp.freedesktop.org/wiki/ )
using Assistive Technology Service Provider Interface (


https://en.wikipedia.org/wiki/Assistive_Technology_Service_Prov
ider_Interface
 ) may be relevant.

Of course software is also developed for technical users. So
there are potentially command line interfaces, D-Bus
interfaces, shared
library interfaces and so on to cater for.

For command line interfaces I'm starting to think GLib's
trap_subprocess may be useful:

http://valadoc.org/#!api=glib-2.0/GLib.Test.trap_subprocess
I'm trying to write functional tests for Genie, but some

features need to stop the compilation process. e.g. attempting
to override a protected method in a class. The test should

trap the error from valac and make sure it matches the

expected error.


Test Output
-----------
This is probably moving away from specification by example
with Gherkin and moving towards unit testing. Specifically
it's about my experience with GLib testing framework reports.

Each test can output the results as TAP (Test Anything
Protocol)
by using the --tap switch. I would recommend TAP because it
is processed more easily by a lot of tools.e.g. Jenkins build
server.


In the past I have done unit tests using a script such as:

#/bin/sh

tests="arrays_multiline
variables_declarations
variables_type_inference"

# Build test binaries
for test in $tests
do
valac $test.gs
done

# Run test binaries
gtester --keep-going -o results.xml $tests

# Fix missing info section from results - see
https://bugzilla.gnome.org/show_bug.cgi?id=668035




info="\n\nUnknown\nUnknown\n\n"
sed -i 's||'$info'|g' results.xml

# Generate HTML report
gtester-report results.xml > report.html


I have not figured out a convenient way to get the output as
TAP,

although a script for automake is available:
https://git.gnome.org/browse/glib/tree/tap-driver.sh




Unit Test of Binaries
---------------------
Vala and Genie produce binaries of course.

So far I have produced a test binary that is a Position
Independent Executable (

https://securityblog.redhat.com/2012/11/28/position-independent
-executables-pie/
) that has all the symbols in the dynamic table. This
allows each unit test to treat the test binary as a

shared object to it can test individual functions, but also
be run as an executable for functional testing.

I haven't figured out how to do this for a production binary.
This would probably a static build so the production binary has
the unit tests aggregated after it in memory and the linker
should use the symbol table and not the dynamic symbol table.

For a test binary you may also want to include the --debug
flag in Valac to include Vala filenames. I think this was tried

for code coverage reports with Valum - see

https://coveralls.io/builds/4147345 for example. Not sure of th
e results.


Vala Bugs to be Aware Off
-------------------------
May be of interest:
https://bugzilla.gnome.org/show_bug.cgi?id=704072
https://bugzilla.gnome.org/show_bug.cgi?id=597999
https://bugzilla.gnome.org/show_bug.cgi?id=739725

Hope all this helps frame any work you may do on Vala
testing.

All the best with it,

Al





--
Chris Daley
Pacific Northwest

e: chebizarro gmail com
w: http://chrisdaley.biz
m: +1601 980 1249

s: chebizarro
tw: chebizarro
tz: PDT
_______________________________________________
vala-list mailing listvala-list gnome orghttps://mail.gnome.org/m
ailman/listinfo/vala-list











--
Chris Daley
Pacific Northwest

e: chebizarro gmail com
w: http://chrisdaley.biz
m: +1601 980 1249
s: chebizarro
tw: chebizarro
tz: PDT
_______________________________________________
vala-list mailing list
vala-list gnome org
https://mail.gnome.org/mailman/listinfo/vala-list
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]