Re: [jokosher-devel] The new testing and quality assurance team!

["Edward Hervey" <bilboed gmail com>]

Heya,

 Great seeing some QA work being started on jokosher.

 I'd just like to add my 2cents regarding the GStreamer part :) In
order for the GStreamer team to have a better idea of what the bugs
really, this is a proposition of the course to follow:
 * use the gstreamer debugging system (GST_DEBUG=...)
 * use it with the jokosher debugging interleaved in it (they should
appear in the 'python' debugging category).
 * Start to reproduce the bug with GST_DEBUG=2,python:5 (show all
warnings and errors from all categories, and show all messages from
jokosher). If that doesn't show you enough information, reproduce it
with more debugging (ex : GST_DEBUG=3,python:5).
 * debug logs can get massively big, and opening logs that big in a
regular editor is going to chew up all your memory and be slow.
Therefore, use 'less -RS' (show colours and doesn't wrap the lines).
loading several hundred of megabytes log with less is really fast and
searching also.
 * When you have that log, start from the bottom and search up ('?'
command) for the proper keyword (ERROR) where your bug is. Then start
looking upwards from that point to see what your issue is.
 * Once you've got a clearer idea of what the issue is, and if it's
not jokosher giving bad data to GStreamer (yes, it might seem amazing,
but people do mistakes and it's not always gst developers:) ), Create
a gstreamer bugzilla entry with:
 ** The jokosher scenario that triggered the bug (the high-level description)
 ** The gstreamer error message, that is the latest error
 ** What you think might have triggered that issue if you can spot it
 ** The debug log <== IMPORTANT, else we'll ask for it. And attach it
in a compressed way (bzip2)
 ** Which version of gstreamer/gnonlin/plugins you are using. If
you're using cvs versions, please report with an up-to-date checkout.
 ** IF you can reproduce the error with a small script (python
scripts are fine), that's even better. Attach that script.

 Finally, not directly related to GStreamer, but will help avoiding
regressions and/or figure out much faster who's to blame (jokosher or
gst) : writing some unit tests is a GREAT IDEA ! Start with the logic
part of jokosher, test all the API that the UI part is using. We
already have unit tests in all underlying layers : gstreamer, plugins,
gnonlin, gst-python. So once you raise that level of security up one
notch, it will really help. You'd be amazed how much bugs you can in
fact preemptively fix that way :) I already started doing so in PiTiVi
using the python unittest system, those tests are fast to write,
although it's tedious. You can check them out in pitivi/tests/ .

  Good luck to everyone,

    Edward


On 5/25/07, Laszlo Pandy <laszlok2 gmail com> wrote:
> **See the bottom for the time and date of the testing team meeting and
> Gstreamer debugging tutorial**
>
> After many bug filled releases I have decided that it is time to put
> some processes in to make sure we don't do anything stupid for the grand
> 1.0 release. To prevent all the bad stuff from slipping through, a new
> testing team will be created.
>
> The official title is the "Testing and Quality Assurance Team" and
> offers for membership are being extended to *any* member of the
> community. All you have to be able to do is run Jokosher and write on
> the wiki if it works or not. Obviously we would like you to do more than
> that if possible, but no programming skills are required (just testing).
>
> Here are the responsibilities of the testing team:
>   * Make a large checklist of everything that should be working for the
> next release.
>   * After each beta release go through the checklist and make sure
> everything is working.
>   * Prioritise bugs in the bug tracker. Small bugs that ought to be
> fixed get lost because other big important bugs take too long to
> implement. Make sure the devs know where the concentrate.
>   * Constantly harass the developers until the priority bugs are fixed.
> This is a perfectly legitimate way for someone on the testing team to
> get the bug they hate most fixed.
>   * Help me (and possibly other devs) implement automated testing.
> Basically I need to sort through the large checklist and see which parts
> can be easily automated.
>
> Pecisk has been given the position of leader of the testing team, but
> anyone else can join. You will need a userdocs.jokosher.org account, and
> possibly a wiki account, which you can ask us for on IRC. Also I can add
> you to the jokosher-bugs team on launchpad so that you can modify bug
> details.
>
> The testing team will have its first meeting to figure out what needs to
> be done, and who should do it on Sunday, June 10th at 16:00 UTC.
> Following the meeting will be a Gstreamer debugging tutorial so everyone
> can get experience tracking down the nasty gstreamer bugs.
>
> Laszlo
> _______________________________________________
> jokosher-devel-list mailing list
> jokosher-devel-list gnome org
> http://mail.gnome.org/mailman/listinfo/jokosher-devel-list


--
Edward Hervey
Multimedia editing developer / Fluendo S.A.
http://www.pitivi.org/