reporting change in percentage of open bugs over time per product

[Luis Villa <luis villa gmail com>]

I don't think I have time for this today, but I wanted to write it down.

The biggest problem we have with traditional reporting of bug counts
in bugzilla is that inflow and outflow is largely determined by
qualities other than actual code quality. In traditional bug tracking,
number of QA people is relatively fixed, number of tests run
relatively fixed/known, and amount of triage and fixing are also
known. On the other hand, we have spikes of all sorts- bugday? yup,
that made us X% less buggy in one day. Yeah. Right.[1] Sun stops
filing bugs against HEAD? Or suddenly pours more resources into
triage? Did our fix rate really go up, or the quality of new releases
really go down, if those things happen? not really. So we have to
figure out some way to control for those things.

We've traditionally had no way of controlling for these things, which
has always made me very reluctant to use our stats for anything
important.

However, it struck me yesterday that most of these factors are fairly
constant across all of our core desktop apps. Bugday? we look at all
apps. Sun/Novell/RedHat/Ubuntu starts/stops paying attention? Fairly
constant across all apps. New distro releases? Fairly constant across
all apps.

So... I think maybe a really interesting stat to look at to find
'ailing' apps is 'percentage of open bugs in the same class of apps.'
What I'd like to see is a graph of where the X axis is 'percentage of
all open bugs that are open in this application' and Y is date. So,
for example, data points for today:

* isgnome open bugs (inc. unconfirmed): 10595
* evo open bugs (ditto): 2503, 23.6% of total
* nautilus open bugs (etc.): 847, 7.99% of total
etc.

So, get that percentage data over time (not easy, unfortunately, but
probably not impossible) and graph it. Was evo 26% of open bugs last
week, and now 23%? Very good sign for them. Nautilus was 7%, now 8%?
Not good for them. The trendlines you'll see there will indicate which
apps are improving and which declining in terms of quality, in a way
that controls for the worst fluctuations that our open system
invariably produces.

Could also whip this into meaningful reports for maintainers-
'controlled for other fluctuations, your # of open bugs was up/down
foo% last week.' Seems like that would be good.

Or so it seems to me. If anyone wants to whip this up it would rock ;)

Luis (can make database dumps available on request for people who want
to play with this)

[1] It *did* make the bugtrack less full of bugs by a nice amount, but
that is not the same as less buggy.