Re: reporting change in percentage of open bugs over time per product

[Luis Villa <luis villa gmail com>]

On 7/24/05, michael chang <thenewme91 gmail com> wrote:
> On 7/22/05, Luis Villa <luis villa gmail com> wrote:
> > The biggest problem we have with traditional reporting of bug counts
> > in bugzilla is that inflow and outflow is largely determined by
> > qualities other than actual code quality. In traditional bug tracking,
> > number of QA people is relatively fixed, number of tests run
> > relatively fixed/known, and amount of triage and fixing are also
> > known. On the other hand, we have spikes of all sorts- bugday? yup,
> > that made us X% less buggy in one day. Yeah. Right.[1] Sun stops
> > filing bugs against HEAD? Or suddenly pours more resources into
> > triage? Did our fix rate really go up, or the quality of new releases
> > really go down, if those things happen? not really. So we have to
> > figure out some way to control for those things.
> 
> If I read this right, this basically says that maybe there should be a
> way for e.g. a bug that is triaged as a duplicate or something to not
> count as a "fixed" but thereby making it look like many bugs were
> fixed, but at the same time for it not to count as "unfixed" (thereby
> making it look like nothing happened).

Well, that is definitely part of it. But there are other factors- for
example, we always have more users during beta than alpha, so more
bugs get reported- even though we're fairly certain the product is
actually more stable and after-the-fact analysis suggests that the
percentage of 'high' priority bugs is lower later. [This particular
issue is the one that scares the most hell out of traditional software
managers, because they've been trained that if the curve of the graph
is going down, that's when you're approaching release, whereas to us,
if the curve of the graph is going up, then you're getting more users
;)]

To further complicate things, if you haven't marked the bugs duplicate
or invalid yet, how do you account for them when they are new? i.e.,
even in the best weeks we only barely break even on bug triage, but
for stats to be most useful they have to count bugs as they come in.
So, a simplified example- in a given week:

120 bugs come in
40 are marked fixed
40 are marked duplicate
20 are invalid/incomplete

that leaves 20 bugs open, net. Odds are pretty good that 12 of those
remaining bugs are duplicate/invalid/incomplete, so only 8 valid bugs
of 120 filed. Do you report 8? 20? so on.

> Is there a way to generate statistics so that triaged bugs don't count
> as "fixed" bugs per se?  Or something like that, e.g. bugs marked as
> duplicates don't indicate "fixed" bugs, or something?  [So the number
> of fixed bugs is compared to the number of open bugs -- since the
> triaged/dups are not open bugs, the fixed bugs will still be positive,
> but not seemingly inflated.]
> 
> This is the way I see it, and my apologies if my interpretation is nonsense.

Not nonsense at all.
Luis