Re: bug in gnumeric: rounding error



Hi Morten,

On Thu, 2006-16-02 at 21:43 -0500, Morten Welinder wrote:

It is actually not a bug.

The problem is that there is no way to represent the exact value of
12/5 with a finite number of base-2 digits.  (In the same way you
cannot represent 1/3 in base 10.)  Gnumeric then uses the closest
possible value instead of 12/5 and this happens to be a tiny bit less
than 2.4.  (In the 1/3 case, you would write something like 0.33333
which also happes to be slightly less than the value you aimed for.)

The value that appears to be 22 and is computed as 10+2.4+...+2.4
is thus a little bit less than 22 and the 22s therefore do not get
counted.

Ok, so what can be done about it?

1. Stay with integers.  It's actually possible to multiply all the
   numbers by 5.  It's not very intuitive, though.

2. Replace (max-min)/5 by (max-min)/5*(1+1e-15).  That ensures that
   the value used to approximate 12/5 is slightly bigger than 2.4

While I understand the explanation I am surprised that even if I ask for
25 decimal digits it doesn't show that it is slightly smaller than 22
but if I take its difference from 22 (the integer) it shows a small
difference in the 7th or 8th digit.

Andreas
-- 
Andreas J. Guelzow
Taliesin Software, Shelties, Pyr Sheps
and Shetland Sheep

Attachment: signature.asc
Description: This is a digitally signed message part



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]