Re: strange rounding of an arithmetic serie
- From: Allin Cottrell <cottrell wfu edu>
- To: Frédéric Parrenin <parrenin ujf-grenoble fr>
- Cc: gnumeric-list gnome org
- Subject: Re: strange rounding of an arithmetic serie
- Date: Thu, 30 Oct 2014 13:13:07 -0400 (EDT)
On Thu, 30 Oct 2014, Frédéric Parrenin wrote:
If I type 0 in A1 and 0.1 in A2, select A1:A2 and then drag the bottom
right corner, I get:
0
0.1
0.2
0.3
0.4
0.5
0.6000000000000001
I am using 1.12.14 on a linux 64 bit machine.
Is this a known issue?
If yes, what is the corresponding bug report?
Yes, it's a very well known issue, but there's no corresponding bug
report -- because the issue is the inherently finite precision of
computer arithmetic. The "double" datatype comprises 64 bits and
cannot model the real line. You've just discovered that 0.6000...
(with an arbitrary number of trailing zeros) is not a number that can
be exactly represented by a double.
Allin Cottrell
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]