Re: strange rounding of an arithmetic serie

On Thu, 30 Oct 2014, Frédéric Parrenin wrote:

OK I see.
But why this problem does not show up in Calc or XL?
It seems they manage to find a workaround.

They're applying some cosmetic heuristic. If you were to type 0.6000000000000001 into a cell, it would show as 0.6000000000000000. I don't know if these programs have an option "Show the number actually stored" but they ought to!

Allin Cottrell

2014-10-30 18:13 GMT+01:00 Allin Cottrell <cottrell wfu edu>:

On Thu, 30 Oct 2014, Frédéric Parrenin wrote:

 If I type 0 in A1 and 0.1 in A2, select A1:A2 and then drag the bottom
right corner, I get:

I am using 1.12.14 on a linux 64 bit machine.

Is this a known issue?
If yes, what is the corresponding bug report?

Yes, it's a very well known issue, but there's no corresponding bug report
-- because the issue is the inherently finite precision of computer
arithmetic. The "double" datatype comprises 64 bits and cannot model the
real line. You've just discovered that 0.6000... (with an arbitrary number
of trailing zeros) is not a number that can be exactly represented by a

Allin Cottrell

Allin Cottrell
Department of Economics
Wake Forest University, NC

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]