Re: strange rounding of an arithmetic serie

[Allin Cottrell <cottrell wfu edu>]

On Thu, 30 Oct 2014, Frédéric Parrenin wrote:

> OK I see.
> But why this problem does not show up in Calc or XL?
> It seems they manage to find a workaround.

They're applying some cosmetic heuristic. If you were to type 
0.6000000000000001 into a cell, it would show as 0.6000000000000000.
I don't know if these programs have an option "Show the number 
actually stored" but they ought to!

Allin Cottrell

> 2014-10-30 18:13 GMT+01:00 Allin Cottrell <cottrell wfu edu>:
>
> > On Thu, 30 Oct 2014, Frédéric Parrenin wrote:
> >
> >  If I type 0 in A1 and 0.1 in A2, select A1:A2 and then drag the bottom
> > > right corner, I get:
> > > 0
> > > 0.1
> > > 0.2
> > > 0.3
> > > 0.4
> > > 0.5
> > > 0.6000000000000001
> > >
> > > I am using 1.12.14 on a linux 64 bit machine.
> > >
> > > Is this a known issue?
> > > If yes, what is the corresponding bug report?
> >
> > Yes, it's a very well known issue, but there's no corresponding bug report
> > -- because the issue is the inherently finite precision of computer
> > arithmetic. The "double" datatype comprises 64 bits and cannot model the
> > real line. You've just discovered that 0.6000... (with an arbitrary number
> > of trailing zeros) is not a number that can be exactly represented by a
> > double.
> >
> > Allin Cottrell

--
Allin Cottrell
Department of Economics
Wake Forest University, NC