Re: ver 1.9.4 use more decimal digits

On 2/03/2009 2:32 PM, bill lam wrote:
On Sun, 01 Mar 2009, Morten Welinder wrote:
Correct.  We need 17 digits for "double":

iirc ieee double (8 byte) use 53 bit (including the implied bit) for
mantissa so that it has

log10(2^53)=15.95459 significant digits

16 digit is already enough. the 17 digits used by new gnumeric does
not add more accuracy or precision assuming that ieee standard has
been taken for granted.

Hi Bill,

The question is: What is the smallest integer N such that, for each possible IEEE 754 64-bit number, when you round it to N decimal digits of precision for storage/transmission purposes and then convert it back to the IEEE form, you get the original number?

The answer is: 17

See this:
page 236, "Binary to Decimal Conversion", where it proves that 9 digits (not 8) are required for single precision, and states that likewise 17 are needed for double precision.


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]