Re: ver 1.9.4 use more decimal digits



On 2/03/2009 3:57 PM, bill lam wrote:
On Mon, 02 Mar 2009, John Machin wrote:
The question is: What is the smallest integer N such that, for each possible IEEE 754 64-bit number, when you round it to N decimal digits of precision for storage/transmission purposes and then convert it back to the IEEE form, you get the original number?

The answer is: 17

Thanks John. That make sense.
BTW there is a compilation option in gnumeric "with-long-double". I'm
not sure if that use 10 byte storage or otherwise. Does it require more
than 17 digit storage format, or more importantly, compromise the
validity of numerical computation that tested for 8-byte double?

Sorry, I don't know what "with-long-double" does -- I'm just an interested onlooker, not a dev :-)



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]