Re: ver 1.9.4 use more decimal digits
- From: John Machin <sjmachin lexicon net>
- To: bill lam <cbill lam gmail com>
- Cc: gnumeric list <gnumeric-list gnome org>
- Subject: Re: ver 1.9.4 use more decimal digits
- Date: Mon, 02 Mar 2009 16:29:39 +1100
On 2/03/2009 3:57 PM, bill lam wrote:
On Mon, 02 Mar 2009, John Machin wrote:
The question is: What is the smallest integer N such that, for each
possible IEEE 754 64-bit number, when you round it to N decimal digits
of precision for storage/transmission purposes and then convert it back
to the IEEE form, you get the original number?
The answer is: 17
Thanks John. That make sense.
BTW there is a compilation option in gnumeric "with-long-double". I'm
not sure if that use 10 byte storage or otherwise. Does it require more
than 17 digit storage format, or more importantly, compromise the
validity of numerical computation that tested for 8-byte double?
Sorry, I don't know what "with-long-double" does -- I'm just an
interested onlooker, not a dev :-)
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]