Re: Re: Re: Re: deco-Math project, step 00_a: exact bin and dec 'ranges' (in gnumeric).
- From: Steven D'Aprano <steve pearwood info>
- To: gnumeric-list gnome org
- Subject: Re: Re: Re: Re: deco-Math project, step 00_a: exact bin and dec 'ranges' (in gnumeric).
- Date: Thu, 8 Jul 2021 08:22:53 +1000
On Wed, Jul 07, 2021 at 06:16:46PM -0400, Morten Welinder wrote:
Gnumeric's rounding here is indeed done with
round_to_int(x * 10^d) / 10^d
except that round_to_int deliberately misrounds 0.5-1ulp to 1. That's
not important here.
Thank you Morten for introducing some reality into this discussion. It's
good to have confirmation that this is what Gnumeric is doing.
--
Steve
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]