Re: deco-Math project, step 00_a: exact bin and dec 'ranges' (in gnumeric).

[John Denker <jsd av8n com>]

On 7/7/21 2:41 PM, Steven D'Aprano wrote:

> Do you know for a fact that Gnumeric implements its rounding by 
> multiplying 0,3000000000000000044 * 1E16 or are you guessing? 

Use the source, Luke.

There's no need for guessing. Here's what gnumeric does:

> double
> go_fake_floor (double x)
> {
>         x = go_d2d (x);
>
>         if (x == floor (x))
>                 return x;
>
>         return (x >= 0)
>                 ? floor (go_add_epsilon (x))
>                 : floor (go_sub_epsilon (x));
> }

That's from .../goffice/goffice/math/go-math.c

Three comments:

1) If I were doing it, I wouldn't do it that way. I would stick
 with POSIX floor().

2) It shouldn't matter! If your algorithm is sensitive to plus
 or minus one machine epsilon, you need a better algorithm.

 For each positive integer, there is exactly one representable
 floating point number where this differs from POSIX floor().

 (For negative numbers, it's wildly non-POSIX, but that's a
  whole different story.)

3) You can work around this if you really care:
  if(floor(x)<=x,floor(x),floor(x)-1)   # for positive x

  But again, it shouldn't matter.