Re: fuzzy rounding

[John Denker <jsd av8n com>]

On 7/7/21 3:16 PM, Morten Welinder wrote:

> Gnumeric's rounding here is indeed done with
>
>     round_to_int(x * 10^d) / 10^d
>
> except that round_to_int deliberately misrounds 0.5-1ulp to 1.

I observe that it's not quite that simple.

I assume the intent is the following:
  If the input differs from the nearest integer by only one epsilon,
  gnumeric uses the integer value instead.
Call this scheme [A].

Let's focus attention on the int() spreadsheet function.

For positive numbers, I observe that the behavior can be
described as follows:
        apply scheme [A], then call POSIX floor().

In contrast, however, for negative numbers I observe that
scheme [A] is not successfully applied. The observed behavior
is identical to POSIX floor().

Diagrams of this are here:
  https://www.av8n.com/hack/img48/non-posix-floor_1.png
  https://www.av8n.com/hack/img48/non-posix-floor_-1.png

The spreadsheet used to compute the diagrams is here:
  https://www.av8n.com/hack/non-posix-floor.gnumeric

I am aware that the function go_fake_floor() in file
 ...../goffice/goffice/math/go-math.c
contains code that seemingly tries to apply scheme [A],
but even so, the observed numerical result does not conform
to scheme [A].

I don't much care, because I try not to do calculations that
are sensitive to the 16th decimal place. Furthermore, it's 
easy to defend against the depredations of scheme [A].
Even so, the observed behavior sure is weird.