Aw: Re: fuzzy rounding

[newbie nullzwei <newbie-02 gmx de>]

 

thank you, accurate analysis and nice visualization,

exactly such inconsistencies I would like to clear up.

This one may be a bit less inaccurate than it appears at first glance, 'int()' rounds positive and negative values 'towards negative infinity', while the granularity of floats / doubles changes at 0 and alternately grows towards +inf for positive numbers, towards -inf for negative numbers.

Accordingly, for negative numbers the first value below a range boundary is 1 ULP of the range boundary away from it, for positive numbers it is only half a ULP since the granularity below the boundary is more fine grained.

The value directly below the range limit is - for positive numbers - often a 'zombie value' which cannot be counted in single steps of the last digit (in doubles), and which would be rounded to 16 digits fall onto the range limit. In this respect one can discuss to assign him still the higher integer, but actually only if one adjusts the value also accordingly ...

 

regards,

 

b.

---

Gesendet: Freitag, 09. Juli 2021 um 00:14 Uhr
Von: "John Denker via gnumeric-list" <gnumeric-list gnome org>
An: "Gnumeric Mailing List" <gnumeric-list gnome org>
Betreff: Re: fuzzy rounding

On 7/7/21 3:16 PM, Morten Welinder wrote:

> Gnumeric's rounding here is indeed done with
>
> round_to_int(x * 10^d) / 10^d
>
> except that round_to_int deliberately misrounds 0.5-1ulp to 1.

I observe that it's not quite that simple.

I assume the intent is the following:
If the input differs from the nearest integer by only one epsilon,
gnumeric uses the integer value instead.
Call this scheme [A].

Let's focus attention on the int() spreadsheet function.

For positive numbers, I observe that the behavior can be
described as follows:
apply scheme [A], then call POSIX floor().

In contrast, however, for negative numbers I observe that
scheme [A] is not successfully applied. The observed behavior
is identical to POSIX floor().

Diagrams of this are here:
https://www.av8n.com/hack/img48/non-posix-floor_1.png
https://www.av8n.com/hack/img48/non-posix-floor_-1.png

The spreadsheet used to compute the diagrams is here:
https://www.av8n.com/hack/non-posix-floor.gnumeric

I am aware that the function go_fake_floor() in file
...../goffice/goffice/math/go-math.c
contains code that seemingly tries to apply scheme [A],
but even so, the observed numerical result does not conform
to scheme [A].

I don't much care, because I try not to do calculations that
are sensitive to the 16th decimal place. Furthermore, it's
easy to defend against the depredations of scheme [A].
Even so, the observed behavior sure is weird.
_______________________________________________
gnumeric-list mailing list
gnumeric-list gnome org
https://mail.gnome.org/mailman/listinfo/gnumeric-list