Aw: Re: Re: Re: Re: deco-Math project, step 00_a: exact bin and dec 'ranges' (in gnumeric).

[newbie nullzwei <newbie-02 gmx de>]

hello @ Steven,

> Do you know for a fact that Gnumeric implements its rounding by
multiplying 0,3000000000000000044 * 1E16 or are you guessing?

from functions.c / gnumeric_round:
    gnm_float number = value_get_as_float (argv[0]);
    gnm_float digits = argv[1] ? value_get_as_float (argv[1]) : 0;

    if (digits >= 0) {
        if (digits <= GNM_MAX_EXP) {
            gnm_float p10 = gnm_pow10 ((int)digits);
            number = gnm_fake_round (number * p10) / p10; ...

> Gnumeric *does* have access to the digit 4 in the 17th decimal place,
because Gnumeric correctly prints it when you set the cell formatting to
17 decimal places.

yes, but that's string conversion which is slow afaik, too slow to use it in calculations,

> In context, you made a comment that when rounding the value (quote) "it
only adapt's the display, but doesn't change the value" which is
incorrect. Rounding changes the value.

that was said to a statement of you that gnumeric already rounds the value correctly for display. i pointed out that that rounding for display is 'display only'.

best regards and thanks for keeping up ...



b.

---
 

Gesendet: Mittwoch, 07. Juli 2021 um 23:41 Uhr
Von: "Steven D'Aprano" <steve pearwood info>
An: gnumeric-list gnome org
Betreff: Re: Re: Re: Re: deco-Math project, step 00_a: exact bin and dec 'ranges' (in gnumeric).

On Wed, Jul 07, 2021 at 06:01:26PM +0200, newbie nullzwei wrote:

> regarding the 4 being rounded up ... the standard rounding process in
> gnumeric has no access to the digit '4', instead in the process of
> rounding to 16 digits the '4' in 17th place becomes powered up to a ',5'
> ('=0,3000000000000000044 * 1E16' -> 3000000000000000,5) and with rounding
> 'ties away from zero' as usual in spreadsheets this rounds up.

Do you know for a fact that Gnumeric implements its rounding by
multiplying 0,3000000000000000044 * 1E16 or are you guessing? If that is
in fact what Gnumeric does, that would explain why its rounding is
buggy. That's not a safe way to implement rounding unless you are very,
very careful, and clearly the implementation is not careful enough :-(

Gnumeric *does* have access to the digit 4 in the 17th decimal place,
because Gnumeric correctly prints it when you set the cell formatting to
17 decimal places.


> in rounding with IEEE proposal (standard) 'ties to even' this won't
> happen, thus one of the next steps i'm going to propose is to implement an
> additional rounding mode and to use that for 'correction rounding' of
> slightly deviating addition results. that might end the
> 0,30000000000000004 story.


In this specific case, *if* Gnumeric implements rounding in the way you
suggest, then simply changing the internal rounding mode back to the
default IEEE-754 rules will fix the issue for *this* case. But I have no
confidence that it is working in other cases.


> > That is incorrect, it changes the value.
>
> key '0,30000000000000004' in a cell, key '= {that cell} - 0,3' in another
> cell, change the display length of the first cell and watch the result in
> the second cell staying unchanged ... the value is not changed, what is
> displayed to you as 0,3 is 0,30000000000000004 in the background.

Of course, if all you do is change the cell format, all you do is change
the way the value is displayed. That is not in question. I understand
that cell formatting only changes the display.

In context, you made a comment that when rounding the value (quote) "it
only adapt's the display, but doesn't change the value" which is
incorrect. Rounding changes the value.



--
Steve
_______________________________________________
gnumeric-list mailing list
gnumeric-list gnome org
https://mail.gnome.org/mailman/listinfo/gnumeric-list