Re: Re: Re: Re: deco-Math project, step 00_a: exact bin and dec 'ranges' (in gnumeric).

[Steven D'Aprano <steve pearwood info>]

On Wed, Jul 07, 2021 at 06:01:26PM +0200, newbie nullzwei wrote:

>    regarding the 4 being rounded up ... the standard rounding process in
>    gnumeric has no access to the digit '4', instead in the process of
>    rounding to 16 digits the '4' in 17th place becomes powered up to a ',5'
>    ('=0,3000000000000000044 * 1E16' -> 3000000000000000,5) and with rounding
>    'ties away from zero' as usual in spreadsheets this rounds up.

Do you know for a fact that Gnumeric implements its rounding by 
multiplying 0,3000000000000000044 * 1E16 or are you guessing? If that is 
in fact what Gnumeric does, that would explain why its rounding is 
buggy. That's not a safe way to implement rounding unless you are very, 
very careful, and clearly the implementation is not careful enough :-(

Gnumeric *does* have access to the digit 4 in the 17th decimal place, 
because Gnumeric correctly prints it when you set the cell formatting to 
17 decimal places.


>    in rounding with IEEE proposal (standard) 'ties to even' this won't
>    happen, thus one of the next steps i'm going to propose is to implement an
>    additional rounding mode and to use that for 'correction rounding' of
>    slightly deviating addition results. that might end the
>    0,30000000000000004 story.


In this specific case, *if* Gnumeric implements rounding in the way you 
suggest, then simply changing the internal rounding mode back to the 
default IEEE-754 rules will fix the issue for *this* case. But I have no 
confidence that it is working in other cases.


>    > That is incorrect, it changes the value.
>
>    key '0,30000000000000004' in a cell, key '= {that cell} - 0,3' in another
>    cell, change the display length of the first cell and watch the result in
>    the second cell staying unchanged ... the value is not changed, what is
>    displayed to you as 0,3 is 0,30000000000000004 in the background.

Of course, if all you do is change the cell format, all you do is change 
the way the value is displayed. That is not in question. I understand 
that cell formatting only changes the display.

In context, you made a comment that when rounding the value (quote) "it 
only adapt's the display, but doesn't change the value" which is 
incorrect. Rounding changes the value.



-- 
Steve